On tuning ext4 for storage nodes

[Alexey]

We use an in-memory cache to write to the database, by default it syncs every hour.

[Zetanova]

Its not about a in-memory cache by itself.

It’s about that they solution of the sqlite lock contention issue
produced of the file-walker random IO issue.

It’s basically impossible anymore to run a node on a single HDD
dedicated or shared with an low IO process.

The solution to add file-system metadata cache in memory or separate dedicated flash storage seams more a workaround then anything else.

What I could extract from the forum, is that a node requires to overcome the file-walker issue around 1GB fs-cache per 1TB data. It makes low HW like raspberry-pi 3,4,5 already not meat this requirement and to run the node as a background task on a server seams on spare/backup drives not feasible.

Then there is the issue with the file directory fragmentation (ext4 and maybe others).
This could be solved to restructure the storage directory to add files only in the current-week directory and files from previous weeks could only be deleted but none added.

But this would require a file-locator component that would be based on a fast directory (database)

If required, even the current hash-map directory layout could be used in parallel.

I know that simpler is better, but to answer the question of the current inventory
to make full scan in random mode on the drive makes little sense.

[JWvdV]

@Zetanova

I know that simpler is better, but to answer the question of the current inventory
to make full scan in random mode on the drive makes little sense.

But that’s is what the badger cache is all about, I thought? Serves the same purpose as a read cache like LVM-hotspotcache or L2ARC.

I mean: aren’t you solving a problem, for which already a solution is in trial mode?

[Zetanova]

@JWvdV

badger cache

I wrote him this even, the badger cache would resolve the file-walker random IO issue.
But not the directory fragmentation and stale/ghost data from partially uploaded chunks.

The badger cache would be require to use it as a file-locator, but then it would not be a “cache” anymore. It would then be more a “honey badger”

The main idea is to drop new chunks only in the latest directory example: storage/2024-32/0/chunkId
this would require a lookup in a fast map/index over a file-locator

Benefits of a file-locator in the “honey badger”

• only the latest directory would require a check on restart or node crash.
• on a new week, the previous week directory would be read and delete only and could be garbage collected or maybe differently optimized in the future
• Multiple drives could be used per node, too extend a node or copy “weeks” slowly over
• A flash+hdd mode for new/hot data and an archive hdd’s
• most likely others …

[Alexey]

@Zetanova

It’s about that they solution of the sqlite lock contention issue
produced of the file-walker random IO issue.

The filewalker will update the database only when it’s finished the scan. So, usually they should not lock a database while working.

@Zetanova

The solution to add file-system metadata cache in memory or separate dedicated flash storage seams more a workaround then anything else.

We implemented a different cache with badger, so it is used to cache a metadata and speed up all filewalkers.

@Zetanova

What I could extract from the forum, is that a node requires to overcome the file-walker issue around 1GB fs-cache per 1TB data. It makes low HW like raspberry-pi 3,4,5 already not meat this requirement and to run the node as a background task on a server seams on spare/backup drives not feasible.

The badger cache should help a low power devices to survive. However, it requires to disable a lazy mode (because the badger even more sensitive to multiple accesses of different processes).

@Zetanova

The main idea is to drop new chunks only in the latest directory example: storage/2024-32/0/chunkId
this would require a lookup in a fast map/index over a file-locator

Suggestion: add one more subfolder level to blobs folders structure

Today I wrote about my problem with slow file deletion on ext4 filesystem (mostly for SLC folders): How do you solve slow file deletion on ext4? Right now Storj uses folder structure “/ab/filename.sj1” with 1024 subfolders for every Satellite. During intensive tests SLC sends tens of millions files. I see more than 50mil records in piece_expiration.db so number of files in each folder can reach 50 000 or even 100 000. And we hope one day Satellites working with customer’s data will surpass the…

and

How do you solve slow file deletion on ext4?

Very often referenced: Benchmark: Deep directory structure vs. flat directory structure to store millions of files on ext4 | by hartator | Medium

[Zetanova]

@Alexey

The badger cache should help a low power devices to survive. However, it requires to disable a lazy mode (because the badger even more sensitive to multiple accesses of different processes).

This problem I don’t understand or better a journal “event-sourcing” would solve it.
In dotnet I could make in a hand full days, but I am at golang a noob.

Maybe I will implement a tech-demo fork.

I will go currently with the workaround with a new rsp5 the zfs+special device metadata with a 5GB/1TB ratio over a partition on a NVMe

[Toyoo]

@Zetanova

In dotnet I could make in a hand full days, but I am at golang a noob.

Maybe you can just try writing high-level pseudocode? Might be enough to show the idea.

[Alexey]

There is a limit in the badge cache implementation, which requires an exclusive access. This is also why it’s an experimental feature.

[Zetanova]

I forgot to ask, did the disabled sqlite implementation create/open the db-file with the WAL feature enabled?

sqlite creates/opens a db file in legacy mode 2001 style with WAL disabled. The writes will modify the file-data in place and they will block reads until the transaction finishes.

With WAL enabled, regular writes will not block reads on the db file.
https://sqlite.org/wal.html

This would be more then the half of my above approach description and most likely this was the issue with the storj-sqlite implementation.

The other optimization/features that a journal offers would only be necessary in high OP scenarios >50k op/s. where one file upload would generate around 2 + chunks operations. Don’t really think that even the big nodes have this action, or?

For completion here are the features that would a journal extend beside sqlite-Wal:

• pass-through where events do not even gets committed into the journal and instantly get applied/processed. This is feature is near the same as a file upload would store its state in memory and writes to the sqlite db only at the end once.

• Differencing pub+sync and pub+flush it is related to pub/sub and event-sourcing topic. The producer, in our case the file-upload process, would know best when it needs critical events/data to be persisted flush and processed at a later stage (can be minutes later) or when it needs to know that all processed events got fully processed sync. As an example for sync, a file upload needs to respond to the uploader and to guarantee that a follow up read-file of the uploader would succeed. As an example for flush, a delete-fileS would publish multiple events followed by a single flush.

[Alexey]

@Zetanova

I forgot to ask, did the disabled sqlite implementation create/open the db-file with the WAL feature enabled?

Why do you think so?

$ ls -l /mnt/x/storagenode2/storage/*.db*
-rw-r--r-- 1 root   root    75304960 Oct 12 09:59 /mnt/x/storagenode2/storage/bandwidth.db
-rw-r--r-- 1 root   root       32768 Oct 12 10:29 /mnt/x/storagenode2/storage/bandwidth.db-shm
-rw-r--r-- 1 root   root        4152 Oct 12 10:29 /mnt/x/storagenode2/storage/bandwidth.db-wal

I can see a wal files.
You also may check the code.

[Zetanova]

The sqlite of the file catalog got removed from the source and was replaced by the file-walker approach with the reason that the sqlite-db is bottle-nacking the file uploads.

[Alexey]

It is not removed, please, check the code.
SQLite databases are still used.
However, we implemented a badger cache:

Badger cache: are we ready?

We are all on 1.108 now. In our nodes we have a really important improvement disabled by default and ready to launch… badger cache! Can we talk about it? Are you testing it? Is it safe to enable to all nodes? Is it better to wait?

And the ultimate feature of

The whole disk/partition allocation feature

Hello, If I set the flag for dedicated disk use, will the disk be 100% used? Or will there be enough space left for the DBs? Or should I move the DBs to another disk for safety?

Please, use them!

[Mitsos]

@Alexey

However, we implemented a badger cache:

We are all on 1.108 now. In our nodes we have a really important improvement disabled by default and ready to launch… badger cache! Can we talk about it? Are you testing it? Is it safe to enable to all nodes? Is it better to wait?

Here’s me complaining (yet again!) without any single reason, just out of the blue. 4 months later and we (=baremetal) still don’t have a way to configure where badger is stored.

[Alexey]

Yeees. But if you so concerned, and run the node without docker, you can still reroute it with a symlinks.
I can help, please post a description of your environment

[Mitsos]

Not that concerned, thank you.

[PieceKeeper]

@Toyoo

-i 65536

Hi @Toyoo, do you think the -i 65536 argument on mke2fs is still a safe value today? I ask because you wrote in another thread (Recommended GB of RAM / TB stored? - #7 by Toyoo), that the average piece size has gone down alot.

[Toyoo]

Looking at the numbers on my nodes I do think it should be good enough for the forseeable future, but I would probably step down any new file systems to ~45000.

With hashstore on the horizon this number probably matters less now though.

[alpharabbit]

The average piece size for all of my nodes is over 200K, so no problem.

[PieceKeeper]

Thanks @Toyoo and @alpharabbit

Merry Christmas everybody

[snorkel]

With the startup-piece-scan now reporting the pieces size and pieces number, it’s easy to calculate the average piece size.
For my 6TB node, the average piece size is: 203845 bytes.