How do you solve slow file deletion on ext4?

You may try to convert it to LVM and add SSD as a cache, not sure that it could help with deletions though.
You may also try to enable an experimental cache feature to see, would it help or not.

      --pieces.file-stat-cache string                            optional type of file stat cache. Might be useful for slow disk and limited memory. Available options: badger (EXPERIMENTAL)

Please note, it requires the lazy mode to be disabled.

If you chose for a so-called hotspot cache, it might speed-up metadata reads. But doesn’t do for writing. If you chose for a write cache it’s vice versa.

See: lvmcache(7) - Linux manual page

Again a reason to go for special vdevs in ZFS in my opinion (because promoting look-ups and deletes; possibility to add cache if you want). Especially if you’re already converting. But I won’t spread the gospel all to much. The longer people keep crippling with their nodes, the bigger the chance they give up the more repair ingress I get… :wink:

1 Like

If you have plain ext4 setup you need to migrate all nodes data to LVM+cache from scratch. There isn’t an easy way or at least that’s what I know.
Btw I’m going to experiment Badger with some nodes. I opened a thread

The fastest way is to use partclone.ext4 much faster than rsync. But you require down time, it will run at near max speed.

This is what I did when I moved from regular ext4 to lvm with ext4

e2image is even faster… But same issue.

1 Like

You know when you kinda want to help people… because you can see the solution to their problem… but you kinda don’t want to help them because you’re sorta competing against them… :stuck_out_tongue_winking_eye:

It’s more like a passive aggressive reaction, because of the ‘stubbornness’ I sometimes feel: ext4 on HDD is failing me, I need to change something but it shouldn’t be too much effort > Yeah, let’s go with LVM + caching, an external journal device or whatever; all options we know about nil on how it works in context of STORJ.
Every change takes effort, bit ZFS with special devs is a real proven one.

1 Like

So is LVMcache with ext4.

1 Like

Sure, show me the storj specific benchmarks!

No, I really really don’t.
There is plenty to go around for everyone.

My nodes greatly sped up when I added LVM cache. That’s all I care for now, and it should work as a proof they can work. I’ve spent too-much-time doing benchmarks anyway, sorry, it’s time for someone else to volunteer at this point.

2 Likes

I believe this thread will regain interest in less than 7 days :slight_smile:
Filestat cache will help on deleting? I don’t think but I will share some results (I enabled cache everywhere now)

The deletion process has been going on since Sunday, the disks of both my nodes are loaded at 100%. Based on previous advice, I switched from windows ntfs first to synology btrfs and then to ext4. I can see that even this will not be good, because the X16s will not be able to handle this load. SSD cache would also degrade quickly, zfs and LVM do not work in synology and docker, I think. I am sad. Unfortunately, Badger cache does not help with this, as far as I can see.

2 Likes

FWIW that was my main worry with ext4+LVM read-only cache (after going from NTFS->ext4 with limited improvements) but has now vanished completely. When I finished migrating last month the SSD was at 96% life. And today… still 96%. That’s with a good amount of test data and cleanup happening lately.

I do expect it will drop a percent every few months and it’s a “consumable” spent on StorJ but if have a SSD to spare it might be worth a shot. I have yet to randomly look at my server and see more than minor HDD activity ongoing. Logs do show mass deletions taking 20 minutes sometimes, barely noticeable and was worth the peace of mind for me. Nodes purr like a cat.

2 Likes

yeah my rule of thumb is that a plain ext4 disk maxes out around 6TB of data when under heavy load (light load can probably go much longer). A long time ago I had a plain btrfs drive and that seemed even worse than ext4.

I don’t know Synology’s caching options. I have my zfs configured to cash only metadata, and you know what? the write load on my SSD is actually very small. totally manageable by consumer gear. (there is a lot of read activity).

in my experience, also, yeah, the badger cache only helps the used space filewalker, not the garbage collection process or trash deletes.

1 Like

I see SSD Cache | DSM - Synology Knowledge Center

There’s both read-only (preferred here) and write cache options. Focuses on random IO, can loose the SSD. Seems similar to LVM & looks reasonable to me - just below zfs special device.

Just wait for it to finish. The current amount of deletions is caused by fixing a bug where exprired TTL data didn’t get deleted. Once it finished it will hopefully not happen again at this scale.

1 Like

When deleting pieces the node calls stat to retrieve the piece size, so theoretically this is cached so the chief operation would be the delete, but maybe @elek can confirm if the cache is also being used here.

Tried following the code but not entirely sure as everything seems to be hidden away in the context object.

[quote="mtone, post:70, topic:27260,
I see SSD Cache | DSM - Synology Knowledge Center
[/quote]

that lead to this interesting option that synology has to pin metadata for cache SSD for btrfs. .

Should I pin all Btrfs metadata to an SSD cache? - Synology Knowledge Center

Even though I normally poo-poo the performance of plain btrfs, this might be a really good option for a storj node. I don’t know if any storjers have used this config lately.

it helps, but not significantly:

The deletion process would be speed up a little bit too, but again - the main time consuming operation is deleting itself.