Trash-Problems on QNAP NAS

[beli]

Hi!
After digging several hours trough all existing trash-cleanup-problems it seem to aim onto slow hardware (qnap nas 4 bay). The storagenode could not catch up getting rid of the trash.

i’m deleting via rm -rfv #trashfolder/2024-08-* since 2 days

Does anyone have a hint for me to optimize trash-handling on ext4 filesystems?
The raid system is fully read-utilized since weeks.

[ifraixedes]

Hi @beli

Let me try to understand your post.

I see that there are 2 things.

• One is that the storagenode native system to clean up the trash folder eats too many IOPS and your QNAP cannot catch up.
• Two, because of the effect of 1, you are deleting the trash folder with rm -rf, which also is going quite slow, and the raid system is fully read-utilized.

Can you state that I understood it well?
Can you say what’s your RAID type (e.g. 5)?

[Alexey]

You may read this thread:

How do you solve slow file deletion on ext4?

Hi! I have a few relatively large nodes 10Tb+, working on Toshiba HDDs (“Enterprise Capacity” series, non-SMR) in Ubuntu 24.04 and formatted in ext4 and mounted with “noatime”. Other hardware is not a “potato” - Ryzen 9 7900 with 32GB RAM. All is used only by a few Storj nodes, 1 node per drive, no RAIDs. Last month these drives were filled to 60-80% with test SLC pieces and now it came time of it’s expirations. The problem is that old files are deleting much slower than it were uploaded. And…

However, there is no ideal solution so far.

[beli]

hi!
Yes - nailed it. i’m deleting trash from august since three days

there is a raid 5 running over 4 disks.

[beli]

Lets hope

How do you solve slow file deletion on ext4?

I don’t expect this amount of trash accumulating again. It was caused by a bug with the TTL database and the bloomfilter not catching this data. This should be fixed now.

[EasyRhino]

beli, it’s ok many of us have been in your position. it’s a hangover from the massive amounts of test data from the summer.

Others on the forums have found that fastest day to manually delete a lot of files is to pipe it from the find command instead of using rm. Because even rm is slow:

Large temporary folder

find temp/ -mtime +3 -type f -delete Make sure you’re in the right folder. This will delete all files in temp/ older than 3 days.

But in general, performance wise, the disk array is suffeering under the weight of millions of tiny files. General things that can boost performance are:

• set up a SSD caching layer for metadata (ZFS, LVM, and synology-btrfs can do this)
• don’t use an array, use one node per disk (after your node shrinks from the deletes you might be able to juggle things)
• more RAM for the system might help
• there are some minor file system tweaks to ext4 and ntfs and zfs that can be made
• use the badger cache feature for storj (this helps wilh the used space filewalker on startup, the reads are kind of slow.

Another note is that the maximum node size is officially 24TB. larger nodes can’t efficiently find garbage in the garbage collection process.

What are the requirements for a Storage Node on V3?

[beli]

Ah, I think I’m slowly starting to understand how things are going.
Alright, the trash gets emptied once, and then everything should run as usual.
I don’t intend to turn the NAS server into a moon rocket
At the moment, my hobby time is still going into the smart home.
Thanks to everyone!

[beli]

fyi

trash folder is deleting by find since three days. rm was working for three days before.

i didn’t understand why storagenode was utilizing the drive full for weeks but not deleting a file

[Alexey]

It likely deletes them, but very slowly because it’s a slow operation on your setup.
The databases should be updated when you restart the node (after deletion would be finished).
If you wouldn’t have any errors related to a filewalker and/or databases, it should update the usage in the databases and on your dashboard.