Trash is not cleaning up

Node Operators troubleshooting
,
1

Hi,
since the big deletes happened due to the satellite decommissioning, on most of my nodes the trash space got freed after 7 days, but not on this one node. It has exactly 1TB of trash since more than 30 days and it seems like no garbage collection is kicking in. It’s sharing the IP with other nodes and only got 30GB ingress this month, so it can’t be new deletes that add up to the 1TB if I look at the space-used chart.

Bildschirmfoto 2023-08-17 um 11.23.29
Bildschirmfoto 2023-08-17 um 11.23.291990×610 44 KB

Bildschirmfoto 2023-08-17 um 11.23.39
Bildschirmfoto 2023-08-17 um 11.23.392750×274 34.3 KB

Bildschirmfoto 2023-08-17 um 11.24.31
Bildschirmfoto 2023-08-17 um 11.24.311462×1358 117 KB

Can I somehow manually trigger GC?

Node Version is 1.85.1. It is a docker-node on Unraid, running on a separate EXT4 CMR disk.

2

What does find /mnt/storagenode/node/storage/trash/ -mindepth 1 -type f -mtime -8 |wc -l say?

What does find /mnt/storagenode/node/storage/trash/ -mindepth 1 -type f -mtime +8 |wc -l say?

I also have files older than 7 days in the trash of my storagenodes, but only a few GB.

It should return 0.

2 Likes
3

that’s weird, it says this:

user@host:/mnt/disks/Storagenode2/storagenode2_data/storage# find trash/ -mindepth 1 -type f -mtime -8 |wc -l
34525
user@host:/mnt/disks/Storagenode2/storagenode2_data/storage# find trash/ -type f | wc -l
34525

Looks like it’s only 34525 files, I expected much more. And all of them are younger than a week.
I also checked the disk now and it reports the following:

/dev/sdf1       5.5T  2.8T  2.8T  51% /mnt/disks/Storagenode2

user@host:/mnt/disks/Storagenode2/storagenode2_data/storage# du -sh ./trash/
9.7G	./trash/

Looks like the node is just reporting a wrong value for trash space used. Maybe that gets fixed with the next file walker run…

4

Ah yes, I gave you the wrong command. -mtime +8 should return 0.

5

ah makes sense, yes it’s returning 0:

user@host:/mnt/disks/Storagenode2/storagenode2_data/storage# find trash/ -mindepth 1 -type f -mtime +8 |wc -l
0
6

I’m seeing something similar on my node as well. Dashboard reports 1.1TB of trash, did a du on it and it only around 225GB

7

This is weird. Do you have any errors related to a filewalker?

8

I restarted the node, file walker ran and now it’s updated and reports only 10GB in trash. So my issue is solved, it was just a reporting issue that got fixed by a new filewalker run.

1 Like