I have a mix of internal and external WD disks. All CMR.
You still didn’t answer the question from here.
If the last line contains “started” then it’s still deleting.
To speed it up you could follow my approach When will "Uncollected Garbage" be deleted? - #441 by donald.m.motsinger
1 Like
2024-09-24T12:10:58Z INFO pieces:trash emptying trash started {“Process”: “storagenode”, “Satellite ID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”}
This is the last line I have about “emptying trash” in the node with more trash.
Then it’s still processing.
You may kind of track of it with the simple
$ sudo ls -lt /mnt/x/storagenode2/storage/trash/pmw6tvzmf2jv6giyybmmvl4o2ahqlaldsaeha4yx74n5aaaaaaaa/ | head
total 0
drwx------ 1 root root 4096 Sep 24 21:15 2024-09-24
drwx------ 1 root root 4096 Sep 22 20:39 2024-09-22
drwx------ 1 root root 4096 Sep 20 04:58 2024-09-20
drwx------ 1 root root 4096 Sep 17 20:34 2024-09-17
Then check the interesting folder to see where is it now.
Is it normal for it to take 1 month to erase about 5 TB?
It took 7 days to delete 2.5TB on my system. find deleted the remaining 7.5TB in ~4 days.
1 Like
I always see up to 8 days worth of trash folders (so up to 9 dated directories including the current day). Like today is 2024-09-25… and I see folders going back to 2024-09-17.
Well, I’ll keep waiting a few more days to see if it reduces, thanks.
You always can use the suggested approach about manual deletion of the folders from the trash older than 7 days, and then restart the node to allow the used-space-filewalker to update the usage.
Honestly, I have a hope on this: https://review.dev.storj.io/c/storj/storj/+/14684
It wouldn’t help my nodes, though, because I do share space from the elsewhere used disks.
Today I have 4 tb less in the trash, but from a node I didn’t reboot.
1 Like
Perhaps you shouldn’t reboot them, the cleaning process could be slow on some setups and may take weeks.
1 Like
Yeah if you wanted to just “let it run” and handles it’s own trash emptying, I would not reboot, because that will trigger used space filewalkers, which will slow down the deletes.
But if you wanted to delete manually, then yes, reboot after the manual delete is done.
1 Like
After one month, my trash is empty! wow
2 Likes
that was, not-sarcstically, quite an emotional roller coaster.
Did it eventually finish deleting by “letting it do it’s thing” or did you intervene with manual deletes?
1 Like
I just reboot it after more than 20 days of doing nothing.
1 Like
Is there something I can do? Out of 20TB I still have 10.99TB of trash and it doesn’t seem to be going down any time soon. I also saw a few hundred GB of “Overused” all of a sudden recently, but that seemed to go away (I thankfully had some spare capacity on the drive).
It has been at least a few months and I still seem to have some really old trash:
/media/Data/Storj/config/storage/trash/qstuylguhrn2ozjv4h2c6xpxykd622gtgurhql2k7k75wqaaaaaa:
total 0
drwx------ 1 root root 4096 Oct 8 19:55 2024-10-08
drwx------ 1 root root 4096 Oct 6 17:59 2024-10-06
/media/Data/Storj/config/storage/trash/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa:
total 0
drwx------ 1 root root 3028 Oct 12 13:22 2024-10-12
drwx------ 1 root root 4096 Oct 10 11:32 2024-10-09
drwx------ 1 root root 4096 Oct 7 21:29 2024-10-07
/media/Data/Storj/config/storage/trash/pmw6tvzmf2jv6giyybmmvl4o2ahqlaldsaeha4yx74n5aaaaaaaa:
total 0
drwx------ 1 root root 4 Oct 6 20:26 2024-10-06
/media/Data/Storj/config/storage/trash/v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa:
total 0
drwx------ 1 root root 4096 Oct 10 12:53 2024-10-10
drwx------ 1 root root 4096 Oct 8 05:15 2024-10-08
drwx------ 1 root root 4096 Oct 6 06:40 2024-10-06
/media/Data/Storj/config/storage/trash/arej6usf33ki2kukzd5v6xgry2tdr56g45pp3aao6llsaaaaaaaa:
total 0
drwx------ 1 root root 4096 Apr 3 2021 2024-04-18
/media/Data/Storj/config/storage/trash/abforhuxbzyd35blusvrifvdwmfx4hmocsva4vmpp3rgqaaaaaaa:
total 0
drwx------ 1 root root 4096 Dec 20 2019 2024-04-18
/media/Data/Storj/config/storage/trash/6r2fgwqz3manwt4aogq343bfkh2n5vvg4ohqqgggrrunaaaaaaaa:
total 0
drwx------ 1 root root 4096 May 27 2020 2024-04-18
You have data from decommissioned satellites. Use the forget satellite command from 
to delete this data.
I am unsure if it deletes data from trash but this will show you if forget-satellite command deletes trash data of decommissioned satellites.
Do update us 
1 Like
Thanks, looks like it did something:
2024-10-12T18:34:57Z INFO Configuration loaded {"Process": "storagenode", "Location": "/app/config/config.yaml"}
2024-10-12T18:34:57Z INFO Anonymized tracing enabled {"Process": "storagenode"}
2024-10-12T18:34:57Z INFO Identity loaded. {"Process": "storagenode", "Node ID": "redacted"}
Satellite ID Status
12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB In Progress
12tRQrMTWUWwzwGh18i7Fqs67kmdhH9t6aToeiwbo5mfS2rUmo In Progress
This was in logs, I guess it didn’t find anything?:
node-1 | 2024-10-12T18:42:56Z INFO forgetsatellite:cleaner cleaning up the trash {"Process": "storagenode", "satelliteID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB"}
node-1 | 2024-10-12T18:42:56Z INFO lazyfilewalker.trash-cleanup-filewalker starting subprocess {"Process": "storagenode", "satelliteID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB"}
node-1 | 2024-10-12T18:42:56Z INFO lazyfilewalker.trash-cleanup-filewalker subprocess started {"Process": "storagenode", "satelliteID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB"}
node-1 | 2024-10-12T18:42:56Z INFO lazyfilewalker.trash-cleanup-filewalker.subprocess trash-filewalker started {"Process": "storagenode", "satelliteID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "dateBefore": "2024-10-12T18:42:56Z", "Process": "storagenode"}
node-1 | 2024-10-12T18:42:56Z INFO lazyfilewalker.trash-cleanup-filewalker.subprocess Database started {"Process": "storagenode", "satelliteID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Process": "storagenode"}
node-1 | 2024-10-12T18:42:57Z INFO lazyfilewalker.trash-cleanup-filewalker.subprocess trash-filewalker completed {"Process": "storagenode", "satelliteID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Process": "storagenode", "bytesDeleted": 0, "numKeysDeleted": 0}
node-1 | 2024-10-12T18:42:57Z INFO lazyfilewalker.trash-cleanup-filewalker subprocess finished successfully {"Process": "storagenode", "satelliteID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB"}
node-1 | 2024-10-12T18:45:20Z INFO forgetsatellite:cleaner cleaning up the trash {"Process": "storagenode", "satelliteID": "12tRQrMTWUWwzwGh18i7Fqs67kmdhH9t6aToeiwbo5mfS2rUmo"}
node-1 | 2024-10-12T18:45:20Z INFO lazyfilewalker.trash-cleanup-filewalker starting subprocess {"Process": "storagenode", "satelliteID": "12tRQrMTWUWwzwGh18i7Fqs67kmdhH9t6aToeiwbo5mfS2rUmo"}
node-1 | 2024-10-12T18:45:20Z INFO lazyfilewalker.trash-cleanup-filewalker subprocess started {"Process": "storagenode", "satelliteID": "12tRQrMTWUWwzwGh18i7Fqs67kmdhH9t6aToeiwbo5mfS2rUmo"}
node-1 | 2024-10-12T18:45:20Z INFO lazyfilewalker.trash-cleanup-filewalker.subprocess trash-filewalker started {"Process": "storagenode", "satelliteID": "12tRQrMTWUWwzwGh18i7Fqs67kmdhH9t6aToeiwbo5mfS2rUmo", "Process": "storagenode", "dateBefore": "2024-10-12T18:45:20Z"}
node-1 | 2024-10-12T18:45:20Z INFO lazyfilewalker.trash-cleanup-filewalker.subprocess Database started {"Process": "storagenode", "satelliteID": "12tRQrMTWUWwzwGh18i7Fqs67kmdhH9t6aToeiwbo5mfS2rUmo", "Process": "storagenode"}
node-1 | 2024-10-12T18:45:20Z INFO lazyfilewalker.trash-cleanup-filewalker.subprocess trash-filewalker completed {"Process": "storagenode", "satelliteID": "12tRQrMTWUWwzwGh18i7Fqs67kmdhH9t6aToeiwbo5mfS2rUmo", "Process": "storagenode", "bytesDeleted": 0, "numKeysDeleted": 0}
node-1 | 2024-10-12T18:45:20Z INFO lazyfilewalker.trash-cleanup-filewalker subprocess finished successfully {"Process": "storagenode", "satelliteID": "12tRQrMTWUWwzwGh18i7Fqs67kmdhH9t6aToeiwbo5mfS2rUmo"}
It did print “emptying trash started”/“emptying trash finished” for other satellites, but not these. 11.01TB/20TB of trash right now.
Missing out on over 50% of my storage +overuse was not a great experience. Storj does have my email why not sending me instructions what I need to do if there are manual interventions needed? I knew two satellites were gone, I had no idea I need to run some commands and I’m sure I’m not alone.
1 Like
We do not use emails for such messages, they are used only by satellites to notify if your node is going offline, suspended or disqualified and after a recover.
We publish such kind of announcements here, on the forum. Please subscribe to at least Announcements, I would also recommend to subscribe to STORJLINGS and Node Operators.
Please wait until it would finish. I hope you submitted it with a --force flag to also delete their data.
I just felt like this was important enough event to use email for this, but I understand you have reasons not to.
I didn’t originally, retried with --force. It has been almost a day since, things didn’t improve a bit, trash is currently at 11.07TB/20TB. Zero observable progress, zero indication from monitoring that something different is happening comparing to usual operation.