The one and only amazing operating system in the world … WINDOWS. 
Which command gave you that result ?
1 Like
Ah yes, I think I have read it somewhere that Windows sorts it like that.
I am on Linux and it seems that ls -U prints out the unsorted list which is the one that the processing order follows as the storagenode software does not sort the date folders which my suggestion is all about:
Deletion order doesn’t matter, only speed is. And seems it doesn’t changed, except the additional time to make it in a sorted order.
So, I will stay with my opinion, that the sort order doesn’t matter, if the system can delete almost the same amount of data for the same amount of time independently of a sort order.
If you wouldn’t constantly restart your node(s) then emptying trash would finish for each of the directories.
Stop tinkering with your nodes. Let them run and all the processes will finish eventually. I might also help not to run a node on a potato 
The amount of folders in the trash folder is not normal. My guess is that they’re all partly emptied because you restarted the node again and again before the emptying could finish. Check the amount of empty xx folders with
find /path/to/trash -type d -empty | wc -l
Maybe some of these date folders only contain empty xx folders. To delete them run
find /path/to/trash -type d -empty -delete
If your OCD wants the oldest folders gone, delete them manually.
Use what you have.
Does not matter.
Here is another one:
ls -U trash/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa
2024-06-24
2024-07-26
2024-08-14
2024-08-15
The oldest folder would be long gone if it was sorted as suggested.
It will empty in this order:
ls -U trash/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa
2024-08-15
2024-08-14
2024-06-24
2024-07-26
Out of interest, if you run
iostat -x /dev/sdb 10 (replace sdb with whatever your node is on)
is your %util constantly at 100?
90/92/97/98/96/99…
So it could be deleting for days and days at full speed, if it is like mine
Edit: since I have loads of space I am going to move the trash folders and avoid the deletes. Deleting 6TB will take too long for no benefit, for me
Can only speak for myself, but seem to have gone from uncollected garbage, to uncollected trash.
The bloom filter completed around 00:05, on the 26 August, moving 5.6TB to trash. So have been waiting for this day for a week.
This morning 2 Sept, just after midnight, loads of activity, and 15 hours later, all quiet once more. But the trash remains, according to the GUI just 100GB was removed - and still no new data arriving.
Here is the activity from ZFS.
My trash folder…
root@HouseOf:~# ls -l /mnt/BogardePool/StorJ-Node/Config/storage/trash total 18 drwx------ 8 apps apps 8 Sep 2 14:35 pmw6tvzmf2jv6giyybmmvl4o2ahqlaldsaeha4yx74n5aaaaaaaa drwx------ 6 apps apps 6 Sep 2 02:14 qstuylguhrn2ozjv4h2c6xpxykd622gtgurhql2k7k75wqaaaaaa drwx------ 6 apps apps 6 Sep 1 12:38 ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa drwx------ 3 apps apps 3 Sep 2 01:06 v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa
I just want to get to get to some paying data really…
Thanks
CC
You know the answer. If you have a problem… run used space filewalker! 
6 Likes
What does
ls -l /mnt/BogardePool/StorJ-Node/Config/storage/trash/pmw6tvzmf2jv6giyybmmvl4o2ahqlaldsaeha4yx74n5aaaaaaaa
say?
The garbage collection from 2024-08-25 on my biggest node still runs (76 xx directories remaining) and about 2 hours ago the empty trash started for that directory. It’s now very slow as the 2 processes are competing.
It says…
root@HouseOfBogarde:~# ls -l /mnt/BogardePool/StorJ-Node/Config/storage/trash/pmw6tvzmf2jv6giyybmmvl4o2ahqlaldsaeha4yx74n5aaaaaaaa
total 131
drwx------ 166 apps apps 166 Aug 26 16:59 2024-08-26
drwx------ 6 apps apps 6 Aug 29 11:06 2024-08-29
drwx------ 637 apps apps 637 Aug 31 00:23 2024-08-30
drwx------ 709 apps apps 709 Aug 31 21:29 2024-08-31
drwx------ 792 apps apps 792 Sep 1 19:08 2024-09-01
drwx------ 128 apps apps 128 Sep 2 12:51 2024-09-02
CC
Then the 2024-08-25 folder didn’t have much in it. Tomorrow the 2024-08-26 folder will be deleted.
If you want to check how much data is in it, run
du -hd 1 /mnt/BogardePool/StorJ-Node/Config/storage/trash/pmw6tvzmf2jv6giyybmmvl4o2ahqlaldsaeha4yx74n5aaaaaaaa/2024-08-26
If it takes too long because of the amount of data, Ctrl+C after the first few folders and multiply the average of these few folders with 1000 to get a rough idea of how much data all the folders have.
3 Likes
Thanks @donald.m.motsinger. So according to this there is hardly anything in the trash!
root@HouseOfBogarde:~# du -hd 1 /mnt/BogardePool/StorJ-Node/Config/storage/trash
16G /mnt/BogardePool/StorJ-Node/Config/storage/trash/pmw6tvzmf2jv6giyybmmvl4o2ahqlaldsaeha4yx74n5aaaaaaaa
8.2G /mnt/BogardePool/StorJ-Node/Config/storage/trash/qstuylguhrn2ozjv4h2c6xpxykd622gtgurhql2k7k75wqaaaaaa
95G /mnt/BogardePool/StorJ-Node/Config/storage/trash/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa
2.9G /mnt/BogardePool/StorJ-Node/Config/storage/trash/v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa
121G /mnt/BogardePool/StorJ-Node/Config/storage/trash
Node is apparently full, GUI reports 5.6TB as trash, and not accepting new data. I know were often asked not to restart nodes, thats its not needed, but not sure what else to try.
Thanks
CC
I totally lost the overview of the Trash stuff 
- Is it safe to say files older than 7 Days can be deleted?
- /data/storage/trash/
-
- Folders which are not named yyyy-mm-dd (aka two letterfodlers) are leftovers which can be removed? (on a node i still have some from 25th apr)
1 Like
AFAIK these will not be deleted as they only have 2-letter names. But you can create a date folder like 1970-01-01 and move them into it.Then the trash chore will pick it up. Make sure you don’t create any nested subfolders in a date folder though.
3 Likes
Restarted - used file walkers are running, no errors, but no change so far. No incomming data, trash been sitting there for over a week and a DU on the trash folder is just 2% of the trash the GUI is showing. Lets see what happens after the used-space filewalker completes.
Getting loads of these in the logs…
2024-09-02T16:11:11Z WARN collector unable to delete piece {"Process": "storagenode", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Piece ID": "WXYEJGK4MCPP6S5SIR7CQVNVBVR44KVXMFHTFIHVF2NBESB2RSSA", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist\n\tstorj.io/storj/storagenode/blobstore/filestore.(*blobStore).Stat:126\n\tstorj.io/storj/storagenode/pieces.(*BlobsUsageCache).pieceSizes:340\n\tstorj.io/storj/storagenode/pieces.(*BlobsUsageCache).DeleteWithStorageFormat:320\n\tstorj.io/storj/storagenode/pieces.(*Store).DeleteSkipV0:362\n\tstorj.io/storj/storagenode/collector.(*Service).Collect:112\n\tstorj.io/storj/storagenode/collector.(*Service).Run.func1:68\n\tstorj.io/common/sync2.(*Cycle).Run:99\n\tstorj.io/storj/storagenode/collector.(*Service).Run:64\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:51\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78"}
In my 4th month with a full node and 70% unpaid and I am about to call it a day on StorJ. Cant say I have not tried - storage / electricity has to be paid for - this is too long with no payment coming in.
CC
In my experience, emptying the trash can take quite a while—sometimes more than a day for multiple TBs. If you’re in a hurry, you might consider manually deleting the folder and then running the used space filewalker. It could be faster, and I expect it to be fairly safe. But I guess it’s best to let the software handle it as it’s supposed to.
As I said - the trash folder had nothing hardly anything in it. So a restart, forcing used-space file walker - has worked ( as I think was suggested by others)
Its just finished, I now have my 5.6TB back after 3 months
CC
4 Likes
With these symptoms I’m afraid that @agente is right
You need to enable the scan on startup if you disabled it (it’s enabled by default) and restart the node. Make sure that all used-space-filewalkers are successfully finished for all trusted satellites.
And you already did it.