Please, replace the path to yours and check
If you are on Linux, then use this command:
docker exec -it storagenode find /app/config/storage/trash -mtime 8 -type f -print
docker exec -it storagenode find /app/config/storage/trash -mtime 8 -type f -print
does not give response…
Which is to be expected as the data is not yet 7 days old.
I do not mind that there is a trash folder or that there is data in it.
What i find interesting is that there is still so much data that gets moved by the garbage collector.
One assumes that when the node is online almost 24/7 year round you should not miss that much delete messages.
every month it is higher than lets say 10GB.
If it is really because of zombie segments, the node should be finished deleting them at some point.
When you have a couple of consecutive months of consistently above 50GB trash, you should run out of zombie segments after a couple of months.
But this has been going on for a couple of months and i do not think it will end soon.
The zombie segment cleaning was a one time thing. My trash has been at most a few gigs for a long time since now. Something does seem off if you have that much. Maybe a bottleneck is causing your node to not process deletes right away for some reason?
Its much better now.
1 Like
The difference is still interesting though.
Larger node, less trash. And this level is fairly consistent on my node.
Raid 6, with all CMR drives. No downtime since 1.6.4 patch, 388h 9m uptime since.
Yeah I have no idea cause this is my other node exactly the same setup and hard drive USB3.0 On rpi4
yeah… im running it on a Rock64 on USB3.
That was a recommended board back in the day.
Do not have an overly high CPU usage of IOWait.
docker exec -it storagenode find /app/config/storage/trash -mtime 8 -type f -print
Ran the above and gave no result also. So all the trash I have are within these 7 days.
This is not a true statement and is an assumption. There are SNO’s with high speed hardware and NVME SSD’s for cache etc who also have large trash folders. I reported this previous in a different thread. So your statement about the SNO having slow hardware is simply not true.
Then what is the cause?
The root cause has not been identified, which is why SNO’s are still complaining about having large volume of trash. For some it’s also impacting any further ingress because free space is not available due to trash.
- Initially, the assumption was made that steffan b satellite didn’t follow the correct delete method, instead moved all data to trash. This has put a large strain on SNO’s hardware having to deal with the trash. StorJ advised this is a one off occurrence, however no official statement on why the large volume of data to be deleted (including zombie segments) was not deleted correctly rather than moved to trash.
was at 15gb a couple of days ago… but only lasted for a day or two…
seems to be steady around 6gb, no clue why…
ofc like @BrightSilence i got ssd caching, and like twice the raw disk iops or better
@kevink and @Pentium100 - how is your trash sizes looking … pentium100 you also got a slog right?
just kinda figured it might be interesting to see if nodes with ssd write caching had an issue…
could it be related to node age… i mean the older data gets the larger the odds might be for it being deleted… if one has data with a high rate of deletion, then trash amounts might be much higher… because it represents the amount of data deleted during the last week…
ofc that would mean a node being affected by this would have a measurable lower growth than it’s ingress numbers might indicate… which i think i have been seeing when doing the bandwidth comparison tread…
so that is something we can test with like GB accuracy
Currently I got 580MB but the trash size is sometimes spiking and then it has multiple GB too even though my node certainly isn’t slow. Wanted to wait until it happens again so I could investigate what’s going on but the trash kept below 1GB since.
But restarts of the node sometimes cause many GB to be moved to the trash only to be moved back to the blobs a few minutes later.
1 Like
Yes, I use a SLOG.
This is the trash directory sizes on the node:
896M 6r2fgwqz3manwt4aogq343bfkh2n5vvg4ohqqgggrrunaaaaaaaa
319M abforhuxbzyd35blusvrifvdwmfx4hmocsva4vmpp3rgqaaaaaaa
655M pmw6tvzmf2jv6giyybmmvl4o2ahqlaldsaeha4yx74n5aaaaaaaa
95M qstuylguhrn2ozjv4h2c6xpxykd622gtgurhql2k7k75wqaaaaaa
252M ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa
2.1G v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa
4.3G total
1 Like
My trash just went down to 0B 
2 Likes
has gone down, and immediately has been filled again.
Thats the point some other SNO’s were making…
again, every month there is this spike in trash data… weird.
153G ./6r2fgwqz3manwt4aogq343bfkh2n5vvg4ohqqgggrrunaaaaaaaa
5.4G ./pmw6tvzmf2jv6giyybmmvl4o2ahqlaldsaeha4yx74n5aaaaaaaa
7.0G ./qstuylguhrn2ozjv4h2c6xpxykd622gtgurhql2k7k75wqaaaaaa
7.4G ./v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa
8.9G ./ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa
18M ./abforhuxbzyd35blusvrifvdwmfx4hmocsva4vmpp3rgqaaaaaaa
182G .
Trash directory size