All, just wanted to post something here. We recognize there is a concern with the restoration from trash that is happening. There is an unknown error in the bloom filter generation process that is causing pieces to incorrectly be deleted. We are working on root causing the issue so that we can resolve it but in the meantime out of an abundance of caution we have been restoring data from trash to preserve pieces.
11 Likes
they are deleted friday-yesterday again, will they be restored again?
So my assumption was correct.
I don’t get all the technicalities of the bloom filters, but aren’t they suppose to not give false negatives?
1 Like
they try also make them more accurate for false positive, so errors happen.
1 Like
I am wondering how they know if the wrong pieces were deleted?
I’m wondering why we have trash again if there is an unknown error they are still looking for. Did they switch to a known good version of the BF generator for this set of trash?
1 Like
If pieces are not in blobs but in trash when customers try to download… 
1 Like
I believe in order to test if it works, they must let the retain process to finish trashing on all nodes. Than, if it was bad, they will restore it in 7 days.
1 Like
I believe it is more than this.
I think the indicator will be that lot of failed audits, or audits can also watch trash and report.
Why is this garbage not paid for SNO? Why does taking care of customers come at the expense of node operators’ income?
For example, I already have ~35 TB of unpaid garbage stored, and that’s ~$50 per month. The garbage collection problem has been going on for a LOOOOONG time!
2 Likes
They get notified if the node finds pieces in the trash folder. Somewhere here they mentioned it. I can’t find where
1 Like
1 Like
Thanks for letting us know. Keep trashing/untrashing until you get it right: the nodes will be fine.
6 Likes
2.96TB trash on my oldest node. Hoorey!
1 Like
wait i can’t keep track if I want lots of trash or not, it’s an emotional roller coaster :).
my largest amount is about 690MB of trash with 3.71TB used right now.
Last month I lost 10TB of data out of 115TB.
I keep track of nodes and subnets. Since december 2023, the no. of /24 subnets remained pretty constant. So stop blaming lots of nodes joining. That’s not it. It’s just slow addoption coupled with free tier deletion.
3 Likes
Do you generate those images automatically? And how?