Look at the trash now… up to almost 5TB! 
I’m excited to see this progress.
Huge thanks to everyone who pitched in to help solve the issue—your support and insights are truly appreciated!
3 Likes
Your trashing performance is better than mine, I’m up to 1.5TB now
2 Likes
wow I lost in trash about 10 TB in one night
2 Likes
I’m actual over 6 TB of trash now! Considering I had more than 7 TB of uncollected garbage and BF leave around 10%, I’m really happy with the results.
@andrew2.hart how much uncollected garbage did you calculate to hold on your nodes?
1 Like
I think I have 5TB, only 3.5TB left to trash…
1 Like
Then you are on your way! My experience so far is GC takes time to trash that much data. Let’s wait and see…
2 Likes
We didn’t apply this feature flag on other satellites (I think), so only SLC should be affected…
However, let’s wait for Monday.
Finally the real usage and reported usage match with each other. So, light at the end of the tunnel…
Up till yesterday, there was a difference of about 9TB between reported usage and real used space.
Great work!
Any others able to confirm?
BTW: even very crappy SMR drives are working now…
4 Likes
Woke up to all my uncollected trash in the trash also
3 Likes
what is this dashboard?
Multinode dashboard.
1 Like
I got a new BF this morning and it’s getting processed right now. It takes about 15 minutes for each xx folder, so it will finish in ~250h or 10 days 
Each xx trash folder has about 70000 files with ~ 10GB in it so far. So the end result will be 70 million files with 10TB moved to trash. And that is on a 14TB node. I upgraded this node from a 3 TB disk just before the tests started. That means I got paid 1 month for the additional space only. After that it mostly held uncollected garbage.
2 Likes
Great clean up, also a great reminder to the only use you have principle. But there’s still a future ahead with more tests and probably also more user data.
2 Likes
Seems your setup is claiming that it need to use a badger cache. It is begs you to do it… will you really refuse?!
I already enabled it some time ago. I guess the slowlyness comes from 7 nodes processing the BF at the same time on the same server. Maybe when the smaller nodes finish, the speed will increase.
After 3 months of watching my uncollected garbage just stacking up, I am finally seeing it removed to trash - half of my 5.6TB uncollected garbage has been moved in the last few hours - finally some paid data coming I hope.
Thanks to those involved in the investigation and fix
CC
4 Likes
I too just got a new bloom filter and it trashed 1.12 TB on my 3.5TB node. Great work team! I’m glad we were able to get this resolved. The satellite used graph almost exactly matches the disk used space now.
4 Likes
Nice! Having the reported Avg TB/M match Used space is good news as we’re paid as expected, just minus the amount of trash tied up for new uploads until it clears. Same result here on all nodes, including actual file system used space. The moons are aligned.
2 Likes