I see a successful trash cleanup for eu1 on the 15th? Do you have trash folders for eu1 older than that? Or are they empty?
My trash situation is as follows, they are the oldest folders:
EU1
Saltlake
AP1
US1
All trash folders each contain 1024 folders
What version is the node running on? If lower than v1.101, you should wait for it to be updated to v1.101; we are still rolling out v1.101.3.
1 Like
v1.99.3
20 charaptes
So what happens when a node at ver. 101 is stopped, removed, restarted and goes to 99? Can it recognise the new dir structure? Will it revert to old trash directories?
4 Likes
Are we expecting new bloom filters for US1 today? I’d like to keep an eye on that node again. I see that a few of my nodes have been updated to v1.101.3, but not the one I was monitoring yet. Another one with a big discrepancy has though. But I’m guessing this round of BF’s will still stick to the 4100003 limit?
You got these results with
?
I updated the link to the more correct PowerShell script.
Yes, It will. The current plan is running a separated bloom filter with 10Mb limit, but also keep the original BF generator for safety. (execution of 10Mb might be too long, and also it requires more memory.
When we have bigger bloom filters, we can send it to test users / volunteers, and double check the behavior. But generating the 10Mb filters is additional 1-2 weeks. (we need to setup a database restored from the backup + generation may take 5-? days…
5 Likes
Sounds good. I have 2 similarly sized nodes with large uncollected garbage amounts. I’d love to test the 10mb bloom filter on one and compare the difference. Let me know how I can help with testing.
3 Likes
Your help will be priceless, because you know - the emulation/mock is a one thing, but the real node - is completely different.
I don’t think @BrightSilence situation is alone, I’ve almost the same stat as him, indicate this is largely systemic problem, very possible solving his problem will solve for us all.
1 Like
So the rest of us will be getting typical mass-produced consumer bloom filters… but you’ll be getting the good stuff? Like artisanal hand-crafted gluten-free free-range pesticide-free organic bloom filters… made in small batches… infused with love… just for you?
Must be nice to be special 
5 Likes
Can be poisonous too. So better have one test subject, than killing all population with a new product.
3 Likes
He is ONE of a kind, kind human being. He’s one of the few who has been around for a long time. He has read the whitepaper, suggested many solutions for problems that we as SNOs faced. He also is polite and a thorough professional when passing on the errors to the dev team rather than using words like “scam” or Storj is “this and that” and how easy it is to “fix” certain things or I have N number of years experience as a “coder”.
TLDR;
2 Likes
Oops. Seems I meet all requirements?!
But I still have no idea, how is it working…
Ok.
We will wait for the next BF (v 1.101.x is a mandatory requirement).
1 Like
Awesome Alexey is the Standard.
- ISO 9001, ISO 27000 certified.
Okay, hang on. I’m not special. @elek mentioned they would want to test with test users or volunteers and I volunteered. As can anyone else. I’m not sure why you assume you can’t just offer to test as well. At this point, @elek hasn’t even responded yet and I don’t even have a clue on whether one of my nodes will be used for testing or what they need me to do to volunteer, so your comment is very premature.
You’re very kind. And I appreciate your words, but I didn’t really know which part of your message to quote as I feel like I’m just another person here in a community of great people. Thanks though.
It certainly is a larger problem and going by the previous conversations Storj is very aware of that. It impacts all nodes with large amounts of pieces on one satellite. They are working on a solution for all nodes with the larger bloom filters sent through a new method. Hopefully that will go live soon after testing.
3 Likes
Not that aware I don’t think, if they were, this is what they should already doing:
- Run several real nodes - not seeding/fake node, but using some sort of snapshot filesystem - so they could travel back in time to test exactly what going on that that particular time. I heard they also using cockroachdb - great, cockroachdb have travelsal back in time query too, they could restore a backup at a particular time and frozen it to do extended testing.
- Throwing s*** at those few nodes, eg: power down when GC is running, or at random time, when the software at most vulnerable, through time, problem will start to emerge…
Even on node I just start less than month ago, I still experience this issue, so this is certainly not a problem with large node. cc @Alexey
1 Like
All due respect, but I have no idea what you are talking about. What we are talking about is at this point a completely known issue caused by bloom filters that are too small. The behavior with only low percentages of trash being cleaned up on my node for example roughly matches the theoretical calculation posted by elek. Larger bloom filters couldn’t be sent before due to limits in DRPC. Those have been resolved for later node versions and that’s going to be tested with a larger bloom filter.
2 Likes