Yes. I believe it starts with US1 first, tomorrow would be EU1 and on Sunday AP1.
3 Likes
Oh it looks like one of my nodes received the bloom filter as we speek. Give me a moment. Looks like it is resending the old bloom filter? Maybe you get 2 GC runs this time. I don’t know if US1 is part of it. In my nodes they all finish with 0 pieces deleted because my nodes have seen this bloom filter. But your node hasn’t seen the US1 bloom filter. I will cross my fingers.
2 Likes
Nothing so far on my end, but I’ll monitor. I did notice something else strange. It looks like Saltlake is resending an old bloom filter every time.
2024-03-07T15:11:13Z INFO retain Prepared to run a Retain request. {"process": "storagenode", "Created Before": "2024-02-17T13:02:55Z", "Filter Size": 122108, "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE"}
2024-03-14T21:01:25Z INFO retain Prepared to run a Retain request. {"process": "storagenode", "Created Before": "2024-02-17T13:02:55Z", "Filter Size": 122398, "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE"}
2024-03-28T19:28:12Z INFO retain Prepared to run a Retain request. {"process": "storagenode", "Created Before": "2024-02-17T13:02:55Z", "Filter Size": 123500, "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE"}
2024-04-04T21:04:39Z INFO retain Prepared to run a Retain request. {"process": "storagenode", "Created Before": "2024-02-17T13:02:55Z", "Filter Size": 124715, "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE"}
2024-04-11T21:05:15Z INFO retain Prepared to run a Retain request. {"process": "storagenode", "Created Before": "2024-02-17T13:02:55Z", "Filter Size": 125172, "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE"}
They all say "Created Before": "2024-02-17T13:02:55Z". Size does differ, so maybe it is recalculated with a different seed, but likely based on the same old data. Probably not a big impact, since I believe data has been pretty static on that satellite.
Oh my mistake. It seems to start with AP1. Thats the bloom filter one of my nodes has received. No resending of the old bloom filters. I was just too exited and was reading my logs wrong.
1 Like
Nice one. I will pass that to the developer team as well. Anything else while we are on it?
1 Like
I caught GC from AP1 in the tail I have open as well on one of my nodes. Will wait for US1.
Not at the moment 
I’ll let you know if I do. I did look closely at the others as well, but don’t see anything out of the ordinary except for missing bloom filters from US1 and same created before date for Saltlake. Do you see that last issue on your nodes as well?
1 Like
SLC is not on my trusted list.
This shows you are more SNO than a dev 
3 Likes
I wonder what sense it could make to ban this satellite? 
2 Likes
Those Bloom filters on BS’s node are increasing in size with each new one. What could be the cause, if it’s the same bloom filter?
I’m starting to receive the bloom filters from US1 on nodes that have not received them since Mar 1 
2024-04-12T13:52:00-07:00 INFO lazyfilewalker.gc-filewalker.subprocess gc-filewalker started {"Process": "storagenode", "satelliteID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "bloomFilterSize": 4100003, "Process": "storagenode", "createdBefore": "2024-04-03T17:59:59Z"}
Side note I’ve also observed v1.99.3 is now minimum version required, slowly inching towards the v1.100 desired changes to trash/garbage collection.
2 Likes
Super! I waited for this to turn log level to info, to finaly be able to check all those walkers.
All US1 and AP1 BFs are sent out. EU1 is under generation…
6 Likes
2 questions:
- Do all nodes in the network receive the bloom filter around the same time and start immediately processing it? Or is this somehow staggered like with the updates to be able to roll back in case of problems?
- When my node now processes the bloom filter and receives an update while doing so und restarts a a cause of that, is the bloom filter then lost and will not resume? So no more deletes in that case?
?! never saw this. Could you please show, where is it proposed?
I’m missed what’s not related, sorry?
1 Like
How you ever put your cans to the top?! 
I believe, that’s almost in the same time (but node has records, when they performed a last one, so I believe they will diverge from the main line).
- When my node now processes the bloom filter and receives an update while doing so und restarts a a cause of that, is the bloom filter then lost and will not resume? So no more deletes in that case?
Perhaps they can start in parallel. We need your logs, if that’s happen!
Am I the only one seeing trash folders like this? Doesn’t really make sense to me:
./pmw6tvzmf2jv6giyybmmvl4o2ahqlaldsaeha4yx74n5aaaaaaaa/2024-04-13/2024-04-13/2024-04-12/36