No bloom filters from US1

My last bloom filter for the AP1 satellite was back in 2024… But my node did go offline at some point so maybe I missed it?

2024-12-30T00:04:11Z    INFO    retain  Moved pieces to trash during retain     {"Process": "storagenode", "cachePath": "config/retain", "Deleted pieces": 10886, "Failed to delete": 0, "Pieces failed to read": 0, "Pieces count": 307407, "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Duration": "4m36.147286569s", "Retain Status": "enabled"}
1 Like

Yes, as far as I know. However, there is not a lot of deletions for some reason. The last BF was at Jan 28 on my nodes.

2025-01-28T02:07:03Z    INFO    piecestore      New bloomfilter is received     {"Process": "storagenode", "satellite": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "creation": "2025-01-27T14:57:30Z"}
2025-01-28T07:39:13Z    INFO    piecestore      New bloomfilter is received     {"Process": "storagenode", "satellite": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "creation": "2025-01-26T01:57:58Z"}
2025-01-28T20:34:04Z    INFO    piecestore      New bloomfilter is received     {"Process": "storagenode", "satellite": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "creation": "2025-01-24T20:26:19Z"}

Interesting. My node with hashstore enabled (AP1 is the only satellite on that node) does not have that text in the log file, But my other nodes do. This node has data saved in both the piecestore and hashstore because it is not doing active migration, just passive migration and new data writes.

I think each storage backend uses own wordings for typical for that backend’s chores.
I believe that you need to search for “compact”. I think they are mutually exclusive, so piecestore BF processing likely disabled due to migration to a hashstore backend.

And also I think we never tested the slow migration + BF processing. On Select they are separate processes running in the different time.

1 Like

Still no bloom filters from AP1… :roll_eyes:

1 Like

AP1 only handles like thirteen 1’s and around seven 0’s. Not much to delete…

2 Likes

Yeah, AP1 doesn’t provide a huge amount of ingress, but interestingly, it’s still more than SLC—and SLC does send out bloom filters to my nodes. Meanwhile, I haven’t received a BF from AP1 in over a month now. So it’s a bit odd that AP1 isn’t sending any.

AP1 might be a small satellite but percentage wise the amount of uncollected and unpaid trash must be huge now. What ist the reason for AP1 not sending bloom filters @Alexey ?

They are dealing with an out of memory error condition when generating the BF. They are working on a resolution.

6 Likes

Can’t this BF be segmented on time periods?
Like generate a partial BF only for pieces older than 3 years; than generate one for pieces 2 to 3 years old; one for 1 to 2 years old; one for pieces under 1 year.

More likely, BF for subdirectories 22…2z, then another for 3a…3z, etc. Would require node code changes though.

But, this process is only bound by RAM because the implementation is deliberately simple. There is a proof-of-concept piece of code that does it with rather insignificant RAM usage.

2 Likes

The satellite knows about those subdirectories? Or is just a local thing?

1 Like

Subdirectory name is just the first 10 bits of piece ID.

There he is


2025-02-11T04:27:21Z

INFO

retain Prepared to run a Retain request.

Process": "storagenode", "cachePath": "config/retain"

{"

"Created Before": "2025-02-

05T12:58:27Z"

• "Filter Size": 4859236, "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhd

xYxKicsiFmxгsLZHeLUtdps3S"}
1 Like

I guess you mean satellite 121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6?

I can confirm getting BFs from AP1 again:

2025-02-11T05:26:16Z    INFO    retain  Prepared to run a Retain request.       {"Process": "storagenode", "cachePath": "config/retain", "Created Before": "2025-02-07T22:57:52Z", "Filter Size": 280174, "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6"}

And they sent a second bloom filter just 11 hours later. Looking good. The first bloom filter resulted in 17.49GB of trash. The second brought it up to 19.93GB.

2025-02-11T02:48:59Z    INFO    piecestore      New bloomfilter is received     {"Process": "storagenode", "satellite": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "creation": "2025-02-10T22:57:52Z"}
2025-02-11T13:55:47Z    INFO    piecestore      New bloomfilter is received     {"Process": "storagenode", "satellite": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "creation": "2025-02-11T02:28:59Z"}