No bloom filters from US1

Node Operators
,
1

I wonder what is going on with US1 satellite? Nothing in the trash folder and uncollected trash is piling up… :thinking:

4 Likes
3

Our devs are working on resolving some issues and expect to issue a new bloom filter soon ™

10 Likes
4

None of my nodes has any trash for the US-satellite. Is there an issue with garbage collection?

The ./trash/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/ directory is completely empty on all nodes.

5
3 Likes
6

Issues seems to be solved, got my first bloom filter after two weeks on one node.

Stats:
123GiB gone to trash, ~3.6TB data before moving → ~3.4% deleted.

If i calculate for my ~85TB it will be arround 2.9TB trash, that’s what i have usually in trash for us1, i was expecting much worse.

3 Likes
7

There he is
2024-11-28T14:17:41Z INFO retain Prepared to run a Retain request. {“Process”: “storagenode”, “cachePath”: “config/retain”, “Created Before”: “2024-11-21T17:59:59Z”, “Filter Size”: 5235703, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”}

3 Likes
8

It looks like we might be running into a BF hiccup again. Unfortunately, my last BF from US1 was a week ago across all my nodes too. Hopefully, it’s just a temporary hiccup, and things will get back on track soon. Fingers crossed for a quick fix! :crossed_fingers:

1 Like
9

I can see I have received one on January 10th.

1 Like
10

It seems like I still haven’t received any BFs from US1, and it’s been over a week since I got any from AP1 and SLC either. At the moment, only EU1 is sending BFs regularly. Is this issue already noticed by the team?

3 Likes
11

Yes, of course. It was on purpose. We are migrated the first part of US1 to a new database backend, so, I would expect to receive a BF tomorrow or so.

Perhaps AP1 and SLC have not much used on your node and there is almost no deletions. My nodes receives BF from them much less frequently too.

1 Like
12

Thanks for clarifying. I guess I’ve missed that news.

13

We didn’t publish it in exact details, however, you may track our status.storj.io

Please note, there will be a Part II.

14

Aw man. I thought some usage was finally adding up! You’re telling me I’ve been piling up trash and the Satellite just hasn’t told me about it yet?

Bah! [shakes fist at sky]

7 Likes
15

I do not know even for my nodes. They also missed a report about Avg Usage from US1, so we will see.

16

This!
They all say “use Cloud! Use Cloud!” But they are all using the wrong Clouds!

2 Likes
17

Ask yourself: Why is this?

2 Likes
18

So at the moment we don’t have any reliable data?

Storj stat wrong data due to US1 ?
Node used space wrong data due to US1 ?
Monthly average wrong due to US1 ?
Therefore monthly estimates are also wrong ?

And we have to expect massive deletions when US1 migration is completed?

19

I have no idea. We will see, I suppose.

1 Like
20

Yes, for the first time in a long time, the data is looking promising. It would be extremely disappointing to see deletions on a terabyte scale again.

1 Like
21

I cannot find a garbage collector messages in my logs for any satellite for some reason, investigating.

Hm, found two for the whole month (my logs are rotated once a month):

$ grep -iE "\s(gc|garbage|bf|bloom)" /mnt/x/storagenode2/storagenode.log
2025-01-23T00:50:46Z    INFO    piecestore      New bloomfilter is received     {"Process": "storagenode", "satellite": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "creation": "2025-01-22T14:57:51Z"}
2025-01-23T19:25:41Z    INFO    piecestore      New bloomfilter is received     {"Process": "storagenode", "satellite": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "creation": "2024-12-10T13:33:13Z"}