Disk usage discrepancy?

Alexey · June 19, 2026, 7:06am

Does your node receives bloom filters from the satellites? You need to search in your logs for piecestore:retain or retain they shows receiving of bloom filters.
For their processing there are should be lines

gc-filewalker (I believe only for piecestore)
for hashstore something like INFO hashstore iterated pieces for retain {"Process": "storagenode", "satellite": "121RTSDpyN...", "duration": "45.12ms", "pieces_checked": 1420551} and INFO hashstore queued pieces for trash {"Process": "storagenode", "satellite": "121RTSDpyN...", "pieces_queued": 1422, "bytes_to_reclaim": 421890021},
for the filter processed INFO retain bloom filter process completed {"Process": "storagenode", "Satellite ID": "121RTSDpyN...", "duration": "1m12s", "status": "success"}

docker logs storagenode 2>&1 | grep -E "retain|hashstore.*pieces"

For deleting from the disk: hashstore + reclaim or compaction.

docker logs storagenode 2>&1 | grep -E "hashstore.*reclaim|compaction"

mr.Viking · June 19, 2026, 10:32am

The logs show reclaim and compaction tasks running without any errors. retain was only found for us1, but I recreated the container this morning, so maybe I just need to wait a bit longer.

I also found some interesting data when comparing the folder sizes across nodes using du -sh:
ap1 - Average disk space: 161 GB, du -sh: 204G
us1 - Average disk space: 3.42TB, du -sh: 3.6T
eu1- Average disk space: 1.01 TB, du -sh: 5.3T

Also, the db folder on ap1 and eu1 has an excessive amount of .dat files—way more than any other node:

ls qstuylguhrn2ozjv4h2c6xpxykd622gtgurhql2k7k75wqaaaaaa/ | wc -l
120
ls v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa/ | wc -l
36

Alexey · June 20, 2026, 5:11am

Of course, more usage - more dat files. Please post logs when you find retain for EU1 and AP1.

mr.Viking · June 24, 2026, 7:30am

Looks like this node isn’t getting bloom filters from eu1 and ap1. Only us1 and saltlake in the logs for the past 4 days.

grep retain

2026-06-21T21:55:38Z	INFO	retain	Retain job queued	{process: storagenode, cache_path: config/retain, satellite_id: 12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S}
2026-06-21T21:55:38Z	INFO	retain	Prepared to run a Retain request.	{process: storagenode, cache_path: config/retain, created_before: 2026-06-18T01:15:10Z, filter_size: 10331848, satellite_id: 12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S}
2026-06-21T21:55:38Z	INFO	retain	Moved pieces to trash during retain	{process: storagenode, cache_path: config/retain, deleted_pieces: 0, failed_to_delete: 0, pieces_failed_to_read: 0, pieces_count: 0, satellite_id: 12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S, duration: 1.008967ms, retain_status: enabled}
2026-06-23T13:07:42Z	INFO	retain	Retain job queued	{process: storagenode, cache_path: config/retain, satellite_id: 12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S}
2026-06-23T13:07:42Z	INFO	retain	Prepared to run a Retain request.	{process: storagenode, cache_path: config/retain, created_before: 2026-06-19T16:15:32Z, filter_size: 10364092, satellite_id: 12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S}
2026-06-23T13:07:42Z	INFO	retain	Moved pieces to trash during retain	{process: storagenode, cache_path: config/retain, deleted_pieces: 0, failed_to_delete: 0, pieces_failed_to_read: 0, pieces_count: 0, satellite_id: 12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S, duration: 3.41548ms, retain_status: enabled}
2026-06-23T22:45:51Z	INFO	retain	Retain job queued	{process: storagenode, cache_path: config/retain, satellite_id: 1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE}
2026-06-23T22:45:51Z	INFO	retain	Prepared to run a Retain request.	{process: storagenode, cache_path: config/retain, created_before: 2026-06-20T16:19:54Z, filter_size: 8830, satellite_id: 1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE}
2026-06-23T22:45:51Z	INFO	retain	Moved pieces to trash during retain	{process: storagenode, cache_path: config/retain, deleted_pieces: 0, failed_to_delete: 0, pieces_failed_to_read: 0, pieces_count: 0, satellite_id: 1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE, duration: 1.570669ms, retain_status: enabled}

Alexey · June 26, 2026, 4:48pm

Weird, asked the team to verify EU1 and AP1 bf generation

Alexey · June 29, 2026, 7:52am

EU1 and AP1 sending BF (at least they were sent 2026-06-25 - EU1 and 2026-06-26 - AP1), did you receive them?

mr.Viking · June 29, 2026, 8:10am

I have a 2GB log limit set. Right now, the earliest event is 2026-06-26t00:15:14z info piecestore downloaded, and because of that, running grep for ‘retain’ only shows results for saltlake:

grep retain

2026-06-26T17:28:06Z	INFO	retain	Retain job queued	{process: storagenode, cache_path: config/retain, satellite_id: 1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE}
2026-06-26T17:28:06Z	INFO	retain	Prepared to run a Retain request.	{process: storagenode, cache_path: config/retain, created_before: 2026-06-22T19:00:10Z, filter_size: 5941, satellite_id: 1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE}
2026-06-26T17:28:06Z	INFO	retain	Moved pieces to trash during retain	{process: storagenode, cache_path: config/retain, deleted_pieces: 0, failed_to_delete: 0, pieces_failed_to_read: 0, pieces_count: 0, satellite_id: 1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE, duration: 3.383974ms, retain_status: enabled}
2026-06-28T22:13:07Z	INFO	retain	Retain job queued	{process: storagenode, cache_path: config/retain, satellite_id: 1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE}
2026-06-28T22:13:07Z	INFO	retain	Prepared to run a Retain request.	{process: storagenode, cache_path: config/retain, created_before: 2026-06-25T16:04:16Z, filter_size: 5926, satellite_id: 1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE}
2026-06-28T22:13:07Z	INFO	retain	Moved pieces to trash during retain	{process: storagenode, cache_path: config/retain, deleted_pieces: 0, failed_to_delete: 0, pieces_failed_to_read: 0, pieces_count: 0, satellite_id: 1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE, duration: 3.096507ms, retain_status: enabled}

mr.Viking · June 29, 2026, 8:17am

I’ll increase the log size, but it looks like ap1 wasn’t even there, and we can’t tell about eu1 anymore. However, the garbage and reclaim sizes are too small for such a big discrepancy.

Alexey · June 29, 2026, 8:24am

Please also show, how much reclaimable

mr.Viking · June 29, 2026, 8:28am

with --hashstore.compaction.alive-fraction=0.6

Alexey · June 30, 2026, 3:39am

Please install stbb from GitHub - elek/stbb · GitHub and run this command (replace the path to yours):

stbb hashstore stat /opt/storj/storagenode1765/hashstore/12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs/s0/meta/hashtbl-00000000000001b8

and for s1 too

mr.Viking · July 3, 2026, 7:15am

Sorry for the late reply, should the node be stopped then?

mr.Viking · July 3, 2026, 7:43am

I performed this while the node was stopped:

12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs/s0/meta/hashtbl-0000000000000039

12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs/s1/meta/hashtbl-0000000000000039

Alexey · July 3, 2026, 8:47am

Shared with developers.

alpharabbit · July 3, 2026, 8:51am

If 2.5 TiB for s0 is true there should be like 5 TiB in EU1 folder…

I guess something wrong with hashtables, use write-hashtbl tool to write new tables.

mr.Viking · July 4, 2026, 9:13am

But the EU folder is actually more than 5TB

Alexey · July 4, 2026, 10:38am

This is why you may try to use a write-hashtbl tool to rewrite your hashtables.

mr.Viking · July 8, 2026, 8:27am

Could you please provide an example of the command to run it? Also, I would like to remind you that the issue had been detected even before the hashstore migration.

Alexey · July 8, 2026, 8:39am

Perhaps with a help of hashstore and this tool we could finally fix that issue (you are the only one, who have it so far).

mr.Viking · July 8, 2026, 9:19am

In progress… What are the next steps once it finishes? Should I run the node or stbb again?