Disk usage discrepancy?

Ottetal · March 18, 2026, 9:48am

How much of the data is marked as being reclaimable?

mr.Viking · March 18, 2026, 9:51am

Node has not yet migrated to the hashstore. There only used and trash indicator.

Ottetal · March 18, 2026, 10:05am

All my nodes are finished converting, so I don’t have any experience with your situration - but passive migration have been enabled for some time. Can you check the size of the hashstore folder?

mr.Viking · March 18, 2026, 10:07am

I’ve blocked migration (storage2migration.suppress-central-migration=true) until I fix the size issue.
16M hashstore/

snorkel · March 18, 2026, 11:36am

Yes… I’ve done it multiple times.

mr.Viking · April 16, 2026, 5:16am

I deleted all the DBs, waited for several GC cycles, and nothing changed Any ideas what else I could try?

snorkel · April 16, 2026, 6:48am

I’m sure that you stopped and rm node and then deleted all db-es.
If all the hashstore migrations finished, just let it be for 1 week starting now. After 1 week, restart the system and wait 2 days before you take a look again. Also delete browser cookies and refresh page with F5.

Ouh, now I saw that you blocked the migration. No need for that. Let the migration finish, no matter what the dashboard tells you.
Stop the node, rm node, activate the migration on all sats, start node. Let it finish, then stop, rm, delete db-es, start.

mr.Viking · April 16, 2026, 7:26am

Thanks, I will try.
20…

mr.Viking · June 12, 2026, 8:46am

Nothing has changed:(
Unless a lot of .dat files appeared in the piece_expirations folder.

Alexey · June 12, 2026, 4:13pm

Could you please elaborate?

mr.Viking · June 14, 2026, 7:28am

I have one node where the “Average Disk Space Used this Month” differs significantly from the actual disk usage. I tried deleting DB files and waiting for several runs of the garbage collector, but the unused space still isn’t freed. I checked — data from old test satellites has been removed. Remaining: 4 folders (saltlake, ap1, us1, eu1).

Alexey · June 14, 2026, 7:56am

Did you migrate to hashstore? If not, I would like to recommend to do so:

The deleted data will not be migrated, and should be deleted by a garbage collector. But the migration may also expose any corrupted data too.

mr.Viking · June 14, 2026, 9:28am

The node was successfully migrated more than a month ago. The blobs folders were checked and empty ones were deleted. As a result, all folders were empty.

Alexey · June 15, 2026, 5:26am

Do you have any errors related to the compaction?

mr.Viking · June 15, 2026, 6:24am

It looks like it doesn’t contain any errors.
docker logs storj | grep compaction

Summary

2026-06-14T07:24:33Z INFO hashstore beginning compaction {"process": "storagenode", "satellite": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "store": "s1", "stats": {"NumLogs":28,"LenLogs":"16.9 GiB","NumLogsTTL":11,"LenLogsTTL":"230.4 MiB","SetPercent":1,"TrashPercent":0.0019879057341460325,"TTLPercent":0.012634890214711439,"Compacting":false,"Compactions":11,"Today":20618,"LastCompact":20617,"LogsRewritten":11,"DataRewritten":"0.8 GiB","DataReclaimed":"126.6 MiB","DataReclaimable":"0 B","Table":{"NumSet":7943,"LenSet":"16.9 GiB","AvgSet":2280087.3745436235,"NumTrash":30,"LenTrash":"34.3 MiB","AvgTrash":1200081.0666666667,"NumTTL":155,"LenTTL":"218.2 MiB","AvgTTL":1476304.1032258065,"NumSlots":16384,"TableSize":"500.4 KiB","Load":0.48480224609375,"Created":20617,"Kind":1},"LogsSkipped":27,"LogsMatched":0,"LogsMismatched":0,"FreeRequired":"5.9 MiB","Compaction":{"Elapsed":0,"Remaining":0,"TotalRecords":0,"ProcessedRecords":0}}} 2026-06-14T07:24:33Z INFO hashstore compaction computed details {"process": "storagenode", "satellite": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "store": "s1", "nset": 7934, "nexist": 7943, "modifications": true, "curr_log_slots": 14, "curr logSlots size": "1.0 MiB", "next_log_slots": 14, "next logSlots size": "1.0 MiB", "candidates": [41], "rewrite": [41], "duration": "29.400993ms"} 2026-06-14T07:24:35Z INFO hashstore finished compaction {"process": "storagenode", "satellite": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "store": "s1", "duration": "1.904806158s", "stats": {"NumLogs":28,"LenLogs":"16.9 GiB","NumLogsTTL":11,"LenLogsTTL":"223.8 MiB","SetPercent":0.9997420666018947,"TrashPercent":0.001346296950015518,"TTLPercent":0.012639747567050523,"Compacting":false,"Compactions":11,"Today":20618,"LastCompact":20618,"LogsRewritten":12,"DataRewritten":"0.8 GiB","DataReclaimed":"133.3 MiB","DataReclaimable":"4.5 MiB","Table":{"NumSet":7934,"LenSet":"16.9 GiB","AvgSet":2281208.046382657,"NumTrash":21,"LenTrash":"23.2 MiB","AvgTrash":1160621.7142857143,"NumTTL":155,"LenTTL":"218.2 MiB","AvgTTL":1476304.1032258065,"NumSlots":16384,"TableSize":"499.9 KiB","Load":0.4842529296875,"Created":20618,"Kind":1},"LogsSkipped":27,"LogsMatched":0,"LogsMismatched":0,"FreeRequired":"5.9 MiB","Compaction":{"Elapsed":0,"Remaining":0,"TotalRecords":0,"ProcessedRecords":0}}} 2026-06-14T11:06:01Z INFO hashstore beginning compaction {"process": "storagenode", "satellite": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "store": "s1", "stats": {"NumLogs":1577,"LenLogs":"1.5 TiB","NumLogsTTL":35,"LenLogsTTL":"3.9 GiB","SetPercent":0.9496026911763423,"TrashPercent":0.0690653846388264,"TTLPercent":0.0024943580003086562,"Compacting":false,"Compactions":11,"Today":20618,"LastCompact":20617,"LogsRewritten":108,"DataRewritten":"31.2 GiB","DataReclaimed":"55.8 GiB","DataReclaimable":"77.9 GiB","Table":{"NumSet":7756401,"LenSet":"1.4 TiB","AvgSet":203187.3490841951,"NumTrash":528071,"LenTrash":"106.8 GiB","AvgTrash":217061.6489979567,"NumTTL":36884,"LenTTL":"3.9 GiB","AvgTTL":112236.92332718793,"NumSlots":16777216,"TableSize":"473.4 MiB","Load":0.4623175263404846,"Created":20617,"Kind":1},"LogsSkipped":1397,"LogsMatched":0,"LogsMismatched":0,"FreeRequired":"5.5 GiB","Compaction":{"Elapsed":0,"Remaining":0,"TotalRecords":0,"ProcessedRecords":0}}} 2026-06-14T11:06:45Z INFO hashstore compaction computed details {"process": "storagenode", "satellite": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "store": "s1", "nset": 7744301, "nexist": 7756401, "modifications": true, "curr_log_slots": 24, "curr logSlots size": "1.0 GiB", "next_log_slots": 24, "next logSlots size": "1.0 GiB", "candidates": [1529, 1371, 1422, 1410, 1458, 1254], "rewrite": [1254, 1371, 1410, 1422, 1458, 1529], "duration": "43.994890532s"} 2026-06-14T11:12:42Z INFO hashstore finished compaction {"process": "storagenode", "satellite": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "store": "s1", "duration": "6m41.508390018s", "stats": {"NumLogs":1573,"LenLogs":"1.5 TiB","NumLogsTTL":34,"LenLogsTTL":"3.6 GiB","SetPercent":0.9509112844599037,"TrashPercent":0.06913235081544064,"TTLPercent":0.0021485498406537034,"Compacting":false,"Compactions":11,"Today":20618,"LastCompact":20618,"LogsRewritten":114,"DataRewritten":"33.0 GiB","DataReclaimed":"58.6 GiB","DataReclaimable":"75.7 GiB","Table":{"NumSet":7744301,"LenSet":"1.4 TiB","AvgSet":203417.46145223436,"NumTrash":519380,"LenTrash":"106.7 GiB","AvgTrash":220509.12025877007,"NumTTL":33475,"LenTTL":"3.3 GiB","AvgTTL":106329.87429424944,"NumSlots":16777216,"TableSize":"472.7 MiB","Load":0.4615963101387024,"Created":20618,"Kind":1},"LogsSkipped":1397,"LogsMatched":0,"LogsMismatched":0,"FreeRequired":"5.5 GiB","Compaction":{"Elapsed":0,"Remaining":0,"TotalRecords":0,"ProcessedRecords":0}}} 2026-06-14T15:16:34Z INFO hashstore beginning compaction {"process": "storagenode", "satellite": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "store": "s0", "stats": {"NumLogs":2605,"LenLogs":"2.5 TiB","NumLogsTTL":27,"LenLogsTTL":"81.8 MiB","SetPercent":1,"TrashPercent":0,"TTLPercent":0.003780949367986108,"Compacting":false,"Compactions":11,"Today":20618,"LastCompact":20617,"LogsRewritten":11,"DataRewritten":"0 B","DataReclaimed":"49.8 MiB","DataReclaimable":"0 B","Table":{"NumSet":8332388,"LenSet":"2.5 TiB","AvgSet":332237.18400055304,"NumTrash":0,"LenTrash":"0 B","AvgTrash":0,"NumTTL":10966,"LenTTL":"9.7 GiB","AvgTTL":954487.7125661134,"NumSlots":16777216,"TableSize":"508.6 MiB","Load":0.49664902687072754,"Created":20617,"Kind":1},"LogsSkipped":2549,"LogsMatched":0,"LogsMismatched":0,"FreeRequired":"6.0 GiB","Compaction":{"Elapsed":0,"Remaining":0,"TotalRecords":0,"ProcessedRecords":0}}} 2026-06-14T15:17:23Z INFO hashstore compaction computed details {"process": "storagenode", "satellite": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "store": "s0", "nset": 8332371, "nexist": 8332388, "modifications": true, "curr_log_slots": 24, "curr logSlots size": "1.0 GiB", "next_log_slots": 24, "next logSlots size": "1.0 GiB", "candidates": [1851], "rewrite": [1851], "duration": "49.284213182s"} 2026-06-14T15:19:02Z INFO hashstore finished compaction {"process": "storagenode", "satellite": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "store": "s0", "duration": "2m27.915553506s", "stats": {"NumLogs":2604,"LenLogs":"2.5 TiB","NumLogsTTL":26,"LenLogsTTL":"77.3 MiB","SetPercent":1,"TrashPercent":0,"TTLPercent":0.0037792743515056215,"Compacting":false,"Compactions":11,"Today":20618,"LastCompact":20618,"LogsRewritten":12,"DataRewritten":"0 B","DataReclaimed":"54.2 MiB","DataReclaimable":"0 B","Table":{"NumSet":8332371,"LenSet":"2.5 TiB","AvgSet":332237.3030965616,"NumTrash":0,"LenTrash":"0 B","AvgTrash":0,"NumTTL":10949,"LenTTL":"9.7 GiB","AvgTTL":955544.5852589278,"NumSlots":16777216,"TableSize":"508.6 MiB","Load":0.49664801359176636,"Created":20618,"Kind":1},"LogsSkipped":2549,"LogsMatched":0,"LogsMismatched":0,"FreeRequired":"6.0 GiB","Compaction":{"Elapsed":0,"Remaining":0,"TotalRecords":0,"ProcessedRecords":0}}} 2026-06-14T20:10:28Z INFO hashstore beginning compaction {"process": "storagenode", "satellite": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "store": "s0", "stats": {"NumLogs":88,"LenLogs":"85.2 GiB","NumLogsTTL":2,"LenLogsTTL":"2.5 MiB","SetPercent":1,"TrashPercent":0,"TTLPercent":0.005367870706207044,"Compacting":false,"Compactions":10,"Today":20618,"LastCompact":20617,"LogsRewritten":14,"DataRewritten":"0 B","DataReclaimed":"45.5 MiB","DataReclaimable":"0 B","Table":{"NumSet":343224,"LenSet":"85.2 GiB","AvgSet":266403.63685523154,"NumTrash":0,"LenTrash":"0 B","AvgTrash":0,"NumTTL":639,"LenTTL":"468.1 MiB","AvgTTL":768102.1596244131,"NumSlots":1048576,"TableSize":"21.0 MiB","Load":0.32732391357421875,"Created":20617,"Kind":1},"LogsSkipped":82,"LogsMatched":0,"LogsMismatched":0,"FreeRequired":"251.4 MiB","Compaction":{"Elapsed":0,"Remaining":0,"TotalRecords":0,"ProcessedRecords":0}}} 2026-06-14T20:10:29Z INFO hashstore compaction computed details {"process": "storagenode", "satellite": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "store": "s0", "nset": 343224, "nexist": 343224, "modifications": false, "curr_log_slots": 20, "curr logSlots size": "64.0 MiB", "next_log_slots": 20, "next logSlots size": "64.0 MiB", "candidates": [], "rewrite": [], "duration": "870.420852ms"} 2026-06-14T20:10:29Z INFO hashstore finished compaction {"process": "storagenode", "satellite": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "store": "s0", "duration": "1.15825603s", "stats": {"NumLogs":88,"LenLogs":"85.2 GiB","NumLogsTTL":2,"LenLogsTTL":"2.5 MiB","SetPercent":1,"TrashPercent":0,"TTLPercent":0.005367870706207044,"Compacting":false,"Compactions":10,"Today":20618,"LastCompact":20618,"LogsRewritten":14,"DataRewritten":"0 B","DataReclaimed":"45.5 MiB","DataReclaimable":"0 B","Table":{"NumSet":343224,"LenSet":"85.2 GiB","AvgSet":266403.63685523154,"NumTrash":0,"LenTrash":"0 B","AvgTrash":0,"NumTTL":639,"LenTTL":"468.1 MiB","AvgTTL":768102.1596244131,"NumSlots":1048576,"TableSize":"21.0 MiB","Load":0.32732391357421875,"Created":20617,"Kind":1},"LogsSkipped":82,"LogsMatched":0,"LogsMismatched":0,"FreeRequired":"251.4 MiB","Compaction":{"Elapsed":0,"Remaining":0,"TotalRecords":0,"ProcessedRecords":0}}} 2026-06-14T22:29:29Z INFO hashstore beginning compaction {"process": "storagenode", "satellite": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "store": "s1", "stats": {"NumLogs":121,"LenLogs":"113.7 GiB","NumLogsTTL":6,"LenLogsTTL":"12.7 MiB","SetPercent":1,"TrashPercent":0,"TTLPercent":0.0054519364741334375,"Compacting":false,"Compactions":12,"Today":20618,"LastCompact":20617,"LogsRewritten":11,"DataRewritten":"0 B","DataReclaimed":"23.7 MiB","DataReclaimable":"0 B","Table":{"NumSet":458806,"LenSet":"113.7 GiB","AvgSet":265988.19286583003,"NumTrash":0,"LenTrash":"0 B","AvgTrash":0,"NumTTL":943,"LenTTL":"634.5 MiB","AvgTTL":705554.460233298,"NumSlots":1048576,"TableSize":"28.0 MiB","Load":0.43755149841308594,"Created":20617,"Kind":1},"LogsSkipped":115,"LogsMatched":0,"LogsMismatched":0,"FreeRequired":"336.1 MiB","Compaction":{"Elapsed":0,"Remaining":0,"TotalRecords":0,"ProcessedRecords":0}}} 2026-06-14T22:29:30Z INFO hashstore compaction computed details {"process": "storagenode", "satellite": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "store": "s1", "nset": 458771, "nexist": 458806, "modifications": true, "curr_log_slots": 20, "curr logSlots size": "64.0 MiB", "next_log_slots": 20, "next logSlots size": "64.0 MiB", "candidates": [151], "rewrite": [151], "duration": "1.841169754s"} 2026-06-14T22:29:35Z INFO hashstore finished compaction {"process": "storagenode", "satellite": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "store": "s1", "duration": "6.689369186s", "stats": {"NumLogs":120,"LenLogs":"113.7 GiB","NumLogsTTL":5,"LenLogsTTL":"7.8 MiB","SetPercent":1,"TrashPercent":0,"TTLPercent":0.005410313382692862,"Compacting":false,"Compactions":12,"Today":20618,"LastCompact":20618,"LogsRewritten":12,"DataRewritten":"0 B","DataReclaimed":"28.6 MiB","DataReclaimable":"0 B","Table":{"NumSet":458771,"LenSet":"113.7 GiB","AvgSet":265997.3480799789,"NumTrash":0,"LenTrash":"0 B","AvgTrash":0,"NumTTL":908,"LenTTL":"629.6 MiB","AvgTTL":727126.2731277533,"NumSlots":1048576,"TableSize":"28.0 MiB","Load":0.4375181198120117,"Created":20618,"Kind":1},"LogsSkipped":115,"LogsMatched":0,"LogsMismatched":0,"FreeRequired":"336.1 MiB","Compaction":{"Elapsed":0,"Remaining":0,"TotalRecords":0,"ProcessedRecords":0}}} 2026-06-15T04:11:34Z INFO hashstore beginning compaction {"process": "storagenode", "satellite": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "store": "s0", "stats": {"NumLogs":2604,"LenLogs":"2.5 TiB","NumLogsTTL":26,"LenLogsTTL":"77.3 MiB","SetPercent":1,"TrashPercent":0,"TTLPercent":0.0037792743515056215,"Compacting":false,"Compactions":12,"Today":20619,"LastCompact":20618,"LogsRewritten":12,"DataRewritten":"0 B","DataReclaimed":"54.2 MiB","DataReclaimable":"0 B","Table":{"NumSet":8332371,"LenSet":"2.5 TiB","AvgSet":332237.3030965616,"NumTrash":0,"LenTrash":"0 B","AvgTrash":0,"NumTTL":10949,"LenTTL":"9.7 GiB","AvgTTL":955544.5852589278,"NumSlots":16777216,"TableSize":"508.6 MiB","Load":0.49664801359176636,"Created":20618,"Kind":1},"LogsSkipped":2549,"LogsMatched":0,"LogsMismatched":0,"FreeRequired":"6.0 GiB","Compaction":{"Elapsed":0,"Remaining":0,"TotalRecords":0,"ProcessedRecords":0}}} 2026-06-15T04:12:05Z INFO hashstore compaction computed details {"process": "storagenode", "satellite": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "store": "s0", "nset": 8332357, "nexist": 8332371, "modifications": true, "curr_log_slots": 24, "curr logSlots size": "1.0 GiB", "next_log_slots": 24, "next logSlots size": "1.0 GiB", "candidates": [2626], "rewrite": [2626], "duration": "31.317851867s"} 2026-06-15T04:12:40Z INFO hashstore finished compaction {"process": "storagenode", "satellite": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "store": "s0", "duration": "1m6.065893563s", "stats": {"NumLogs":2603,"LenLogs":"2.5 TiB","NumLogsTTL":25,"LenLogsTTL":"75.4 MiB","SetPercent":1,"TrashPercent":0,"TTLPercent":0.0037785391929161033,"Compacting":false,"Compactions":12,"Today":20619,"LastCompact":20619,"LogsRewritten":13,"DataRewritten":"0 B","DataReclaimed":"56.2 MiB","DataReclaimable":"0 B","Table":{"NumSet":8332357,"LenSet":"2.5 TiB","AvgSet":332237.61603877513,"NumTrash":0,"LenTrash":"0 B","AvgTrash":0,"NumTTL":10935,"LenTTL":"9.7 GiB","AvgTTL":956581.1416552354,"NumSlots":16777216,"TableSize":"508.6 MiB","Load":0.4966471791267395,"Created":20619,"Kind":1},"LogsSkipped":2549,"LogsMatched":0,"LogsMismatched":0,"FreeRequired":"6.0 GiB","Compaction":{"Elapsed":0,"Remaining":0,"TotalRecords":0,"ProcessedRecords":0}}}

Alexey · June 15, 2026, 7:18am

You may try to increase the option --hashstore.compaction.alive-fraction from the default value of 0.2 to something like 0.6, it should reclaim more space. However, I don’t think that it would help more.
What’s your last payout says about used space?

mr.Viking · June 17, 2026, 12:15pm

Alexey · June 18, 2026, 3:36am

What the size of the hashstore folder? What’s filesystem?
Please also provide a result of

df --si -T

What’s value for

      --hashstore.compaction.alive-fraction float                if the log file is not this alive, compact it
(default 0.25)

in the config.yaml it will be

hashstore.compaction.alive-fraction: 0.25

mr.Viking · June 18, 2026, 9:24am

du -sh hashstore/
9.1T    hashstore/

Filesystem     Type  Size  Used Avail Use% Mounted on
/dev/sdb1      ext4   18T   17T  1.9T  90% /media/14020

All disk space usage beyond 9.1 belongs to my personal files and is not associated with the node.

hashstore.compaction.alive-fraction is not specified anywhere.

Important clarification: this node was running with the file walker turned off for a long time. If I recall correctly, during the transition from the free tier to the trial accounts, there was a similar issue where the satellite failed to compile a large delete list. Could I be experiencing the same issue now? P.S. File walker has been enabled for 3 months already.

Node ID: 12uAaLnsWXuXvWu5piH5XBQ1BfrKgCw3iT2VGYvpYnk94MEK7QR

snorkel · June 19, 2026, 4:26am

You can ceck all the params here:
https://forum.storj.io/t/config-yaml-linux-docker-all-parameters/24850?u=snorkel