maybe also increase alive-fraction slightly to be more aggressive.
hashstore.compaction.alive-fraction: 0.35
hashstore.compaction.rewrite-multiple: 1
As space is cleared, increase alive-fraction and rewrite-multiple.
1 Like
Thank you for sending over the metrics.
I checked them, found the following numbers:
available_space{scope="storj_io_storj_storagenode_monitor",field="recent"} -2.43368264214e+11
used_space{scope="storj_io_storj_storagenode_monitor",field="recent"} 3.843368264214e+12
allocated_space{scope="storj_io_storj_storagenode_monitor",field="recent"} 3.6e+12
Looks like the used_space is higher than the allocated_space.
I also checked the space usage reported by hashstore:
hashstore{scope="storj_io_storj_storagenode",satellite="_121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6",field="LenLogs"} 1.71994976064e+11
hashstore{satellite="_12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S",scope="storj_io_storj_storagenode",field="LenLogs"}
hashstore{satellite="_12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs",scope="storj_io_storj_storagenode",field="LenLogs"} 8.95136642198e+11
hashstore{satellite="_1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE",scope="storj_io_storj_storagenode",field="LenLogs"} 5.4425246464e+10
(Be aware if you check this metric. There are db="s0" and db="s1" variants. The one without the db="..." is a sum)
The summary of this is:
; 1.71994976064e+11 + 2.71984537088e+12 + 8.95136642198e+11 + 5.4425246464e+10
3841402235606
Based on your screenshot, OS reports 3 940 322 660 352 used bytes (and 4 000 768 323 584 capacity).
Which means you have 98920424746 (98 GB) used space which doesn’t belong to any hashstore logs or hashstore table (old piecestore files? temp files?)
I don’t know what is your configured allocated settings, but I assume you set a high number (like 4TB). When the disk is smaller, that will be used.
In case of dedicated disk the reserved bytes are also deducted.
Which means that you have 3.7 TB available (300GB is the default reserved bytes for dedicated disk space measurement).
This explains why you didn’t receive any new data. It simply reserves some bytes.
Couldn’t see any bug in the calculation here, just unexpected reserved bytes.
But ready to debug further.
Two more comments for adjusted configuration values:
- You can also set 0 as a value of
rewrite-multiple
hashstore.compaction.rewrite-multiple: 0
It will rewrite only the hashtbl (usually 4-16GB), and only delete log files which can be full reclaimed (100% unused bytes). Can be useful
- In case of using a
storage2.monitor.reserved.bytes=100GB, I would recommend to usehashstore.compaction.rewrite-multiple: 5(The highest hasthable size is 16GB, 16 * 6 = 96GB. Which means rewriting table + 5 log files should be less than 100GB)
1 Like
How do I get these metrics?
Trash folder is 0b, temp files 0, blobs files 0.
So this 98 gb is used by hashstore.
limit is 3.6TB so it havely overused, but it takes data in.
I can confirm that this is a bug. But this is UI only.
The dedicated disk space monitor don’t calculate the UsedForPieces field which is used by the UI. (because it doesn’t know anything about pieces, it just calculates free space quickly).
Fortunately it shouldn’t affect the behavior of storagenode. The counter which reports the free space to satellite is based on TotalSpace which is calculated well.
That’s why the dedicated called experimental, not all the use-cases tested yet, just the one what we use…
We use it for all select nodes (with default 300GB reserved bytes). We don’t have UI on select storagnodes (disabled, as we also disabled bandwidth monitoring). We monitoring everything with Prometheus (in fact: Victoria Logs) and Grafana.
That’s why we didn’t notice this, yet.
But should be fixed. Thanks for the report.
3 Likes
is it alocated space in config in windows GB or decimal GB
Decimal if you use 100 GB, 1024 based (IEC) if you use 100 GiB
1 Like
So with 4TB HDD, even 3.9TB in config should be fine, as it will leave about 90GiB free, but it eat it out till 0 is free
if we account that compaction need 100GB as temp buffer. So 3.8TB should be fine also?
What is the minimum amount should be to be safe?
I expected it would work like, IF
And OS Reports less than 96GB(+16Gb) available on storage, then stop Ingress, and possibly trigger compaction - possibly more aggressive than normal - restart ingress - if space available.
for me i see
hashstore.compaction.alive-fraction: 0.60
hashstore.compaction.rewrite-multiple: 1-5
work better then default values,
for example 10 is to big, when node is almost full it will try to delete to many and get out of space.
Something is still not ok. 
With these settings and 130 GB of free space on the drive, I still got error message, but only for this sat and this hashstore. For all other sats and stores the compactions finished without any error.
hashstore.compaction.alive-fraction: 0.60
hashstore.compaction.rewrite-multiple: 2
There are many compact once started-finished log lines before this one for this sat and hashstore:
2025-11-02T10:42:24+01:00 INFO hashstore compaction computed details {"satellite": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "store": "s0", "nset": 819518, "nexist": 819518, "modifications": false, "curr logSlots": 21, "next logSlots": 21, "candidates": [170, 163, 11, 49, 221, 271], "rewrite": [221], "duration": "134.2811ms"}
2025-11-02T10:43:12+01:00 INFO hashstore records rewritten {"satellite": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "store": "s0", "records": 3234, "bytes": "653.7 MiB", "duration": "47.8330963s"}
2025-11-02T10:43:19+01:00 INFO hashstore hashtbl rewritten {"satellite": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "store": "s0", "duration": "6.8350189s", "total records": 819518, "total bytes": "193.1 GiB", "rewritten records": 3234, "rewritten bytes": "653.7 MiB", "trashed records": 0, "trashed bytes": "0 B", "restored records": 0, "restored bytes": "0 B", "expired records": 0, "expired bytes": "0 B", "reclaimed logs": 1, "reclaimed bytes": "1.0 GiB"}
2025-11-02T10:43:19+01:00 INFO hashstore compact once finished {"satellite": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "store": "s0", "duration": "54.8623751s", "completed": false}
2025-11-02T10:43:19+01:00 INFO hashstore compact once started {"satellite": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "store": "s0", "today": 20394}
2025-11-02T10:43:19+01:00 INFO hashstore compaction computed details {"satellite": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "store": "s0", "nset": 819518, "nexist": 819518, "modifications": false, "curr logSlots": 21, "next logSlots": 21, "candidates": [242, 78], "rewrite": [242], "duration": "148.4247ms"}
2025-11-02T10:44:18+01:00 INFO hashstore records rewritten {"satellite": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "store": "s0", "records": 4658, "bytes": "0.8 GiB", "duration": "59.2994527s"}
2025-11-02T10:44:28+01:00 INFO hashstore hashtbl rewritten {"satellite": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "store": "s0", "duration": "9.6203496s", "total records": 819518, "total bytes": "193.1 GiB", "rewritten records": 4658, "rewritten bytes": "0.8 GiB", "trashed records": 0, "trashed bytes": "0 B", "restored records": 0, "restored bytes": "0 B", "expired records": 0, "expired bytes": "0 B", "reclaimed logs": 1, "reclaimed bytes": "1.0 GiB"}
2025-11-02T10:44:28+01:00 INFO hashstore compact once finished {"satellite": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "store": "s0", "duration": "1m9.1163372s", "completed": false}
2025-11-02T10:44:28+01:00 INFO hashstore compact once started {"satellite": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "store": "s0", "today": 20394}
2025-11-02T10:44:28+01:00 INFO hashstore compaction computed details {"satellite": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "store": "s0", "nset": 819518, "nexist": 819518, "modifications": false, "curr logSlots": 21, "next logSlots": 21, "candidates": [42, 58, 158, 23, 25, 177], "rewrite": [42], "duration": "149.4079ms"}
2025-11-02T10:45:05+01:00 INFO hashstore compact once finished {"satellite": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "store": "s0", "duration": "37.196131s", "completed": false, "error": "hashstore: writing into compacted log: write E:\\hashstore\\12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S\\s0\\52\\log-0000000000000152-00000000: There is not enough space on the disk.", "errorVerbose": "hashstore: writing into compacted log: write E:\\hashstore\\12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S\\s0\\52\\log-0000000000000152-00000000: There is not enough space on the disk.\n\tstorj.io/storj/storagenode/hashstore.(*Store).rewriteRecord:1247\n\tstorj.io/storj/storagenode/hashstore.(*Store).compactOnce.func8:989\n\tstorj.io/storj/storagenode/hashstore.(*Store).compactOnce:1003\n\tstorj.io/storj/storagenode/hashstore.(*Store).Compact:707\n\tstorj.io/storj/storagenode/hashstore.(*DB).performPassiveCompaction:528"}
2025-11-02T10:45:05+01:00 INFO hashstore finished compaction {"satellite": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "store": "s0", "duration": "31m8.0747009s", "error": "hashstore: writing into compacted log: write E:\\hashstore\\12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S\\s0\\52\\log-0000000000000152-00000000: There is not enough space on the disk.", "errorVerbose": "hashstore: writing into compacted log: write E:\\hashstore\\12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S\\s0\\52\\log-0000000000000152-00000000: There is not enough space on the disk.\n\tstorj.io/storj/storagenode/hashstore.(*Store).rewriteRecord:1247\n\tstorj.io/storj/storagenode/hashstore.(*Store).compactOnce.func8:989\n\tstorj.io/storj/storagenode/hashstore.(*Store).compactOnce:1003\n\tstorj.io/storj/storagenode/hashstore.(*Store).Compact:707\n\tstorj.io/storj/storagenode/hashstore.(*DB).performPassiveCompaction:528", "stats": {"NumLogs":222,"LenLogs":"209.0 GiB","NumLogsTTL":14,"LenLogsTTL":"1.0 GiB","SetPercent":0.9241635205648558,"TrashPercent":0.02468625681708181,"TTLPercent":0.0036978976311172005,"Compacting":false,"Compactions":0,"Today":20394,"LastCompact":20394,"LogsRewritten":31,"DataRewritten":"24.2 GiB","DataReclaimed":"29.1 GiB","DataReclaimable":"15.8 GiB","Table":{"NumSet":819518,"LenSet":"193.1 GiB","AvgSet":253038.67228053563,"NumTrash":24055,"LenTrash":"5.2 GiB","AvgTrash":230274.82286426937,"NumTTL":6814,"LenTTL":"0.8 GiB","AvgTTL":121772.53037863223,"NumSlots":2097152,"TableSize":"128.0 MiB","Load":0.3907766342163086,"Created":20394,"Kind":0},"Compaction":{"Elapsed":0,"Remaining":0,"TotalRecords":0,"ProcessedRecords":0}}}
2025-11-02T10:45:05+01:00 ERROR hashstore compaction failed {"satellite": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "error": "hashstore: writing into compacted log: write E:\\hashstore\\12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S\\s0\\52\\log-0000000000000152-00000000: There is not enough space on the disk.", "errorVerbose": "hashstore: writing into compacted log: write E:\\hashstore\\12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S\\s0\\52\\log-0000000000000152-00000000: There is not enough space on the disk.\n\tstorj.io/storj/storagenode/hashstore.(*Store).rewriteRecord:1247\n\tstorj.io/storj/storagenode/hashstore.(*Store).compactOnce.func8:989\n\tstorj.io/storj/storagenode/hashstore.(*Store).compactOnce:1003\n\tstorj.io/storj/storagenode/hashstore.(*Store).Compact:707\n\tstorj.io/storj/storagenode/hashstore.(*DB).performPassiveCompaction:528"}
it is to low space left for compaction. how much space do you have there?
try hashstore.compaction.rewrite-multiple: 1