Single node and multinode dashboards shows different values after migration to hashstore

[Alexey]

This is a visual defect. Your node is actually full - please, check via multinode dashboard.

See

Single node and multinode dashboards shows different values after migration to hashstore

It doesn’t seem to make much of a difference, it’s not hundreds of GB. So, seems my assumption is correct, the not reclaimed space is considered as free on a single node dashboard, while a multinode dashboard is more accurate, because it uses minimum of free space in the allocation and on the disk. I also think they would match, if there is more free space on the disk, than in the allocation.

[Vadim]

It is not only a visual bug but API return the same.

[Alexey]

Of course. I mean that the displaying a wrong free space on the single node dashboard is a bug.
It just calculated as (allocated) - (used) - (trash) = (free), but the multinode dashboard uses API, which returns a minimum between (free) calculated and free on the disk. Maybe it accounts not compacted garbage as still used too.

There in the same thread I published some options which can be used to tune how often and how deep the compaction would work, of course, in an expence of IOPS.

[Vadim]

How can i push better compaction there? on 3 tb node, it is a lot of trash 0.5tb

[Vadim]

It is not only a dashboard error, node try to fill HDD till last byte, where is 5gb buffer?? Also compaction cat make its work, because no space left for tmp files

[Vadim]

After node restart it started to show overused and free also, so there is some errors in space calculation during live, node dont check hdd free space at all and write it till 0 left.

After restart i found in log

2025-10-24T16:16:41+03:00	ERROR	blobscache	piecesTotal < 0	{piecesTotal: -2042624}
2025-10-24T16:16:41+03:00	ERROR	blobscache	piecesContentSize < 0	{piecesContentSize: -2042112}
2025-10-24T16:16:41+03:00	ERROR	blobscache	satPiecesTotal < 0	{satPiecesTotal: -2042624}
2025-10-24T16:16:41+03:00	ERROR	blobscache	satPiecesContentSize < 0	{satPiecesContentSize: -2042112}

is it possible that node use this numbers in calculation?

[Balage76]

Single node and multinode dashboards shows different values after migration to hashstore

OK, good. For my node, --hashstore.compaction.alive-fraction=0.65 was too heavy; the compaction run lasted longer than 12 hours, and no ingress during the compaction run, even if there was free space. While, --hashstore.compaction.alive-fraction=0.35 was too small, no significant improvement. So now --hashstore.compaction.alive-fraction=0.45. The 1st compaction run (for the biggest satellite) was 11 hours, but then up to 4 hours. The free space in the multinode dashboard increases, but there …

There is an option for compation rate, you can play with it.
I also see quite high extra space usage, around 15% of the total used space on my node.

[Vadim]

can i start compaction by hand?

[Vadim]

someone know how to convert this commands to use them in config file?
hashstore.compaction.alive-fraction=0.65

[Ambifacient]

@Balage76

I also see quite high extra space usage, around 15% of the total used space on my node.

That’s interesting, I am only seeing around 6% with the default settings averaged across nodes, ranging from 2.5% to 16%.

[alpharabbit]

# if the log file is not this alive, compact it
# hashstore.compaction.alive-fraction: 0.25

# max size of a log file
# hashstore.compaction.max-log-size: "1073741824"

# controls if we collect records and sort them and rewrite them before the hashtbl
# hashstore.compaction.ordered-rewrite: true

# power to raise the rewrite probability to. >1 means must be closer to the alive fraction to be compacted, <1 means the opposite
# hashstore.compaction.probability-power: 2

# multiple of the hashtbl to rewrite in a single compaction
# hashstore.compaction.rewrite-multiple: 10

# if set, call mlock on any mmap/mremap'd data
# hashstore.hashtbl.mlock: true

# if set, uses mmap to do reads
# hashstore.hashtbl.mmap: false

# path to store log files in (by default, it's relative to the storage directory)'
# hashstore.logs-path: hashstore

# if set, call mlock on any mmap/mremap'd data
# hashstore.memtbl.mlock: true

# if set, uses mmap to do reads
# hashstore.memtbl.mmap: false

# if set, writes to the log file and table are fsync'd to disk
# hashstore.store.sync-writes: false

# default table kind to use (hashtbl or memtbl) during NEW compations
hashstore.table-default-kind: memtbl

# path to store tables in. Can be same as LogsPath, as subdirectories are used (by default, it's relative to the storage directory)
# hashstore.table-path: hashstore

[RecklessD]

@alpharabbit

# if the log file is not this alive, compact it
# hashstore.compaction.alive-fraction: 0.25

Does this mean, that it does not compact the file until it has less that 25% data or 25% deleted or something else?

How often does a compaction run?

What triggers a compaction to run?

Can we trigger a compaction manually?

[Alexey]

No, as far as I know, but this parameter just speed it up to come earlier.

[jammerdan]

@Alexey

I mean that the displaying a wrong free space on the single node dashboard is a bug.

Is there a fix in the making? Looking up manually multiple times a day if a node is full (either via Multinode or CLI) is not fixing it.

[Alexey]

No, nobody submitted a bug on GitHub so far. But I shared this issue with the team.

[Vadim]

How can i free up space on HDD? because compaction cant start due to lack of space for temp files, hdd has only 6kb

[Alexey]

Perhaps you can move hashtables to SSD?
Because I do not see any way how else you can increase the storage there.

[Vadim]

how can i move it? …

[Alexey]

1. Stop the node
2. Move hashtables to SSD/other HDD
3. Change the path with the option

@dancekid

hashstore.table-path (if you have SSD, use that one, but monitor degradation)

4. Save the config
5. Restart the node

[Vadim]

hashstore.table-path: I:
node starts and stops

ERROR failure during run {“error”: “Failed to create storage node peer: hashstore: missing hashtbl when log files exist\n\tstorj.io/storj/storagenode/hashstore.NewStore:237\n\tstorj.io/storj/storagenode/hashstore.New:96\n\tstorj.io/storj/storagenode/piecestore.(*HashStoreBackend).getDB:252\n\tstorj.io/storj/storagenode/piecestore.NewHashStoreBackend:117\n\tstorj.io/storj/storagenode.New:604\n\tmain.cmdRun:84\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.InitBeforeExecute.func1.2:389\n\tstorj.io/common/process.InitBeforeExecute.func1:407\n\tgithub.com/spf13/cobra.(*Command).execute:985\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1117\n\tgithub.com/spf13/cobra.(*Command).Execute:1041\n\tstorj.io/common/process.ExecWithCustomOptions:115\n\tstorj.io/common/process.ExecWithCustomConfigAndLogger:80\n\tstorj.io/common/process.ExecWithCustomConfig:75\n\tstorj.io/common/process.Exec:65\n\tmain.(*service).Execute.func1:107\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:93”, “errorVerbose”: “Failed to create storage node peer: hashstore: missing hashtbl when log files exist\n\tstorj.io/storj/storagenode/hashstore.NewStore:237\n\tstorj.io/storj/storagenode/hashstore.New:96\n\tstorj.io/storj/storagenode/piecestore.(*HashStoreBackend).getDB:252\n\tstorj.io/storj/storagenode/piecestore.NewHashStoreBackend:117\n\tstorj.io/storj/storagenode.New:604\n\tmain.cmdRun:84\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.InitBeforeExecute.func1.2:389\n\tstorj.io/common/process.InitBeforeExecute.func1:407\n\tgithub.com/spf13/cobra.(*Command).execute:985\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1117\n\tgithub.com/spf13/cobra.(*Command).Execute:1041\n\tstorj.io/common/process.ExecWithCustomOptions:115\n\tstorj.io/common/process.ExecWithCustomConfigAndLogger:80\n\tstorj.io/common/process.ExecWithCustomConfig:75\n\tstorj.io/common/process.Exec:65\n\tmain.(*service).Execute.func1:107\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:93\n\tmain.cmdRun:86\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.InitBeforeExecute.func1.2:389\n\tstorj.io/common/process.InitBeforeExecute.func1:407\n\tgithub.com/spf13/cobra.(*Command).execute:985\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1117\n\tgithub.com/spf13/cobra.(*Command).Execute:1041\n\tstorj.io/common/process.ExecWithCustomOptions:115\n\tstorj.io/common/process.ExecWithCustomConfigAndLogger:80\n\tstorj.io/common/process.ExecWithCustomConfig:75\n\tstorj.io/common/process.Exec:65\n\tmain.(*service).Execute.func1:107\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:93”}
2025-10-26T09:18:34+02:00 FATAL Unrecoverable error {“error”: “Failed to create storage node peer: hashstore: missing hashtbl when log files exist\n\tstorj.io/storj/storagenode/hashstore.NewStore:237\n\tstorj.io/storj/storagenode/hashstore.New:96\n\tstorj.io/storj/storagenode/piecestore.(*HashStoreBackend).getDB:252\n\tstorj.io/storj/storagenode/piecestore.NewHashStoreBackend:117\n\tstorj.io/storj/storagenode.New:604\n\tmain.cmdRun:84\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.InitBeforeExecute.func1.2:389\n\tstorj.io/common/process.InitBeforeExecute.func1:407\n\tgithub.com/spf13/cobra.(*Command).execute:985\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1117\n\tgithub.com/spf13/cobra.(*Command).Execute:1041\n\tstorj.io/common/process.ExecWithCustomOptions:115\n\tstorj.io/common/process.ExecWithCustomConfigAndLogger:80\n\tstorj.io/common/process.ExecWithCustomConfig:75\n\tstorj.io/common/process.Exec:65\n\tmain.(*service).Execute.func1:107\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:93”, “errorVerbose”: “Failed to create storage node peer: hashstore: missing hashtbl when log files exist\n\tstorj.io/storj/storagenode/hashstore.NewStore:237\n\tstorj.io/storj/storagenode/hashstore.New:96\n\tstorj.io/storj/storagenode/piecestore.(*HashStoreBackend).getDB:252\n\tstorj.io/storj/storagenode/piecestore.NewHashStoreBackend:117\n\tstorj.io/storj/storagenode.New:604\n\tmain.cmdRun:84\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.InitBeforeExecute.func1.2:389\n\tstorj.io/common/process.InitBeforeExecute.func1:407\n\tgithub.com/spf13/cobra.(*Command).execute:985\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1117\n\tgithub.com/spf13/cobra.(*Command).Execute:1041\n\tstorj.io/common/process.ExecWithCustomOptions:115\n\tstorj.io/common/process.ExecWithCustomConfigAndLogger:80\n\tstorj.io/common/process.ExecWithCustomConfig:75\n\tstorj.io/common/process.Exec:65\n\tmain.(*service).Execute.func1:107\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:93\n\tmain.cmdRun:86\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.InitBeforeExecute.func1.2:389\n\tstorj.io/common/process.InitBeforeExecute.func1:407\n\tgithub.com/spf13/cobra.(*Command).execute:985\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1117\n\tgithub.com/spf13/cobra.(*Command).Execute:1041\n\tstorj.io/common/process.ExecWithCustomOptions:115\n\tstorj.io/common/process.ExecWithCustomConfigAndLogger:80\n\tstorj.io/common/process.ExecWithCustomConfig:75\n\tstorj.io/common/process.Exec:65\n\tmain.(*service).Execute.func1:107\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:93”}