Beside the prefix db, is there any other one which can be deleted without any issue?
The multinode dashboard usually shows a free space on the disk, not in the allocation unlike the single node dashboard.
It was this bug:
However, it’s was fixed a long time ago. Did you update it to the latest version?
Дело не в баге, а в том, что нода перестает принимать трафик тогда, когда мультинод панель рапортует об остатке 5 GB свободного места. При этом локальная панель будет показывать что доступно ещё 100GB.
Спутникам видимо уходит таже информация, что передается по api мультинод панели, поэтому её показания и остановка трафика совпадают.
Начались эти расхождения после миграции в новый тип хранилища.
Yes until the overusage would not gone. Or you may increase an allocation to include this overusage.
Could you please try to use a multinode and check, what’s it showing?
I shared this topic with the team though.
Can you measure these files?
I think the single-node dashboard counts uncompacted garbage as free space in the allocation (allocated minus used and trash, and overused as the sum of used and trash minus allocated), but the multi-node dashboard shows the minimum of the two - free space on disk and free space in the allocation.
My node would have the same issue. Because a few non-zero pieces were remaining with migration errors, unfortunately, active migration was disabled again. Then storage/used_space_per_prefix.db was removed and the storagenode was restarted several times, but no improvements. The daily ingress in these weeks is very low now because of the small free space as the multi-node dashboard. 15% loss due to non-accounting of hashtable files is a relatively big issue. How to run the compaction process more frequently?
In the single-node dashboard:
In the multi-node dashboard:
Real usage and the log:
$ df --si .
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/lc4t.2-sn 4.0T 4.0T 49G 99% /mnt/sn
$ sudo du --si -d 2
8.8M ./orders/unsent
1.4G ./orders/archive
1.4G ./orders
130M ./bin
4.1k ./retain
62k ./storage/trash
4.0T ./storage/hashstore
2.1G ./storage/blobs
4.1k ./storage/temp
1.1M ./storage/piece_expirations
4.0T ./storage
109M ./bin.bak
4.0T .
$ sudo docker compose logs sn | grep 'finished compaction'
sn | 2025-10-14T10:06:35Z INFO hashstore finished compaction {"Process": "storagenode", "satellite": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "store": "s0", "duration": "19m54.709136766s", "stats": {"NumLogs":1276,"LenLogs":"1.2 TiB","NumLogsTTL":21,"LenLogsTTL":"4.0 GiB","SetPercent":0.8423822059645659,"TrashPercent":0.0157085459710036,"TTLPercent":0.0031560252403736596,"Compacting":false,"Compactions":0,"Today":20375,"LastCompact":20375,"LogsRewritten":9,"DataRewritten":"5.5 GiB","DataReclaimed":"8.3 GiB","DataReclaimable":"198.5 GiB","Table":{"NumSet":4793859,"LenSet":"1.0 TiB","AvgSet":237564.78173262917,"NumTrash":131875,"LenTrash":"19.8 GiB","AvgTrash":161039.214892891,"NumTTL":24580,"LenTTL":"4.0 GiB","AvgTTL":173586.8087876322,"NumSlots":16777216,"TableSize":"1.0 GiB","Load":0.2857362627983093,"Created":20375,"Kind":0},"Compaction":{"Elapsed":0,"Remaining":0,"TotalRecords":0,"ProcessedRecords":0}}}
sn | 2025-10-14T10:26:13Z INFO hashstore finished compaction {"Process": "storagenode", "satellite": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "store": "s0", "duration": "1m19.696005383s", "stats": {"NumLogs":357,"LenLogs":"332.1 GiB","NumLogsTTL":24,"LenLogsTTL":"165.6 MiB","SetPercent":0.856663742720245,"TrashPercent":0.03889065942489055,"TTLPercent":0.007656081937115525,"Compacting":false,"Compactions":0,"Today":20375,"LastCompact":20375,"LogsRewritten":3,"DataRewritten":"0.8 GiB","DataReclaimed":"1.1 GiB","DataReclaimable":"47.6 GiB","Table":{"NumSet":1602526,"LenSet":"284.5 GiB","AvgSet":190624.6550857833,"NumTrash":45052,"LenTrash":"12.9 GiB","AvgTrash":307825.7885110539,"NumTTL":2484,"LenTTL":"2.5 GiB","AvgTTL":1099078.5958132045,"NumSlots":4194304,"TableSize":"256.0 MiB","Load":0.38207197189331055,"Created":20375,"Kind":0},"Compaction":{"Elapsed":0,"Remaining":0,"TotalRecords":0,"ProcessedRecords":0}}}
sn | 2025-10-14T10:53:41Z INFO hashstore finished compaction {"Process": "storagenode", "satellite": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "store": "s0", "duration": "6.653909717s", "stats": {"NumLogs":67,"LenLogs":"54.7 GiB","NumLogsTTL":11,"LenLogsTTL":"148.3 MiB","SetPercent":0.784106607543559,"TrashPercent":0.14094159436540926,"TTLPercent":0.005242806020343196,"Compacting":false,"Compactions":0,"Today":20375,"LastCompact":20375,"LogsRewritten":2,"DataRewritten":"0 B","DataReclaimed":"1.8 MiB","DataReclaimable":"11.8 GiB","Table":{"NumSet":415699,"LenSet":"42.9 GiB","AvgSet":110868.24341651051,"NumTrash":265193,"LenTrash":"7.7 GiB","AvgTrash":31238.355039537244,"NumTTL":651,"LenTTL":"293.9 MiB","AvgTTL":473362.48233486945,"NumSlots":1048576,"TableSize":"64.0 MiB","Load":0.3964414596557617,"Created":20375,"Kind":0},"Compaction":{"Elapsed":0,"Remaining":0,"TotalRecords":0,"ProcessedRecords":0}}}
sn | 2025-10-14T12:07:47Z INFO hashstore finished compaction {"Process": "storagenode", "satellite": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "store": "s1", "duration": "2m13.037090911s", "stats": {"NumLogs":78,"LenLogs":"71.0 GiB","NumLogsTTL":7,"LenLogsTTL":"261.3 MiB","SetPercent":0.7987678102658371,"TrashPercent":0.1256065987616196,"TTLPercent":0.003829898709930142,"Compacting":false,"Compactions":0,"Today":20375,"LastCompact":20375,"LogsRewritten":3,"DataRewritten":"1.4 GiB","DataReclaimed":"2.0 GiB","DataReclaimable":"14.3 GiB","Table":{"NumSet":600410,"LenSet":"56.7 GiB","AvgSet":101377.14844189804,"NumTrash":398416,"LenTrash":"8.9 GiB","AvgTrash":24023.878237821773,"NumTTL":376,"LenTTL":"278.3 MiB","AvgTTL":776187.914893617,"NumSlots":2097152,"TableSize":"128.0 MiB","Load":0.2862977981567383,"Created":20375,"Kind":0},"Compaction":{"Elapsed":0,"Remaining":0,"TotalRecords":0,"ProcessedRecords":0}}}
sn | 2025-10-14T17:12:13Z INFO hashstore finished compaction {"Process": "storagenode", "satellite": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "store": "s1", "duration": "5m16.236611964s", "stats": {"NumLogs":496,"LenLogs":"467.5 GiB","NumLogsTTL":29,"LenLogsTTL":"534.2 MiB","SetPercent":0.8614615050572666,"TrashPercent":0.06317335483841352,"TTLPercent":0.006919658479376321,"Compacting":false,"Compactions":0,"Today":20375,"LastCompact":20375,"LogsRewritten":3,"DataRewritten":"1.7 GiB","DataReclaimed":"2.0 GiB","DataReclaimable":"64.8 GiB","Table":{"NumSet":2162064,"LenSet":"402.8 GiB","AvgSet":200027.54689222891,"NumTrash":85161,"LenTrash":"29.5 GiB","AvgTrash":372405.1405690398,"NumTTL":3296,"LenTTL":"3.2 GiB","AvgTTL":1053949.8252427184,"NumSlots":8388608,"TableSize":"512.0 MiB","Load":0.2577381134033203,"Created":20375,"Kind":0},"Compaction":{"Elapsed":0,"Remaining":0,"TotalRecords":0,"ProcessedRecords":0}}}
sn | 2025-10-15T10:13:41Z INFO hashstore finished compaction {"Process": "storagenode", "satellite": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "store": "s1", "duration": "24m51.771701733s", "stats": {"NumLogs":1467,"LenLogs":"1.4 TiB","NumLogsTTL":22,"LenLogsTTL":"6.1 GiB","SetPercent":0.8438551688492478,"TrashPercent":0.029552986441314648,"TTLPercent":0.003915283587831826,"Compacting":false,"Compactions":0,"Today":20376,"LastCompact":20376,"LogsRewritten":16,"DataRewritten":"6.2 GiB","DataReclaimed":"12.0 GiB","DataReclaimable":"226.6 GiB","Table":{"NumSet":5394185,"LenSet":"1.2 TiB","AvgSet":243783.22689377543,"NumTrash":181863,"LenTrash":"42.9 GiB","AvgTrash":253232.1061898242,"NumTTL":43032,"LenTTL":"5.7 GiB","AvgTTL":141786.02793270125,"NumSlots":16777216,"TableSize":"1.0 GiB","Load":0.3215184807777405,"Created":20376,"Kind":0},"Compaction":{"Elapsed":0,"Remaining":0,"TotalRecords":0,"ProcessedRecords":0}}}
sn | 2025-10-15T13:01:26Z INFO hashstore finished compaction {"Process": "storagenode", "satellite": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "store": "s1", "duration": "47.066889171s", "stats": {"NumLogs":77,"LenLogs":"70.7 GiB","NumLogsTTL":6,"LenLogsTTL":"379.8 MiB","SetPercent":0.8018428028720366,"TrashPercent":0.12609076387048776,"TTLPercent":0.003840712376594924,"Compacting":false,"Compactions":1,"Today":20376,"LastCompact":20376,"LogsRewritten":5,"DataRewritten":"2.2 GiB","DataReclaimed":"3.0 GiB","DataReclaimable":"14.0 GiB","Table":{"NumSet":600409,"LenSet":"56.7 GiB","AvgSet":101376.81789580102,"NumTrash":398416,"LenTrash":"8.9 GiB","AvgTrash":24023.878237821773,"NumTTL":375,"LenTTL":"278.0 MiB","AvgTTL":777458.3466666667,"NumSlots":2097152,"TableSize":"128.0 MiB","Load":0.2862973213195801,"Created":20376,"Kind":0},"Compaction":{"Elapsed":0,"Remaining":0,"TotalRecords":0,"ProcessedRecords":0}}}
sn | 2025-10-15T14:07:05Z INFO hashstore finished compaction {"Process": "storagenode", "satellite": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "store": "s0", "duration": "21m15.941708848s", "stats": {"NumLogs":1273,"LenLogs":"1.2 TiB","NumLogsTTL":21,"LenLogsTTL":"3.5 GiB","SetPercent":0.8449455994320159,"TrashPercent":0.01564268887257689,"TTLPercent":0.0027199315172603076,"Compacting":false,"Compactions":1,"Today":20376,"LastCompact":20376,"LogsRewritten":24,"DataRewritten":"13.8 GiB","DataReclaimed":"21.1 GiB","DataReclaimable":"194.7 GiB","Table":{"NumSet":4790652,"LenSet":"1.0 TiB","AvgSet":237800.40367741176,"NumTrash":123343,"LenTrash":"19.6 GiB","AvgTrash":170991.83123484917,"NumTTL":24175,"LenTTL":"3.4 GiB","AvgTTL":151694.54022750776,"NumSlots":16777216,"TableSize":"1.0 GiB","Load":0.28554511070251465,"Created":20376,"Kind":0},"Compaction":{"Elapsed":0,"Remaining":0,"TotalRecords":0,"ProcessedRecords":0}}}
sn | 2025-10-15T17:15:08Z INFO hashstore finished compaction {"Process": "storagenode", "satellite": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "store": "s1", "duration": "6m10.024067948s", "stats": {"NumLogs":494,"LenLogs":"466.2 GiB","NumLogsTTL":28,"LenLogsTTL":"337.0 MiB","SetPercent":0.8582647783937796,"TrashPercent":0.057691772382851926,"TTLPercent":0.006893520086405976,"Compacting":false,"Compactions":1,"Today":20376,"LastCompact":20376,"LogsRewritten":7,"DataRewritten":"3.6 GiB","DataReclaimed":"5.3 GiB","DataReclaimable":"66.1 GiB","Table":{"NumSet":2147645,"LenSet":"400.1 GiB","AvgSet":200039.09115333308,"NumTrash":70752,"LenTrash":"26.9 GiB","AvgTrash":408160.74898236094,"NumTTL":3286,"LenTTL":"3.2 GiB","AvgTTL":1050097.3536214242,"NumSlots":8388608,"TableSize":"512.0 MiB","Load":0.2560192346572876,"Created":20376,"Kind":0},"Compaction":{"Elapsed":0,"Remaining":0,"TotalRecords":0,"ProcessedRecords":0}}}
sn | 2025-10-16T04:24:51Z INFO hashstore finished compaction {"Process": "storagenode", "satellite": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "store": "s0", "duration": "1m47.04561691s", "stats": {"NumLogs":362,"LenLogs":"338.0 GiB","NumLogsTTL":24,"LenLogsTTL":"168.1 MiB","SetPercent":0.8541697396987857,"TrashPercent":0.03274392689925647,"TTLPercent":0.007478848685927101,"Compacting":false,"Compactions":1,"Today":20377,"LastCompact":20377,"LogsRewritten":6,"DataRewritten":"1.7 GiB","DataReclaimed":"2.1 GiB","DataReclaimable":"49.3 GiB","Table":{"NumSet":1609633,"LenSet":"288.7 GiB","AvgSet":192574.67735564566,"NumTrash":35256,"LenTrash":"11.1 GiB","AvgTrash":337038.55865668255,"NumTTL":2495,"LenTTL":"2.5 GiB","AvgTTL":1087792.5018036072,"NumSlots":4194304,"TableSize":"256.0 MiB","Load":0.38376641273498535,"Created":20377,"Kind":0},"Compaction":{"Elapsed":0,"Remaining":0,"TotalRecords":0,"ProcessedRecords":0}}}
Hello @shka,
Welcome back!
Are you sure that hashtable files are so big? Please measure them.
./storage/hashstore had 4.0T by du --si as above. Or should I measure it by the other method?
Because of the less free space, I restarted the storagenode with --hashstore.compaction.alive-fraction=0.65. Let’s see in a few days.
There are two components - logs (the data) and hashtables - indexes to that data in the logs to locate pieces.
The folder hashstore contains both. So, seems, you means something other, not hashtables, right?
Likely a not reclaimed free space, then it’s match my assumptions.
The difference is a simple math: (allocated space) - (used space) - (trash) = (free space), which matches what’s showed by the single node dashboard.
The multinode dashboard seems using an API, thus the math is different - it looks like it select a minimum between free space in the allocation and free space on the disk. Since API is used to notify the satellite about that the node has only 5GB of free space which is meaning a full node.
How about this? The logs+hashtables in storage/hashstore is 3.9TB now, but hashtables only (hashtbl-* in storage/hashstore/*/*/meta) are less than 5GB.
$ sudo du --si -d 2 ./storage/hashstore
499G ./storage/hashstore/12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs/s1
367G ./storage/hashstore/12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs/s0
865G ./storage/hashstore/12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs
8.0M ./storage/hashstore/meta
76G ./storage/hashstore/121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6/s1
60G ./storage/hashstore/121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6/s0
135G ./storage/hashstore/121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6
1.6T ./storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s1
1.4T ./storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s0
2.9T ./storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S
3.9T ./storage/hashstore
$ sudo du --si ./storage/hashstore/*/*/meta
135M ./storage/hashstore/121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6/s0/meta
135M ./storage/hashstore/121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6/s1/meta
1.1G ./storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s0/meta
2.2G ./storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s1/meta
269M ./storage/hashstore/12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs/s0/meta
537M ./storage/hashstore/12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs/s1/meta
By the way, with --hashstore.compaction.alive-fraction=0.65, new compaction began. It seems compactions for two satellites are running now (thus no finished compaction log message yet), and one runs for more than three hours. But about 40GB of storage space was released, and the ingress comes again. I’ll see more longer…
Filesystem is full.
Node Size should be 3.8TB if disk/partition is 4TB.
It doesn’t seem to make much of a difference, it’s not hundreds of GB. So, seems my assumption is correct, the not reclaimed space is considered as free on a single node dashboard, while a multinode dashboard is more accurate, because it uses minimum of free space in the allocation and on the disk. I also think they would match, if there is more free space on the disk, than in the allocation.
As soon as an overusage was gone. If you would have a multinode dashboard, likely your node is freed up some space.
OK, good.
For my node, --hashstore.compaction.alive-fraction=0.65 was too heavy; the compaction run lasted longer than 12 hours, and no ingress during the compaction run, even if there was free space. While, --hashstore.compaction.alive-fraction=0.35 was too small, no significant improvement.
So now --hashstore.compaction.alive-fraction=0.45. The 1st compaction run (for the biggest satellite) was 11 hours, but then up to 4 hours. The free space in the multinode dashboard increases, but there is 300GB difference with the single-node dashboard still. I’ll continue to watch.
after hashstore i see some nodes have problems with accounting space
in reality node is full, has 12kb space. it is 3 TB hdd
version is 1.139.6
how can i push it to free up trash?