Difference in Used Space on Drive and Dashboard

Node Operators troubleshooting
, ,
1

Hello together,

after the migration to hashstore there is a significant difference in the Used Space on the Drive and on the Dashboard.

grafik
grafik605×634 23.4 KB

As the Trash is in the same log-Files as the Used you need to add both.

So 375.68 + 99.24 = 474,92 GB.

When I type du -h --max-depth=1 /mnt/STORJ-node/storage/hashstore/ it is showing the Used Space in GiB:

grafik
grafik1126×131 81.6 KB

It is there 491 GiB.

But with du -h --si --max-depth=1 /mnt/STORJ-node/storage/hashstore/ which is actually GB and not GiB it is showing:

grafik
grafik1132×132 78.1 KB

A whopping 527 GB on the disk.

So its:

Dashboard:   474,92 GB
du -h:       491,00 GiB
du -h --si:  527,00 GB

It’s on all hashstore. On a fast SSD and renewed used_space_per_prefix.db.

2

Did you try to stop and delete the container, remove the prefix database and run it back?

3

Yes, but unfortunately it doesn’t solve the issue. Still persist’s, higher usage on disk than displayed on dashboard.

4

What OS?

What Filesystem?

5

It is Linux-Debian with ext4 on the System-SSD.

6

Dashboard not known for its accuracy for current activity - doesn’t handle time zones.

If u want stats, better of with Prometheus and Grafana imo - live plus history, more detail.

1 Like
7

Do you see the same on the multinode dashboard?

8

I’m not using the multinode dashboard. Am using Prometheus with Grafana, but it is showing there the same wrong values.

9

How many compactions are passed?

10

Here the last compaction:

2025-09-17T08:05:46Z    INFO    hashstore       hashtbl rewritten       {"Process": "storagenode", "satellite": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "store": "s1", "duration": "2.083559471s", "total records": 530626, "total bytes": "137.3 GiB", "rewritten records": 4855, "rewritten bytes": "0.9 GiB", "trashed records": 0, "trashed bytes": "0 B", "restored records": 0, "restored bytes": "0 B", "expired records": 0, "expired bytes": "0 B", "reclaimed logs": 2, "reclaimed bytes": "2.0 GiB"}
2025-09-17T08:05:46Z    INFO    hashstore       compact once finished   {"Process": "storagenode", "satellite": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "store": "s1", "duration": "2.131740762s", "completed": false}
2025-09-17T08:05:46Z    INFO    hashstore       compact once started    {"Process": "storagenode", "satellite": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "store": "s1", "today": 20348}
2025-09-17T08:05:46Z    INFO    hashstore       compaction computed details     {"Process": "storagenode", "satellite": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "store": "s1", "nset": 530626, "nexist": 530626, "modifications": false, "curr logSlots": 21, "next logSlots": 21, "candidates": [92, 9], "rewrite": [9, 92], "duration": "46.035415ms"}
2025-09-17T08:05:49Z    INFO    hashstore       hashtbl rewritten       {"Process": "storagenode", "satellite": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "store": "s1", "duration": "2.003207111s", "total records": 530626, "total bytes": "137.3 GiB", "rewritten records": 2852, "rewritten bytes": "1.0 GiB", "trashed records": 0, "trashed bytes": "0 B", "restored records": 0, "restored bytes": "0 B", "expired records": 0, "expired bytes": "0 B", "reclaimed logs": 2, "reclaimed bytes": "2.0 GiB"}
2025-09-17T08:05:49Z    INFO    hashstore       compact once finished   {"Process": "storagenode", "satellite": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "store": "s1", "duration": "2.050232578s", "completed": true}
2025-09-17T08:05:49Z    INFO    hashstore       finished compaction     {"Process": "storagenode", "satellite": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "store": "s1", "duration": "12.837014514s", "stats": {"NumLogs":171,"LenLogs":"158.6 GiB","NumLogsTTL":19,"LenLogsTTL":"6.9 GiB","SetPercent":0.8659534944896888,"TrashPercent":0.17157360792870982,"TTLPercent":0.03947939869254945,"Compacting":false,"Compactions":0,"Today":20348,"LastCompact":20348,"LogsRewritten":16,"DataRewritten":"3.9 GiB","DataReclaimed":"11.0 GiB","DataReclaimable":"21.3 GiB","Table":{"NumSet":530626,"LenSet":"137.3 GiB","AvgSet":277829.1613302024,"NumTrash":71957,"LenTrash":"27.2 GiB","AvgTrash":405928.1973400781,"NumTTL":92719,"LenTTL":"6.3 GiB","AvgTTL":72489.24479340804,"NumSlots":2097152,"TableSize":"128.0 MiB","Load":0.2530221939086914,"Created":20348,"Kind":0},"Compaction":{"Elapsed":0,"Remaining":0,"TotalRecords":0,"ProcessedRecords":0}}}
11

I believe with hashstore, deleted peices are held for 7 days, then flagged for removal on day 8, then actually removed on next compaction.

So the figures from node dashboard, are never really going to match the storage device exactly.

12

Perhaps you are right, I do not know.
My past tests with storj-up has showed the difference too, but longer it run - the precise it become.
I do not know is it still the case?