The space on one node has been reset to zero for the second time. First time it has happened start of last month, now it has happened again starting this month.
Badger is on. After a restart the filewalker runs and updates to the correct used space number. Badger seems somewhat healthy as the filewalker completes fast.
On restart or during run I do not see any unusual errors. No database errors no badger errors or alike.
What else could I check?
Maybe you have had a database locked errors somewhere in between?
For me it sounds like a bad records in the database though. Maybe you need to re-create all related to pieces or reload them - then bad records should gone.
Are these piece_spaced_used.db and and storage_usage.db used_space_per_prefix.db?
Then I would simply recreate them.
Yes, they are. This is why I suggested it as a first option. Of course you would need to restart the node with enabled scan on startup to fill-up them back.
I have replaced
piece_spaced_used.db
storage_usage.db
used_space_per_prefix.db
Filewalkers on all satellites have completed already yesterday. However used space is still displaying completely wrong.
I see ERROR messages concering the badger like this:
2025-07-02T22:26:56Z ERROR blobscache satPiecesTotal < 0 {"Process": "storagenode", "satPiecesTotal": -768}
Should I remove the cache files and restart the node?
It should be updated automatically. Since I’m out of ideas, why it doesn’t work on your node, then you may try to recreate a badger cache.
It goes up very slowly, like 1 GB per 3 hours or something.
However as the used space filewalkers have completed since yesterday I would expect that the space calculation from the should now be visible somehow.
The node is showing with 7GB used space now.
This could be only in the case, if the data is not updated in the databases for some reason, like the database is locked error.
I have checked the log once more there is no locked error and no database error.
I have restarted the node again and immediately the used space shows a much higher number probably the correct one. So I don’t know what the issue is.
Maybe it re-reads the stat after some time (like once per 12h or on the next restart) or maybe the filewalker is failed last time and finished after the next restart (the badger cache allows the filewalker to finish scan faster, because previously scanned pieces already in the cache, thus their second scan was faster).