Free space reported incorrectly

Hello gang.

As it was mentioned somewhere else on the forum, the dedicated disk functionality is deprecated. Based on this I did switch back to standard space accounting on nodes that do run hashstore (I do not run piecestore anymore).
After the switch some of the nodes are reporting as they are out of free space, where in fact there is space available (one example):


image

Some of the nodes are facing the opposite problem - they did under report the used space, causing the nodes to be unable to start, complaining there is no free space left on the device.

These are all configured with storage.allocated-disk-space: 100 TB.
The databases for these nodes are on separate drives and there is nothing else using the space on the partition.
The directories with the original piecestore were all deleted a long time ago, as were files in trash and garbage directories.
These are all running ext4.

Not sure how the accounting is done on hashstore, and if any kind of a startup scan is needed, but this was exactly the issue we were facing with piecestore and its accounting.
I had zero such problems when on dedicated disk.

Any quick solution for this? As these are supposed to be maintenance free as we know.

Thank you!

I am seeing the same issue on countless nodes all on hashstore.
Gotten to the point data has stopped coming in on some nodes, due to false overuse. Resulting in less ingress, and therefor less earnings.

Not a fix in miles.

Could you please provide results with this tool:

It appears setting the storage2.piece-scan-on-startup: true and then restarting the node fixes the issue. I confirmed this on multiple.
Just restarting the node doesn’t fix the problem.
I can only assume the node probably was accounting for pieces from the blobs directory which no longer were there, and startup scan flushed those bytes away.

Yes, if all pieces were migrated, but the databases (they used for piecestore) were not updated, it can explain this.