With one of the latest updates it seems like the space used calc used in the storagenode API has changed. Assume the following scenario: Your node has 5TB of data stored on disk. In your config you set the storage.allocated-disk-space to 1TB.
Previously it showed the used space of 5TB correctly in the API, now it shows a negative amount that im not sure how it is calculated.
Its overall bad approach, to calculate used space we calculate space used by pieces of each satellite, it can’t decrease to 1 TB if it was 7 TB even since u changed allocated space, that’s why you would receive negative space, but since we had negative space problems before we added check to code that calculates if allocated-used-trash < 0 we recalculate directory’s free space. To avoid this wrong calculations we are going to add partial graceful-exit for extra pieces after decreasing allocated space(it is in our roadmap). If you have any ideas or suggestions how to handle this right now we would kindly check it and try to implement.
On my node my free space was negative (by 30 GB) running on v1.11.1. After the update to v1.12.3, the node shows the correct used space, but now is showing free space equal to the amount of physical space available on the disk. This might just be a reporting problem, since the node is not starting any new uploads to fill this “free” space.
Any chance this recalculation could be the problem?
Partial graceful exit would be a great feature, but I would prefer it to not trigger automatically. Sometimes it can be useful to briefly reduce the load on your node by lowering the allocation to just not accept any new uploads. But that doesn’t mean you necessarily want to get rid of the data. So I suggest having a separate command to actually free up the space.
This is really confusing for the end user though, since depending on the situation, this number now means something completely different.
I recently had a fairly extreme version of this problem. I run one node on a Drobo device, which uses thin provisioning. It has a 16TB volume, but far less physical disk space. I had only 1.9TB assigned to Storj, which I now wanted to lower to 1.8TB. (Drobos get slow when they fill up beyond 75%, so I wanted to lower it slightly so it would eventually drop below that threshold. I’m not in a hurry for this though, so don’t want to trigger partial exit even if it were an option.)
This results in the CLI dashboard now showing the available space in the thin provisioned volume. Which is meaningless.
The web dashboard is even more surprising!
Apparently the node is currently using negative 13TB!
For what it’s worth, it does look like the node doesn’t actually tell the satellites that it has free space, since it’s not getting new data. But it’s really confusing to track how much data is actually available or how far above the assigned space the node is.
As an end user I would by far prefer that the node ALWAYS displays available space compared to the assigned space. Even if that leads to negative numbers, as that at least makes it clear that the current usage is above the assigned space.