The repair job is currently hitting a lot of storage nodes that have no space left but at the same time still, report free space to the satellite. This can happen if the free space cache is getting inaccurate. I have seen this on my own storage node. The free space calculation is executed only once on startup, cached in memory and we try to keep the cache accurate by increasing or decreasing it with every upload and delete. In the meantime, I have stored about 100GB personal data on the drive that I am using for one of my storage nodes. The storage node was not aware of that and still reporting that it has 100GB available but unable to accept any uploads.
The fix is obvious. Run the free space calculation more than once and if the disk is full “unsubscribe” from node selection. The only problem with that obvious fix is the pure number of affected storage nodes. I would like to know if there are any other ways to trick out the free space cache. Ideally from storage nodes that don’t use the hard drive for anything else. Please check your storage node log for possible out of space errors and let’s try to find out how we managed to get to that point.
Note: In the current release the storage node is not showing the correct free space values. That is a known issue and will be fixed with the next rollout. More accurate is the free space value that you find in the storage node logs on every upload started line.
I don’t understand this part. Do you mean that when node is reporting the free space, it sends the calculated_free and does not instead use min(fs_real_free - desired_fs_free, calculated_free)?
tune2fs -mN /dev/sdXY
Will set a percentage of root reserved blocks on ext filesystem. Idk how it works with docker since docker runs things as root, but if node runs unprivileged the amount of available space will change.
On windows there are per-user disk quota settings that can also be changed in runtime.
The hard part of this process, is that Storj need to know how many storj files there, but not all files to prevent cheating with space used. But dashboard cold just show Space alocated-real used space(with OS dependent conversion) result shold be more then 500MB then all OK can get data
This siimply check on fly how much space is used and how much is free at all on HDD.
Also result shold be bigger then space Alocated to storj minimum then it means that HDD is present.
As far as I understand, the “trick” in this case - is use the space shared with storagenode storage.
We “trick” the cache for free space calculator, it’s just not aware of the our usage.
The storagenode has a different mechanism to prevent uploads to the full storage, and there is a difference.
The storagenode do not report the physical available space during the work, it do it only based on calculations on start and accounting added/deleted files by storagenode, but not ours
That is what happend on my storage node but why do we have so many storage nodes with the same problem? That indicates that there is another bug or that most storage nodes are using the same hard drive for other personal data. I would like to rule out other bugs and questioning if the cache value is accurate or not and in which situation is it getting wrong?
I had this error also, but it repored node full but there was lot of free space, and HDD used only to Storj
There is no DB on this HDD also only pure data.
This is something that underestimates free space, but it seems garbage collection causes used space to be counted twice. My guess it is added to the trash size but not removed from the storage size.
It could also be multiple nodes running on the same storage. There is no advantage to doing that, but it wouldn’t surprise me if some SNOs still do this.
Btw, if the cause of these deviations is outside the nodes data, then you only need to check free space on the disk. There’s no need to do full inventory again like on startup. This should be a very low impact on resources.
Node even in theory cannot know how much space is left for her. For example, when several nodes live on the same array and quotas are not set.
All that it can is react to no free space io error.
I like that idea. I was concerned that the current free space check would catch it too late but I have overseen that we execute it on every upload. We could keep the free space on disk and cache in sync all the time. As soon as there is less than 500MB free the “unsubscribe” function would be executed. I think that is a good solution for the moment. I will try to communicate my wish to the developer team.
In the background we have pushed a v1.6.5 release to the windows nodes but not to docker. It turns out that the impact was higher than we would have expected. That looks like the root cause for the high number of affected nodes is simply that storage nodes share the drive with the windows operating system. It only needs 500 MB to run into this bug. Now I am thinking that the solution might be as simple as it sounds and there is no hidden layer behind it.
More important would be the question how you manged to get to that point. Have you stored any other files on the drive that you are using for the storage node? Why is the free space value wrong in the first place. Sure if the value gets wrong I would expect a high number of errors in the logs. Basically every upload should error out.
Sorry, I miss one important thing… This logs from uplink, I just try to upload files to the network like a client and see it. So, many storage nodes have this issue with space calculation.
All my storage nodes did not have any issues with space, because it exclusively used by storage node and not shared with anything else.
Some of my nodes also have storage space problem and thay also exclusivly used by storj, even there is no DB on disk, only storj stored files. So manly this is errors accuring in space calculations during node runing
There’s also the reported-less-than-actually-used bug with used disk space still in the open, how about you release a fix for this first? It’s confusing. I have 180 GiB unaccounted for.
By this Windows’ developer comment, the system drive on Windows will very often be slower on its own due to the way the system configures access to the file system. I suspect it wouldn’t hurt to add “use non-system drive for the storage if possible” to recommendations somewhere in the documentation.
I’m better to wait for fix.
However, it’s obvious by the other reason - when you use the system drive for OS and for storage, your node lose race more often anyway because of concurrency access to the same resources from OS and from storagenode.
Note that it’s even more than that. Not only OS file accesses interfere with the operation of a storage node, but also the system drive has additional filters installed for intercepting I/O operations that are not in use on other drives. This means that even if most OS files might be in cache and not hit the drive directly, the latency will be higher.
