Doesn’t work, if you allocated say 1TB from 2TB available. So, your node will report that it has 1TB of free space, even if the allocation is full. No go.
This will be broken by the first issue, the free space is calculated wrong, because it’s a free space on the disk, not in the allocation.
Will not work, with different FS they will use a different amount of space, simplest example: exFAT and also ZFS without compression and with 128KiB sector/cluster/whatever minimum size per file.
Then free space = Free Space as reported by OS - (Total Disk Space as reported by OS - Allocated Space).
That would eliminate the 10% we’re asked to leave free, or whatever portion of the drive was not allocated, from the value.
If I had a 2TB drive and I have allocated 1TB to it, with the rest being available, I would assume at some point that I will expand the node, so 1TB free would be technically correct. I don’t think anyone would be sitting on unused capacity for long.
If I had a 2TB drive and the world’s most bloated OS (ie windows) taking up 1TB of it, the node would report 0 available space, which is technically correct as well.
Or we could scrap the idea of “allocated space” and just go with use the full drive up to X free capacity.
How would trash not work? This file is currently 128KiB. I don’t care if the underlying storage stores it in a single 128KiB sector or 128x1KiB sectors. What I do care is that at the end of the 7 day period that trash would supposedly be deleted (allegedly), I would have 128KiB more space available. Take this value when retaining, store it in a file, move on to the next file. At the end of the run, the file says X bytes (the sum of all files that were moved). I’m expecting that in 8 days (7+1) I would get back X(ish) bytes. As a reminder, we are talking about something purely cosmetic afterall.
Also doesn’t work for the cases when you have a third-party usage (like Chia or your own files).
You may assume anything, but the satellite is forced to believe that what’s you allocated is a true storage (it doesn’t check it), however, sooner or later the truth will be discovered, when your node cannot accept any piece (we have a precaution check - if the node has less or equal than 500MB - it will advertise itself as full).
I would not lean only on that though anyway - we may have a bug. It will be fixed as soon as it’s discovered, but it will not help your node to survive in the suggested approach.
And the harsh truth would be an error to the client when trying to upload that “sorry, I’m out of space, can’t take that piece”. The client reports this to the satellite. The satellite updates a field in the current node database: update field “reported_errors” by +1.
NOTE: typed that before you updated with the FAQ.
With the current issues I can’t imagine anyone actually trying to keep chia plots alongside a node. Well, I know a guy who did. He deleted them a couple of days later. Cause it just doesn’t work.
And you could always do a du to get the space used by files outside of the storj node’s directory, and add that back in.
Have you considered going to a model where, when you allocate the space on a drive, it actually reserves the entire space as one big file (that can be resized on command)? And pieces and everything else are stored and indexed within that one big file? There’s another coin called Subspace that will go to mainnet soon that does just that, and resizing the allocation is literally instant, though I suppose in Storj’s case there’d have to be a limit on resizing it downward to the point where it would eliminate pieces.
Yep, and this is not good. However, it’s much better, than when this will happen always. And yes, the node will be removed from a hot cache on the satellite and will not be advertised anymore to the customers.
This is basically what’s node do. You may measure the required time 
Well, not outside the storage directory, but inside it. What’s outside - doesn’t matter for Storagenode.
No. It’s too close to a Proof of Storage, which is not our model. And I’m sure, most of Operators will not like your idea.
I completely agree that it’s not good. By keeping the current system we managed to save one out of the 24000 currently active nodes that ignored recommendations and used his OS drive as storage space for storj.
We can work together to save the rest of the 23999 currently active nodes that are actually in trouble.
I know a du can take a while, IF it’s dealing with a million small files. If it’s a bunch of chia plots, it goes very quickly.
But even if someone was trying to dual-use the drive with something else with a lot of small files, I still can’t imagine it taking over 48 hours, which is what it looks like my current filewalker is going to take. At least.
I’m guilty. I use my disk for VMs on the same host. Because, you know - share what’s not used right now…
And my nodes performs well, even if they are on Windows. Two Docker for Windows nodes and the GUI one. All uses the own disk to share space.
And they do not have almost any issues. Why yours have?
… it matters if you’re trying to calculate free space by Free Space = Free Space as reported by OS - (Total Disk Space as reported by OS - Allocated Space) + Space Used Outside of Storagenode. I mean, the possibility that someone is using up space outside of the storage node was the reason you rejected the idea of using this method for the calculation. So. Yeah. You said that such storage would matter.
Unfortunately - yes. We do not require to allocate the full disk, so you may run it with any spare free storage, thus we are limited to calculate the used space (by the node) and the free space (in the allocation).
I know, use what you have. Reading the recent announcement seemed to suggest that we are anticipating a tsunami of incoming data (otherwise testing based on a dozen petabytes wouldn’t be needed), so the network has grown beyond the “use what you have” stage. We are now in the commercialization of the network, ie “buy storage for storj as storage fills up”.
We can go back and forth all day (I’m not being unrespectful or attacking anyone). The current system is cracked. We need to fix it before it breaks apart. If anyone disagrees that the current system is broken, I can direct him/her/it to my previous posts in this thread. prefix-named folders are still present on several of my nodes. These prefix-named folders have a modified time (according to output shown above) of April 18th. It is currently May 2.
Well. Considering how tiny and how many millions of tiny files there are in a Storj node, it’s hard to imagine that counting the files outside of a storage node (for the small subsection of SNOs who try to dual use) could take longer than counting the space inside. I would imagine it would in almost all cases go far far far far quicker.
So. Again. Free space = Free Space as reported by OS - (Total Space as reported by OS - Allocated Space) + Space Used In Directories Outside Of The Storj Directory.
I like this idea. I consider this an update to SIP#001 (Storj Improvement Proposal #001).
I propose we go ahead and update the draft SIP#001 and put it up for SNO voting.
EDIT: Joking aside, there are permission issues with reading outside of the storagenode’s scope. It can’t du what it doesn’t have permission to.
Hmm. Well. If fixing these issues required specifying --device /dev/sda or whatever in the docker image, I’d do it in a heartbeat.
For the record, my filewalker just entered hour 25 of 95% disk utilization trying to add up JUST the files for the US satellite. Can’t wait for it to start EU. Assuming a version update doesn’t restart the thing and waste all that work and wear on my drive before it’s done. Again.
You mean that the storagenode software should touch files outside of its own scope?
No. Please don’t. There is enough of a mess inside.
We hope that very much.