I don’t see any config change in 1.114rc to enable this new space feature.
of course it would, by definition – that’s what the used space is, what filesystem reports it to be.
I’d argue adding up the sj1 files is an educated guess (among other reasons – while you are done adding up last file, first few could have been deleted and a few more added). Asking filesystem how much free space is there or how much is used – is the atomic source of truth.
1 Like
Asking a filesystem how much space is free… tells you nothing about how much the Storj node is using. It could be 90% cat videos, and 1% .sj1’s. Which is why Storj’s used-space today is calculated by adding up the blobs. This enhancement allows a node to assume it’s the only user-of-space… so subtracting free-space from total-space should be very close to storj-used-space.
It’s a useful upgrade, and I’m sure many SNOs will enable it. But it’s useless on mixed-use filesystems.
No, the discussion is already in the context of the node having exclusive use of the disk. There are no cat videos.
Full quote:
This patch identifies the available space in a different way. It ignores all the calculated used space, but checks the available space in the running partition. UI will show fake numbers, but it should work."
Hence, the last statement in this quote is wrong. If anything, the number will become more accurate.
This enhancement tells the node to assume it’s the only thing using the disk. It’s not making all other directories on the filesystem read-only or anything. You drop one .txt file anywhere on that drive and the used-by-storj number is incorrect. I believe that’s what they mean by “fake”… because the node isn’t actually checking. It’s hoping.
2 Likes
Ok, but the numbers will be fake only if user breaks the promise of exclusive use. I’ll just assume it’s a poor choice of words.
2 Likes
I haven’t promised exclusive use. Hey, there 14TB, which will be used by storj in about 2 years (it’s very optimistic). Why that’s empty space shouldn’t be used by my cat video?
Off-topic: Do you have a feel for how fast you think your node is filling a disk? Some people are collecting estimates here so we can all compare.
I have. As a tradeoff, I don’t have to deal with running used-space filewalker (in 2 years when that drive fills up, you’ll see it take a couple of weeks to get through used-space filewalker, per run). The drive is dedicated to storj, and storj alone. The only metric I care for the node to keep track of is the free space (which in theory it should do every 10 mins, based on what the OS is reporting = instantly). Once per restart, the trash folder gets scanned for the space it occupies.
That leaves us with how much space is actually used by the node (park the “fake” space reports aside and the per satellite usage, I don’t care). Used space (for dashboard display) = max drive size (as reported by OS) - trash (see above) - free space (as reported by OS) - space reserved (set by me in config.yaml). Quintillions of IOPS saved = happy SNOs = happy clients = Storj can claim even smaller carbon footprint = win-win situation.
Looking forward to that single reply “but the most important piece of information in modern world is the per satellite usage, I cannot get through my day without knowing this extremely important detail!!!11oneeleven”.
TL;DR: Some people are willing to trade space for convenience. This change is exclusively for them.
1 Like
my 4TB node doing it for ~1.5 hours
The “method of three” does not apply in storj nodes, as I have said numerous times in the past. You can’t extrapolate that a 14TB node will take 5 hours for a used-space.
Can’t comment on badger, I don’t plan to enable it any time soon.
2 Likes
For me it looks like an equivalent of df.
because it doesn’t include any change for the UI, so the UI will still use a DB.
In that case you are in my shoes, you will not enable this feature and would relay on a used-space-filewalker, and that all updates of the databases are finished without any issues.
2 Likes
Im not 100% sure, but I believe right now the 5GB buffer when ingress is halted actually depends on the used space calculations and not the actual available space. Therefore, if the used space calculation is wrong (as it tends to be sometimes), then it may actually use the last 5GB buffer instead of stopping ingress.
No, the whole point of that check is to avoid filling the disk if the used space calculation is not accurate.
In fact, I have set node allocation size to 200TB, and then applied a quota on a dataset to the actual size I want node to be limited to. This works correctly.
It checks a minimum available space from both values, either of them will trigger a “node is full” signal.