On a statistical approach for the used space file walker

[snorkel]

If I set a storage quota for the Storagenode dir, and I allocate in config more space, what happens when it hits the quota limit?
Dose’t it see it like the drive is full and lets 5GB free in that quota limit?
Or tries to go pass the quota and the OS stops it at quota limit?
Could it crash the node?

[Alexey]

@snorkel

Dose’t it see it like the drive is full and lets 5GB free in that quota limit?

I guess this one, If the OS reports available space for this user equal to quota. If not, then it may crash.
On Windows it reports the free space available before quota (if you enable it to be not only reporting, but also as a limit).

[mike]

I second the idea for an option to define that the whole drive/mountpoint is used for a node.

The one size fits all for options is not valid for the diversity of nodes in the network anyways - many examples of options and design decisions that is already in place to support diversity.

In my design all nodes are on dedicated disks, perhaps a vote poll could be started on the forum to shed some light on the distribution of nodes on shares storage via dedicated ?

It would certainly save 50-100mill iops on many nodes at every restart or upgrade. Thus, leaving resources available for other core activities like user uploads and downloads

[Toyoo]

@mike

The one size fits all for options is not valid for the diversity of nodes in the network anyways - many examples of options and design decisions that is already in place to support diversity.

To reiterate what has been said many times—Storj development team is too small to support many ways to do the same thing. You can argue ad infinitum that something needs to be done, but unless there is also human time to implement this thing, it will just not show up.

[pangolin]

@Toyoo

Storj development team is too small to support many ways to do the same thing.

Why rejecting the idea of alternative node software then?

[Toyoo]

The answers are given already in this thread: Alternative Node Software

[Alexey]

You could start a separate idea with this be an option. Like suggested here:

Avg disk space used dropped with 60-70%

Something like a “storage.use-os-free-space: true” config.yaml flag would be really nice (even if you maybe also had to specify the mount/drive to go with it). I don’t care how much space a node uses: just that it tries to leave 500GB free.

However @Toyoo is right

@Toyoo

To reiterate what has been said many times—Storj development team is too small to support many ways to do the same thing.

Perhaps a new experimental feature would help better:

@Andrii

2fceb6c storagenode/blobstore: blobstore with caching file stat information (mod time, size)

[mike]

I would need to disagree. I’m not sure if a language barrier makes my point come across wrong.

There’s already many options available to tune the node for individual needs. Examples could be scan at startup, max concurrent connections etc.

Btw - this is a thread to brainstorm on how to optimize the system, not a random
request based on personal preferences. This would make the iops requirements 0 for used space calculations.