What happens if a node runs out of space?

Pac · September 2, 2020, 9:48am

Hello

I’m wondering what happens when a node runs out of space on its storage disk.
This topic has been kind of discussed in other threads (like these: Dashboard.sh /no space left on device: unknown & What happens to the storagenode when diskspace is all used by Storj) but I don’t know if things changed since then, and it was not clear how exactly it impacts nodes reputations/score/disqualification.

So, let’s imagine the following fictive scenario:
I set up a node to run on a 2TB disk, but configured so it uses only 1TB for Storj, so I can use the rest for my personal files. After several months, the node gets close to its limit around 900GB and goes on happily for filling up the last remaining 100GB it is allowed to use.
But in the meantime, I was not paying attention and did add more than 1TB of personal files to the disk, and it now turns out to be full with no remaining space whatsoever for Storj (or me).

What is going to happen to this node?

Would it get suspended? (Probable I would say, if database files gets malformed because it cannot properly write to them, depending on how sqlite3 handles that)
Could it get disqualified because of that?
Is the node software capable of detecting proactively when the remaining disk space is getting very low (e.g. less than 500MB free) so it stops accepting new data, in a “clean” way?

LarsOS · September 2, 2020, 12:42pm

I am not an expert, but I believe the answers are:

Yes.
Yes.
No.

I wouldn’t share a single drive for Storj and other data, unless it was a huge drive and a temporary arrangement.

baker · September 2, 2020, 12:50pm

As far as I know the node software only rechecks total disk space available when the node is restarted. I am pretty sure if you did run out of space the dbs would be damaged, which would likely cause suspension as the node would probably encounter fatal errors. It shouldn’t cause immediate disqualification as that is only due to failed audits.

You might be able to mitigate this by running the node software as a different user and setting up user quotas for the disk.

nerdatwork · September 2, 2020, 1:10pm

Yes it does. Currently your node is chosen for upload if you have at least 500MB free space. This is after setting up your node to have minimum 500GB actual free space. When it goes below 500MB it will accept few uploads then stop accepting them. Personally I have seen it go as low as ~230MB of free space left.

Pac · September 2, 2020, 1:55pm

I know the Node software does that already with regards to the remaining space allocated to it. But does that really apply to the actual free space remaining on the disk?

Agreed, I’m not doing that either, I was just wondering just in case.

baker · September 2, 2020, 1:56pm

I think this is only true in that it checks how much allocated space is left. Lets say you have a 2 TB disk and allocate 1 TB to storj. If storj is currently using 500 GB and the user consumes 1.45 TB, the node is still going to think you have 500 GB available and won’t stop at the 500 MB limit.

Edit: See below

nerdatwork · September 2, 2020, 2:51pm

It will be caught here

github.com

storj/storj/blob/65e6b177735bdd9cc6994a1210c59db448ec9bdd/storagenode/monitor/monitor.go#L101


      
          		service.log.Warn("Disk space is less than requested. Allocated space is", zap.Int64("bytes", service.allocatedDiskSpace))
          	}
          
          	// on restarting the Piece node server, assuming already been working as a node
          	// used above the alloacated space, user changed the allocation space setting
          	// before restarting
          	if totalUsed >= service.allocatedDiskSpace {
          		service.log.Warn("Used more space than allocated. Allocated space is", zap.Int64("bytes", service.allocatedDiskSpace))
          	}
          
          	// the available disk space is less than remaining allocated space,
          	// due to change of setting before restarting
          	if freeDiskSpace < service.allocatedDiskSpace-totalUsed {
          		service.allocatedDiskSpace = freeDiskSpace + totalUsed
          		service.log.Warn("Disk space is less than requested. Allocated space is", zap.Int64("bytes", service.allocatedDiskSpace))
          	}
          
          	// Ensure the disk is at least 500GB in size, which is our current minimum required to be an operator
          	if service.allocatedDiskSpace < service.Config.MinimumDiskSpace.Int64() {
          		service.log.Error("Total disk space is less than required minimum", zap.Int64("bytes", service.Config.MinimumDiskSpace.Int64()))
          		return Error.New("disk space requirement not met")

littleskunk · September 2, 2020, 3:02pm

No. Even if you have no space on disk you can still respond to audits because everything needed for that is in memory and not on disk. You will fail to write down the order and you will not get paid for download traffic but that should be the only impact.

If you don’t get suspended you also don’t get disqualified.

The storage node is checking free space on every upload. As soon as only 500MB are left it will unsubscribe from the node selection by sending a message to the satellite. The satellite is caching the nodes table for 3 minutes. So within 3 minutes after hitting 500MB free space the upload should stop.

This doesn’t stop you from filling the 500MB with personal data. So you can still get to the point that the node is 100% full.

Pac · September 2, 2020, 3:26pm

Thanks @littleskunk for these official details

So in short, we obviously should avoid this as much as possible, but it’s not the worse that could happen to a Storage node.

Cheers