Disk filled, and now what?

Hi xavidpr4,

I do not mean to be alarmist, but your current configuration is potentially dangerous and could lead to database corruption. It looks like you have allocated 930 GB on a drive that holds 980 GB (i.e. you have allocated 95% capacity). Although it could run just fine like this, it is not recommended. I think there are a few things going on here that have caused confusion. Don’t worry, I went through this exact problem at one point.

You hard drive capacity is rated by the manufacturer in Gigabytes (GB, base 10). When you run the command df -h you are actually seeing the capacity expressed in Gibibytes (GiB, base 2). If you run the command df -H (note the capital H) you will see the capacity expressed in terms of GB. If I recall, the size column will show you the size of the drive without the ext4 5% reserved amount already subtracted. So if you want to allocate 90% of the disk that is available, you need to calculate 90% of the 95% that is left after ext4 reservation.

The other side of this is that the storagenode will allocate space in whatever units you specify. So if you specify the units as GB, it will allocate based on gigabytes. If you specify as GiB, it will allocated based on gibibytes.

So run ‘df -H’ and look at the size column. It will probably show 984 GB. Take this and subtract the 5% ext4 reserve space:
984 GB x 0.95 = 934.8 GB
Then take 90% of this value:
934.8 GB x 0.90 = 841.3 GB

The reason you see 511MB free space is that the node is designed so that when it sees there is only 500 MB free space, it stops accepting uploads.

You can run the node with less than the 10% space, but you would be doing so against recommendations. This 10% extra space is especially important with small disks like this one to allow for overallocation, databases, moving of trash and garbage collection.

TL;DR The allocated space value for your storage node should be (probably, depending on what you see from df -H) 840 GB. You can change the setting in your docker run command and the node will slowly regain free space as pieces are deleted. This would be my recommendation. If you need instructions on how to change parameters and restart the node, they can be found here.

6 Likes