"piecestore protocol: out of space(...)" - action required?

Hi, just wonder should i report any feedback on this.
So i got those errors now, as my node indeed got all of his allocated 0.7TB space filled.
Dashboard shows
“Disk Space Remaining:
Should i do something?
No upload, no download traffic, only 60MB a day, he keep getting new positons in log, all of “piecestore protocol: out of space(…)”,

Edit: disk space remaining changed to “48.90KB” so it not on minus now, seems its getting sorted out by itself after 1-2 days…

Edit2: but “upload faild” in logs, keeps adding in

Of course, your node is full.
Please, note - all wordings from the customer perspective. All uploads are ingress to your node and all downloads are egress from your node.

i mean there is no egress, ingress as well coz its full.

There is no guarantee that you would receive egress traffic all the time on your node. Testing and real customer downloads are intermittent, not constant. Most likely download testing will resume soon, and then you will again see egress traffic on your node.

Helen its not the problem, i have other node that is working just fine as we speak, and as this one don’t work. I think my duty is to provide feedback, so we can find possible problems early and prevent them. I decided to write this, because no one else mentioned here the “out of space” case. I think its not optimal what this node does, if morenodes will behave like that its a waste of resources. He should be doing egress as usual. This node is somehow broken because of me who allow it to be too much on and offline. Other node behaving normal. Other peoples nodes as well.

I see there one problem, if allocated all the available space, the node could fill it up almost to zero remaining and do not have more space to write orders to the database.
Please make allocations with 10% of reserved, this could save the node in edge cases.

1 Like

I really think 10% reserved is a little bit too big for big drive like 10 TB (1 TB reserve?), isn it?

Even 100 GB of database is already very big, and most probably will degrade performance for most

I have to agree regarding the big reserve on a huge disks. However, we still in the beta, the storagenode could overuse the allocation if we introducing a bug. Better to be safe at least in the beta.
You could take this risk if you want, and allocate more space.

1 Like

Another option would be to store the databases on a different volume than the data chunks. This is what I do, and it prevents me from worrying about database corruption resulting from a partial write that can’t be completed due to the disk being full.

I would not recommend such setup for everyone. There are more points of failure than with a standard setup.

Maybe. The databases are on the system volume in my case, though. If that disk dies, the whole box is going down anyway.

In case of separate HDD for data (including databases) it will survive the system disk failure.

1 Like

Indeed… but if the system volume fails then downtime is likely going to be >5hrs and so the node would get disqualified in production anyway.

In my case the system volume is on a RAID1 so I’m not worried about that. :slight_smile:

I don’t have any inside information, but I think that 5 hour number may change in the future. And even if it doesn’t, having data and databases in tact at least gives you a chance at recovering your node. I’d always go with that solution.