"piecestore protocol: out of space(...)" - action required?

Hi, just wonder should i report any feedback on this.
So i got those errors now, as my node indeed got all of his allocated 0.7TB space filled.
Dashboard shows
“Disk Space Remaining:
-1,79MB”
Should i do something?
No upload, no download traffic, only 60MB a day, he keep getting new positons in log, all of “piecestore protocol: out of space(…)”,

Edit: disk space remaining changed to “48.90KB” so it not on minus now, seems its getting sorted out by itself after 1-2 days…

Edit2: but “upload faild” in logs, keeps adding in

Of course, your node is full.
Please, note - all wordings from the customer perspective. All uploads are ingress to your node and all downloads are egress from your node.

i mean there is no egress, ingress as well coz its full.

There is no guarantee that you would receive egress traffic all the time on your node. Testing and real customer downloads are intermittent, not constant. Most likely download testing will resume soon, and then you will again see egress traffic on your node.

Helen its not the problem, i have other node that is working just fine as we speak, and as this one don’t work. I think my duty is to provide feedback, so we can find possible problems early and prevent them. I decided to write this, because no one else mentioned here the “out of space” case. I think its not optimal what this node does, if morenodes will behave like that its a waste of resources. He should be doing egress as usual. This node is somehow broken because of me who allow it to be too much on and offline. Other node behaving normal. Other peoples nodes as well.

I see there one problem, if allocated all the available space, the node could fill it up almost to zero remaining and do not have more space to write orders to the database.
Please make allocations with 10% of reserved, this could save the node in edge cases.

1 Like

I really think 10% reserved is a little bit too big for big drive like 10 TB (1 TB reserve?), isn it?

Even 100 GB of database is already very big, and most probably will degrade performance for most

I have to agree regarding the big reserve on a huge disks. However, we still in the beta, the storagenode could overuse the allocation if we introducing a bug. Better to be safe at least in the beta.
You could take this risk if you want, and allocate more space.

1 Like

Another option would be to store the databases on a different volume than the data chunks. This is what I do, and it prevents me from worrying about database corruption resulting from a partial write that can’t be completed due to the disk being full.

I would not recommend such setup for everyone. There are more points of failure than with a standard setup.

Maybe. The databases are on the system volume in my case, though. If that disk dies, the whole box is going down anyway.

In case of separate HDD for data (including databases) it will survive the system disk failure.

1 Like

Indeed… but if the system volume fails then downtime is likely going to be >5hrs and so the node would get disqualified in production anyway.

In my case the system volume is on a RAID1 so I’m not worried about that. :slight_smile:

I don’t have any inside information, but I think that 5 hour number may change in the future. And even if it doesn’t, having data and databases in tact at least gives you a chance at recovering your node. I’d always go with that solution.

@Alexey I just started getting this same message “piecestore protocol: out of space” but I have plenty of space left. My docker started with “-e STORAGE=“510G”” and if you look at the partition I . created for storj:

$ df -h /opt/storj/
Filesystem                    Size  Used Avail Use% Mounted on
/dev/mapper/bigstorage-storj  600G  477G  123G  80% /opt/storj 

There is plenty of space left. The dashboard says “-2.1 MB” available.

Can it be that Linux hold some space free so that it cant get overfilled?

That doesn’t make sense. Linux doesn’t do things you don’t tell it to do. There is no such protection in linux. You can literally fill up every single bit in linux and it won’t prevent you from doing so. I created a 600GB partition and told Storj to only use 510GB. When you ask linux how much free space is on that partition, it responds “123G are free”. But if you ask Storj how much free space is there, Storj is replying “-2.1 MB”. Storj should be seeing at least 30GB more space.

This is caused by the difference between decimal and binary units. Your OS uses binary units (1024) Storj can do both. GB for decimal (which you used now) GiB for binary. The dashboard shows you decimal units as well though.

1 Like

It’s remained space in the allocation, not the free space on the disk.
And as @BrightSilence mentioned we everywhere uses decimal units, but Linux uses a binary units.

1 Like

Since filling up my node, my bandwidth usage has been extremely low. Am I losing out on payments because of this? Less network traffic == less payout? Or is the network bandwidth payment a much smaller portion compared to simply storing the data?

1 Like