Node full but keeps getting data

When the problem arose, people like @littleskunk (see UPDATE: the SNO Garbage Collection & Disqualification bug has been fixed above) suggested to reduce the size allocated to your node to the minimum: 500GB.

You might try that, but I’m not sure it’s the same issue in your case.

Hi all,

Same Issue at my Node…
I have some free space on my Server… But i think something is wrong here?
I’m on Version v0.29.3

Thanks for help
Markus

Hello
I have 3 nodes that reached storage limit but still receive new data. It is lasting for a 3 days. How to stop receive data above the limit? I have a little free space on that nodes and soon it will be dq.
Nodes is 0.29.3

Restart your node and it will stop receiving data.
It might be the same bug I reported a few days back: https://github.com/storj/storj/issues/3751

thanks.
hm, looks like one of my nodes stop to receive new data without restarting it. Right at that moment ( a couple of hours ago) when I write this topic.
I have look at logs and there are only download events for now.
I cant check it on another nodes because i have increased a storage for few GB. It is need to wait when it fills up again.

In my case, it was rather severe, because it ended up using close to 12% more space than the allocated amount.

node still seems alive (before and after container restart). 0 bytes available on its storagenode partition before container restart. reduced -e STORAGE with 500MB. paste showing stats after container restart. i somehow immediately got 3.4GB of free space after container restart.
https://pastebin.com/DxahUJXu

so all in all, all of this actually seems to tick very, very well on its own, because:

  1. Im cutting corners and not saving 10% of disk space as a “buffer”
  2. Node hadn’t crashed
  3. It must have been piled up garbage collection, which got correctly dumped after container restart.

I am also not seeing the fix work on my node. Previously, I had 750Gb allocated to my node, and storage available was negative. I reduced stoage to 500Gb, followed the instructions here, and restarted my node. The dashboard has been reporting -314 GB for 3 days now. I’m running version 0.29.3.

Same here. My node say -7.9 GB available and is 7.9GB over allocated space.

it start to get new data again,
so I restarted it. only download messages in logs for now
let’s see what will happen next

I am having same issue. After investigating more and monitoring the node. It looks like there might be a bug in how Remaining space is reported on SNO dashboard. My free space was 22GB when Remaining space reached 0 (a week ago). No it is reporting -31GB for remaining space while my free space is 40GB. So files actually were removed but my remaining space reported did not reflect it correctly. Because of that I did not change space allocation on my node and I never ran out of space, even though Remaining space on the SNO dashboard is telling me it is -31GB now.

Is there a fix planned? Since my last power 12 days ago, my free Disk Space was going from -17.14GB via -16.50GB to now -17.52GB. Or is there anything i can do? I’m on Version 0.31.12.
Thanks Markus

The initial problem of getting data is fixed now.

The correct displaying of available space seems to be a problem but is mainly discussed in another thread.

My “out of space” error condition has not stopped the satellites from trying to send me more data. As a result I have received over 2000 upload attempts that have failed in the last 24 hours. The success rate is just 1.119%. wasting bandwidth. screenshot below. For now I have lowered my storage limit down to 500G so that my dashboard now shows negative storage space available. Hopefully the satellites will stop sending me stuff that my node rejects.
outofspace2

If they don’t stop although your dashboard shows negative space, then create a new bug report because this should be fixed in 0.31.12

I think this may be a new bug that started with 0.31.12. I have created a bug report in a different thread. It seems like with this latest version of the software there is more than one formula used for space calculations that are being used simultaneously which is causing inconsistent behavior on my node as a result of having trash in my trash folder. When my disk space available(as reported by the dashboard) is about equal to space used by trash, my node reports that I have space available but will start rejecting uploads with an error that says “out of space” even though the dashboard still says I have 11G of space available, which is the size of the contents of the trash folder.

Since I have lowered my storage setting to 500G my node is showing -44.2 GB and I have stopped receiving ingress. I just need to make sure that my storage setting is not too close to the amount of space I’m actually using. That seems to be when the node gets confused.

1 Like

This needs to be tested at the margins.

2 posts were merged into an existing topic: “piecestore protocol: out of space(…)” - action required?

Is this issue with receiving pieces when the node is full solved?

pretty sure it is. did not experience it anymore

1 Like