UPDATE: the SNO Garbage Collection & Disqualification bug has been fixed

UPDATE Jan 1, 2020: this issue is now resolved

for details see Changelog v0.28.4 https://github.com/storj/storj/releases/tag/v0.28.4

Warning: Garbage collection is interfering with free space calculation. If your storage node is full and executes garbage collection it might accept more pieces than it has free space, leading nodes to become disqualified. We are working on a fix. Please reduce your allocated size to make sure the storage node does not run into a crash.

6 Likes

Please send an email notification too since not all SNOs are using forum.

2 Likes

I think that you shouldnt worry if you set up a real space allocated and it is not more than you really have. For my experience garbage coollector doesnt influence for real allocated space, just for calculating and dashboard. And it reduce space just in dashboard then finished, but it doesnt delete data…

This is something that only already full nodes should be concerned with, and to be on the safe side, they should reduce their allotted space to allow for the garbage collection to run its course without crashing their node. The issue is currently being fixed so this is just a temporary problem, but should not be ignored.

2 Likes

Screenshot_2019-12-20%20Node%20Dashboard

I have about 50 GB additional space on my HDD (mathematically not allocated to Storj). Should I be concerned, and reduce allocation? If so, how much?

Node ID: 1bAU7mutrJqvcUZPVZY5aYZRSpDvUYLGFnQZSby7nhYmBHwhgy

1 Like

I would keep monitoring that negative number on your space remaining which is how much you have went over your allotted space already. If it doesn’t come close to the extra available space on your drive that is still free (be sure it really is not already occupied by some other files) then you should be fine.

In my case the dashboard was showing -200GB after garbage collection and on the next restart it was showing 200GB free space. My storage node is filling that quickly and at some point will stop responding to any audit requests because the drive is full. Please take the warning serious. You can reduce the size to the minimum 500GB to avoid that outcome.

4 Likes

and thinking about that. My storage node has executed the garbage collection for 1 satellite. I am still waiting for the other 3 to send me the garbage collection bloomfilter. Now I have to follow my own advice and change my allocated size to 500GB because I can’t predict how much space the other 3 satellite will “free up”

3 Likes

Is that a concern if my 4-TB disk is less than half full?

as mentioned in the posts above, this only concerns those SNOs who have already full nodes.

My node is full with 1,2 TB. The HD has 1,4 TB. What Do you mean with reduction to 500 GB? In my opinion it’s not possible. The smallest possible full node has 500 GB. The most are bigger and can’t be made smaller.

assigning an allotted space of 500 GB will assure that there will be no more attempts to write more data to your node, of course it will not make any of the already stored data disappear. This would only be possible if partial graceful exit was already implemented. It is only a way to assure you won’t accidentally crash your disk because an attempt was made to write data when there was no more physical space available.

1 Like

It is not the same thing if i have a full 1,2 TB node? There will be no more attempts to write more data to the node.

Now I have reduced the node sizes as far as it was possible. But I am still unsure with the full nodes…

I have to downsize the full node with 1200 GB (test)data to 500 GB?

Downsize to 500GB didn’t work for me…
I downsize my Node to 500GB yesterday and it was ok.
Since 3 hours now I have a new folder in the Trash folder and its growing.
I’ve restart the Node and its looks good…for now.
Two big folders with over 90GB are now in the Trash folder.

First time I read this Info in my Logs (It appear the first time on 2019-12-18)
INFO trust Scheduling next refresh {“after”: “5h7m14.578680324s”}

That is completely okay.

Okey thanks.
But what kind of data is that? And when are the deleted?
The Trash folder is only getting full, when the Temp folder is getting crasy with a lot of incomplite downloads like this one:
blob-289422680.partial (4.096 KB)
They only showing up for less than a second.
Should I let it running when its happenend again?
My HDD space will not be less by this kind of error.
I don’t know why, but my disk space remaining will go far under -2.2TB
(Need a third screen to see the whole bar :smiley: )

It’s data that was deleted by the uplink, but for some reason not from your node. It’s kept in trash for a week in case there is something wrong and data that shouldn’t have been deleted was removed. so it will be gone after a week.

2 Likes

A post was split to a new topic: Error response from daemon: Container is restarting

More than 7 days have passed since the transfer to “trash” and nothing was removed … as it was 30.4GB free a week ago, so there is still 30.4 free now …