Node was just disqualified on europe-west

Hi, I have a long running node (with high performance if you ask me) on my server, node ID 12DpuxieuH7SBVtA4zTtKHafvNyBEtqBm7QEZdSKqmUsHLch1ey , which has now been disqualified on Europe-West :frowning:

The server suffered hardware damage during transport when moving to a new home. It took some time to diagnose the problem which ended up to be motherboard failure. As we were moving at that time, it took a while to diagnose the problem, get new parts and get the server up and running again. Fortunately all the data was intact and I was able to reinstall, which also allowed me to upgrade the software as well (Debian upgrade from Stretch/OMV4 to Buster/OMV5).

The node is now back online and all was looking well, but I a small mistake mapping the storage directory to the new docker image. After about 30 minutes I noticed this because usage on the dashboard was too low, so I took the node offline and corrected the docker mapping to the old storage dir.

During this time, the node was already getting in new data quickly, but obviously failed to audit the old data. Europe-West is probably closest to me and was quickest, but now has an audit score of 59.9% so that means disqualification.

Your node has been disqualified on Tue, 25 Aug 2020 07:57:32 GMT . If you have any questions regarding this please check our Node Operators thread on Storj forum.

Is there any way to fix this? I hope this can be repaired because permanent disqualification seems a bit harsh for a small mistake like this. I have my node running since the alpha version of v3 and really did my best to be a good SNO and restore the data.

Welcome to the forum @mdr!

Disqualification is permanent and there is no way to fix it. If you have fixed the mapping then your node should work properly with other satellites.

It’s indeed unfortunately not possible to fix the satellite that has already disqualified your node. You should copy the blobs you received in the other path to the correct blobs folder though to prevent other satellites from missing data. That will prevent failed audits on files you have received in the mean time. Please only copy the blobs.

If you want to get back on the europe-west-1 satellite you would have to start a new node. This is really only worth it if you have a lot of free space to fill. Otherwise the remaining satellites will make use of the space anyway and you shouldn’t really worry about missing one.

Happy to re-join the forum, it has been a while (my bookmark was still to the old storj community site).

I’m not a programmer, a bit sad that a small mistake has these consequences :pleading_face:
My node is probably closest to europe-west for upload speed and bandwith, because this satellite stores by far the most data (much more than US and Asia) so it really is a loss. To make matters worse, the new blobs are already gone so now I’m at risk for audits on those as well. :sob:

The old node had about 7TB data on an 8TB drive, long vetted and had past the period for release of held amounts. I’ll leave it running for a while and see how things go.

What would you advise if I wanted to start a second node on the same server? Shrink the size of the old node to make room for the new one and build reputation? At the moment I have no unused empty drives, I would have to add a sata controller to add more.

1 Like

Make every hdd new node. I you mess 1 node, more posible other survive.

If you want to do it on the same HDD, you can start the new node on the same HDD. Then I would first wait until that node is vetted on all satellites (with the exception of stefan-benten as there is pretty much no activity on that satellite at the moment).
After that your new node gets the same amount of traffic your old node would have gotten. So that’s a good time to limit the size of the old node to 500GB which is the minimum. That would force all new data to the new node. You could run them in tandem like that for a long time and just see how it goes. The upside is that the new node will be on all satellites again and will start gaining reputation and going through the held back periods. If your old node survives, it can still get you income on the remaining satellites. If it gets disqualified on other satellites… well at least you’re already working on replacing it. Eventually you may want to reduce complexity and do a graceful exit on the old node, so you don’t have 2 of them running on the same HDD.

Thanks for the advice!

So far (7.5hrs uptime) the old node seems to be surviving on the other satellites, with audit scores slowly recovering. But I don’t like missing europe-west so I’ll propably take up your advice at some point and start a slooow migrate to a new node (held back will take a lot of time, but I can patient while the old node remains online).

Also I’ll see if I can rearrange some disks anyway to make space for a node on another disk. Not being able to accept more data just doesn’t feel right :wink: But not at the same time, to prevent vetting from taking too long.

1 Like