Received Node Disqualified Email But Node Still 100% Audit 100% Suspension

Today I received an email saying that my node is disqualified on US1 although it was working just fine. Then I checked my node, the scores are still 100%.


What can be wrong here?

Update: another email that the node was also disqualified on EU1. One more node also got disqualified on US1???

Have you checked the log to see any pieces getting uploaded/downloaded on said DQed satellites ?

I stopped the node. Should I start it again?

I think I made a mistake by starting the node before rsync is finished. But I wonder why the audit score on dashboard isn’t updated at all? The data is still there. What can I do now? @Alexey pls help.

Just making sure, does rhe node id contained in the email match your node’s id? Dont want it to be referencing an old node and not the one you are currently running.

I figured out the problem now.

  • I wanted to migrate the node to a bigger disk.
  • Somehow rsync stopped before sync is done.
  • I didn’t know about that so I started the node on the new disk
  • Got DQed, but the dashboard score somehow still stuck at 100%

the data still here, intact, just need to finish the sync. Can I somehow get requalified?

that’s 2 nodes, one 29 month old, one 13 month old. :frowning: that hurts :frowning:

This looks like due to rsync stopping early, your node got a bunch of failed audits because some data was still missing, and the audit score plummeted. You can check the logs to see the DQ reason and failed audits to confirm this.

Reason that the dashboard hasnt been updated could be that your node hasnt gotten a reputation score update since the audits started failing.

As far as I know, unless due to a clear bug from the satellite, DQs are permanent.

3 Likes

I checked the logs, many failed download with file not exists error. That’s a waste. I hope there’s some kind of requalification. The data is still there. Both nodes got DQed on EU1 and US1. Any advice what should I do with these nodes?

Sadly that’s the truth :point_down:

Unfortunately, DQ is permanent. It is not some setting we can just flip a switch on. Engineers have told us that it cannot be reversed. You can start over with a new identity, or use it for any Sats that havent DQ’d the node if you want.

2 Likes

Yep, this and satellites don’t communicate with nodes any more after DQ. So the scores won’t be updated anymore.

@nyancodex sorry to hear it. It sucks losing a node, especially 2. But migrations are something you need to keep a close eye on. Don’t assume any step goes right, check.

3 Likes

So the data of DQed sats will get deleted itself or I have to manually delete them? Sorry first time to get DQed after 3 years of SNO. :frowning:

1 Like

You can use the forget feature to clean up.

Get the satellite ID’s from here: Satellite info (Address, ID, Blobs folder, Hex)

2 Likes

Thanks for your kind words. Yeah I usually redirect rsync output to a file then check its timestamp from time to time with ls -liat, if the file last mod time stops changing then I assume the sync is completed, which is very lazy and stupid…

1 Like

Many tks. :sob: 20 chars

I get it, you wanted to get your node online again as soon as possible. I’ve done dangerous things too, managed to survive them and learned to not worry too much about a little more offline time if it means making sure the data is all there before I start it.

5 Likes

Use tail -f file then u see only the last lines of content and you don’t have to assume.

2 Likes

Or use -aP and watch the progress live. Run rsync in a screen session to not interrupt it.

2 Likes

What’s the Storj equivalent to “measure twice, cut once”?

1 Like

“DQed is permanent”…

1 Like