Node disqualified without suspension mode or errors registered in the log

Node Operators troubleshooting
1

Hello everyone without absolute notice at all no suspension e-mails, no notifications on the dashboard my node has been disqualified from the whole network. I have been checking the logs for the whole month of august and run the different scripts and there has been no audit failures at all. Despite that there is a 28 h gap on my log file that is unexplicable and after that gap the node restarted automatically and in that restart nothing in the log looks bad.
I have uploaded the node log if someone can downloaded it to check it and give me some hints on why I have been disqualified or if it is reversible from the Storj side.
Thanks for the help

Link to log https://mega.nz/file/J59xxbKQ#8griXIwXwDp204EgtNCeDFqPQ0gVr0yj_H6vV9YQ8vw

2

Why did you change log level from info to debug ?

3

Unfortunately DQ with audit score below 0.6 is permanent and not reversible.
The audit could be failed only if your node is online, but can’t provide requested pieces for audit either because it’s lost, or corrupted or inaccessible.
Seems your 28 hours blackout in the log could explain that. Perhaps your hardware was not healthy during this time, and data was inaccessible.
The problem with your hardware was between

	Line 4047638: 2020-08-10T09:57:25.141Z	INFO	piecestore	download started	{"Piece ID": "2XCFQ7C7SHLKRCEK72674VAJ4DULCFI4FEVEHKXCRA2SK4WAVALQ", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Action": "GET_REPAIR"}
	Line 4047639: 2020-08-12T13:44:20.258Z	INFO	Configuration loaded	{"Location": "/app/config/config.yaml"}

I can ask developers to take a look to logs on satellites for your node.
Since you use a docker on Windows, then I would like to recommend to rollback the Docker desktop to 2.1.0.5:

4

Thank you Alexey, I would like to know what happened on the satellite side because the node was working and online but nothing was recorded but also why the node didn’t trigger suspension before DQ.
I use docker on a Centos machine should I do any rollback or other stuff software related.
Cheers

5

The suspension is applied only in case of “Unknown” errors. Timeout, “file not found” or “wrong hash” are known.
In your case:

M:  "Verify: piece not found (audit failed)"
   N:  "audit:verifier"
   Node ID:  "==redacted=="
   Piece Hash Verified:  true
   T:  "2020-08-10T10:02:43.204Z"
   error:  "file does not exist; piecestore: (Node ID: ==redacted===, Piece ID: FL7GA2IGOLC264J6QAMHBF3WR6YXAIF3KJQCFWMY5CAR5EZPSMCQ): file does not exist"

a lot of such errors. Seems your system lost a lot of data.

I don’t think so

6

Thanks so much alexey then it has to be what I suspected and it is that for whatever reason and without my action the permissions on the folder that storj used changed (when I was checking, the folders had a padlock and an x icon on them and I was suspecting the worst) It can not read it can not write of course it will fail the audits. Well that is life.
Do you know any reason for the unwanted and unasked permission change Alexey?

Thanks

7

Maybe sudo docker run ... was used when you first run storagenode, and only docker run ... after a restart?
Maybe the user running docker run ... has not been added to the docker group?

8

You can view the history of the commands issued in cli as a logged in user:
history

as root:
sudo su and password
history

Maybe you will notice something that would indicate what I wrote earlier

9

I think @peem is right. The usual way to mix permissions is to run with a sudo and then run as a casual user or vice versa.
You should use either first or the second, but not mix them. Or add rights to the sudoers group (wheel in case of CentOS) and to the node’s user.
In such case even in mix, all should work too.
But better do not use a sudo and add your node’s user to the docker group and apply rights to that user.

10

Thank you both @Alexey and @peem for the help and the teaching, much appreciated. It had to be during some automatic restart of the node as I always start it with sudo and let it go until it requires my attention.

1 Like