Storage Node Suspension Score

[Storgeez]

@littleskunk

Most audit failures will decrease the suspension score except missing or corrupted pieces.

Do access denied errors (due to filesystem permissions) fall under “missing or corrupted data”? Had a bunch popup after a recent storage system update. Luckily I was looking at the log following the restart.

Also would be nice if SN detected a “data inaccessible” situation to prevent disqualification due to accidents (data directory not mounted, etc.).

[Alexey]

Your idea is posted there:

Prevent broken mount points from DQing nodes

We have read it multiple times that mount points are breaking, were done wrong after moving a node or replacing the harddrive. This often results in the nodes being DQed for missing pieces. This is my proposal to solve this problem: When the node starts, it checks in with the satellite. Now without any files, the node doesn’t know if it is new or pretty old. So let the satellite answer the check-in with some information about the node (e.g. used space on the satellite or 20 random piece-ids or…

Make the node crash with a fatal error if the storage path becomes unavailable

While similar to the suggestion posted here, the implementation of this one should require no changes to satellite code and focuses on taking the node offline instead of failing audits, when a storage location becomes unavailable. After discussing with @kevink we decided this is different enough to post as a separate suggestion. We seem to have 2 issues that we want to prevent. The node starts over at 0 if the mount point isn’t available at start. The node keeps running if the storage locatio…

The workarounds:
Linux:

Prevent broken mount points from DQing nodes

For logs written to a file tail -f /volume1/storj/v3/data/node.log | awk '/(ERROR|canceled|failed).*GET_AUDIT/ {system ("docker stop -t 300 storagenode")}' For logs in docker docker logs -f --tail 20 storagenode | awk '/(ERROR|canceled|failed).*GET_AUDIT/ {system ("docker stop -t 300 storagenode")}' Please be aware I haven’t tested the docker version. But it should work. This stops your node if it encounters a single audit failure, that might be overly aggressive as it would also kill your …

Windows:

Disqualified after 2 hours of [edit] failed audits?

@Hacker you can use the below powershell script to monitor for when the usb disconnects. When it detects a usb disconnection event, it will shutdown the storagenode. All you have to do is change the $driveLetter -eq ‘D:’ with the Letter of your usb drive. I know it is not a solution but it helps prevent those audit errors. If you combine this with uptimerobot monitoring you should get notified since the storagenode goes offline. Register-WmiEvent -Class win32_VolumeChangeEvent -SourceIdentifi…

[Storgeez]

Ah my bad, I forgot to check the idea portal.

[570RJ]

Instead of trying to specifically catch errors when disk is not accessible vs real corruption, wouldnt it be easier to just dont worry about differentiation between audit failures and count all of them towards suspension. That way, it does not matter wether you have real audit failure, or transient ones. Either way node is suspended and user can try to fix it. Difference of real vs transient audit failures comes when user tries to fix and lift suspension. user wont be able to fix real audit failures while and node will get DQed in couple days.

I see above approach much simpler to implement and manage - no branching logic, simple to understand and manage.

[Storgeez]

Yes, exactly, I don’t know why they went the split way - seems like a solution looking for a problem. In the end, the type of error can be masked by the SNO, since all result in unretrieved data.

[Alexey]

@570RJ

Instead of trying to specifically catch errors when disk is not accessible vs real corruption, wouldnt it be easier to just dont worry about differentiation between audit failures and count all of them towards suspension.

Is there such a thing as an "un-disqualify"?

I already said what I think about suspension instead of disqualification: It should be extended to work as a slow disqualification: forbid not only PUT, but GET too (see storj/audit-suspend.md at 5786595d141ff8b8c8be21e5e8696dd73e04d08d · storj/storj · GitHub); take any costs from your held amount to recover missed pieces if repair trigger is fired; reduce reputation and levels of trust (i.e. - put 25% to held amount, then 50%, then 75% depending on level of loses); enable vetting again. Th…

[Storgeez]

That’d be great if you could specify for the node to shutdown in case of such errors, some might have temporary storage problems that are more like “node unavailable” than “node unhealthy”. Would be nice have max.audit.errors.to.shutdown in the configuration that would protect the node temporarily in case of some kind of failure.