Disqualified for unknown reason

After reading this [http://nfs.sourceforge.net/#faq_d10] I think I might understand the statement about incompatibility of SQLite and NFS (which, BTW, I am using to export storage from the NAS VM (*) to the StorJ VM running on the same ESXi host), but I still do not understand why this is an issue when the NFS export in question is used by only one client, as the locks do not work across different NFS clients, but should still work on the same NFS client.

Nonetheless, I will try to share the NAS space to the StorJ VM thorough iSCSI (running StorJ on the NAS itself is not an option for me) after I return home, but that sounds like a major remodeling of the setup…

(*) HBA is passed through as a raw device to the NAS VM, which eliminates most issues of running a ZFS based NAS as a VM.

Because of this:
https://www.sqlite.org/howtocorrupt.html#_filesystems_with_broken_or_missing_lock_implementations

However, in your case your node has been disqualified because it’s lost access permanently or temporarily to the storage.

So you would rule out a corrupted SQLite database as culprit for lost data (or lost references to data)?

EDIT: The data was not lost as it seems:

[root@storj blobs]# ll abforhuxbzyd35blusvrifvdwmfx4hmocsva4vmpp3rgqaaaaaaa/g5 | grep 7vnd
-rw-------. 1 root root  2317568 Dec  9 02:26 7vndiznyfv2osyb2ocf5lxux7ialxi2txvwnecjcpldn7lxteq.sj1

The corrupted database should not affect audits.
Satellite requests any random piece and expect the correct hash from the storagenode. Storagenode confirming that it accepted the challenge and should give the right hash for requested piece.
If storagenode accepted the challenge (so it’s online), but didn’t gave the requested hash within a timeout (5 minutes), then it will be placed into the containment mode and will not receive any new piece from that satellite until give the requested hash, it will be asked for it three more times. If node is unabe to deliver the requested hash the audit considered as failed.
If the storagenode would fail significant amount of audits in row it will be disqualified.

1 Like

Thanks @Alexey for the expansion. I have two questions regarding the logging of this auditing process:

  1. Can I see in the logs whether the piece was found and/or a hash sent?
  2. When a hash is sent, does the satellite reply with the info whether or not the hash was correct? If yes, is this logged on the storage node?

And is it possible for me to create the hash of one of the pieces in question (which apparently reside on my disk) manually and somehow check its correctness? I am asking this because I can still rule out that data got corrupted at rest on the HDD, so the reason must be something else.

Yes, you’ll see a downloaded line with the type GET_AUDIT if it was successful. If you see a download failed line with GET_AUDIT the audit failed for some reason. The error will mention the specific reason, like “file does not exist”.

I don’t believe there is a log for this, but satellites do communicate back the total number of audits and the number of successful ones. You can find these using the Storage node dashboard API (v0.19.0) .

I don’t think it is. During audit the satellite retrieves a stripe (a small bit of a piece) from all nodes hosting it. The satellite then tries to recreate the data using erasure codes and detects which pieces are corrupt. This is not a process you can do as a single node, it requires the stripes from the other nodes.

1 Like

Thanks for the hints and infos! This is what I can see in the logs: All lines in the logs from four days before the disqualification till about three days after which contained (case insensitive) “audit” were all “GET_AUDIT” messages of severity INFO of either type “download started” or “downloaded”. None of them mentioned any kind of error. The only strange thing with these messages that I noticed is that there are more “download started” (95) entries than there are “downloaded” entries (32), but after a short check it seems that some pieces have multiple “download started” entries with fewer or just a single “downloaded” entries. (*)

The only “download failed” messages were all not of type “GET_AUDIT” and related to network issues, probably for being too slow (use of closed network connection, connection reset by peer, broken pipe).

I’m still pretty much in the dark here. Is there a way of increasing the log level? For my next install that is, as in the meantime I have been disqualified from all four satellites…

Edit:
(*) It seems there are three more unique piece ids with “download started” than with “downloaded”:

[root@storj ~]# zgrep GET_AUDIT storj.20200107.log.gz | grep "download started"|cut -f5-|jq '."Piece ID"'|sort -u|wc
     26      26    1430
[root@storj ~]# zgrep GET_AUDIT storj.20200107.log.gz | grep "downloaded"|cut -f5-|jq '."Piece ID"'|sort -u|wc
     23      23    1265