So ZFS slowly started after a reboot, meanwhile docker quickly restarted. As such docker started my nodes as if they were fresh nodes. (It didn’t wait for ZFS to mount the old storj directory, it just created a new one).
A couple nodes got dq’d after a few hours of this. So i shut them all down to take a look and figure out what happened.
Evidently on 4 or 5 of my nodes, I now have two storage data directories.
One has hundreds of GBs in it, the other has around 13GB in it. This is consistent across all 5 nodes.
I’m failing audit’s because files don’t exist.
Can I rsync the blobs from the 13GB partition, to the normal storj partition?
Also, to fix issues like this, I am doing what others with USB drives and such recommend - I am going to put the identity files in the root storage data directory so that if something isn’t properly mounted, then it will just fail.
You can move blobs from the the mountpoint to the actual disk. However, if nodes are disqualified - you can start from scratch. If audit score below 0.6 the DQ is permanent and not reversible.
To prevent such thing in the future please put the storagenode to the subfolder on the disk. In such case the container would not start from scratch, it will just fail.
Thanks Alexey - these are rather new nodes, and i was only DQ’d from one sat, so i’m going to put them into my ‘sandbox’ of nodes that if they get dq’d dont matter (for testing new hardware, etc).
You should definitely use a subfolder.
I never had that problem with zfs on Ubuntu 20.04, all my drives are mounted before docker starts (storj mountpoint is on an external eSata drive). But should docker start without my mountpoint available, it would crash because I chose /media/kevin/STORJ/STORJ as my storj directory but the mountpoint is /media/kevin/STORJ. So the node crashes if the subfolder is unavailable.
Live and learn. It actually says this in the documentation but I figured zfs mountpoints were a little different than the main issues reported here which were with usb hdd’s.
Now i gotta go through and change all my old nodes so they are consistent with the new ones. There goes my sunday!