So ZFS slowly started after a reboot, meanwhile docker quickly restarted. As such docker started my nodes as if they were fresh nodes. (It didn’t wait for ZFS to mount the old storj directory, it just created a new one).
A couple nodes got dq’d after a few hours of this. So i shut them all down to take a look and figure out what happened.
Evidently on 4 or 5 of my nodes, I now have two storage data directories.
One has hundreds of GBs in it, the other has around 13GB in it. This is consistent across all 5 nodes.
I’m failing audit’s because files don’t exist.
Can I rsync the blobs from the 13GB partition, to the normal storj partition?
Also, to fix issues like this, I am doing what others with USB drives and such recommend - I am going to put the identity files in the root storage data directory so that if something isn’t properly mounted, then it will just fail.