HDD degraded while hashstore migration, lots of files in blob folder

I tried to migrate three nodes to hashstore. Two nodes went fine, but the third one’s disk degraded during the migration (gradually increasing sector errors). I stopped the node with the corrupted disk, migrated all files to the new disk using rsync, and restarted the node. Based on the disk usage, it looks like the conversion to hashstore is finished and the node is running normally, but there are about 33GB of files left in the blob directory.

How should I proceed?

If old disk is zfs single disk data is lost. Otherwise you might try tools like ddrescue.

Node is online and running (lots piecestore uploaded / downloaded). Lost only few files from blob directory during data transfer to another disk according to log from rsync.

But an hour ago ago a node was disqualified from the saltlake satellite. The others are fortunately OK. I have to go through the logs and look at it.

Regarding the migration to hashstore, I have these errors in the log. Is there any solution?

If hash doesn’t match files are useless anyway. You can delete them after migration. I have seen a small number of such files in blobs folder on most of my nodes after migration.

3 Likes