Two cases - same issue.
Situation 1: Looks like there is something wrong with external HDD or enclosure (don’t sorted out yet). The problem is that once in the 1 or 2 months, RPi looses the mounted folder. It only helps when the RPi is restarted. After restart mount comes back all OK.
Situation 2: RAID6, lost 1 drive (out of 10), and also lost mounted folder. It should not be like this, RAID6 can lost 2 drives and should keep working, have no idea why, but it’s not about it. Ubuntu runs on mirrored SSD’s, so the system keeps working fine. But without mounted folder After restart same story as with RPi, disk came back, mounted point as well. All working fine.
In both scenarios docker/storagenode keeps running. But instead of writing data to the external drive / raid, it starts from zero writing to the RPi’s sd card, and to the RAID1 where ubuntu is installed. Mounted folder (when the drive is not mounted) is always empty. But after these two situations i see that there are new DB’s with the new files. Also zabbix showing growing data from 0 again.
Storagenode instead of detecting failure, keeps writing data to the wrong folder (well it’s the same folder, but not the same drive). This will lead to the problem. Because once the satellite will try to retrieve data that was written to the SD card / raid1 , instead of mounted external drive/raid6 it will not find that data.
In my opinion storagenode should always check this and stop working after the mounted drive is lost. Because instead of 2 db’s i have now 4db’s
How to deal with it?
Will it 100% will be disqualified in the future, once it will not find that data, written to the sd card/raid1 ? Should i kill these nodes my self and start again or is there any way to merge the data?