I lost power suddenly on my Ubuntu software disk mirror and when the system came back up it was in a read only status. I found this out after docker was running for a while and saw that no data was allowed to be written to the array. I shut down Docker and rebooted the system and all came back up and I didn’t suffer but 2 “Recoverable Failed” audits. Since then the node has come back to 100% on all scores. I feel I was fortunate that I saw the power failure and was able to get a quick handle on the array problem. My question is, what happens if this array would have been failed for several hours? Can the node sustain a long period of not being able to write data being sent to it? All of the database, temp, and identity directories are on the array so no data could have been written since the whole volume was read only. This is my first experience with a Linux array falling out of sync going into a R/O status so I’m trying to head off any future disasters…
If the node is reachable, but nothing is writable, I guess auditing will lead to suspension and disqualification - but not within hours, it will take some time.
Anyway, I strongly advise to install an alerting service, which will update you a) on success statistics as well as b) almost immediately, if an error occurs. I strongly belief, docker logs would have been written anyway (in the RAM?), so that the script would have been alerted you. Pls check it out here:
I’m currently working on a project / web service which will monitor your Storj node and notify you of changes etc in exactly that… Audit/suspension/online scores and some other things too
