My disk switches to "read-only" mode and makes the node restarts indefinitely

Node Operators troubleshooting
1

Hi,

Since a few days, one of my nodes keeps restarted.
I noticed that the disk switches to “read-only” mode but I don’t know why.

I recently changed the port of my USB disk few days ago. Maybe it could the cause?

My config

  • Debian VM
  • Local attached storage (not usinf NFS or CIFS or SAMBA) through USB 2, such as my other storage nodes

Recap of the symptoms

  • The disk witches to “read-only” mode, which seems to cause the node restart indefinitely
  • Several errors can be found in the log, such as:
    - Download failed
    - failed to add order
    - upload failed
    - FATAL Unrecoverable error (seems to be due to read-only filesystem)

How can I share with you my logs in a readable manner?

What I’ve tried

  • stop the storagenode, unmount the disk and remount it (this way is writable) and then restart the node. It seems to work at the beginning (few hours max) and then the problem happens again…

Do you think it could be due to a corrupted disk?
It seems to be related to this post but I would to have your opinion.

Thanks for your help

2

Sounds like the disk is defective. Happened for me a few times (unrelated to Storj). The read only is so, that you can rebuild raids.

2 Likes
3

I had that problem when my ext4 superblock was corrupted. So check your disk and filesystem.
Also happened on my external USB enclosures. They seem to be a lot more prone to this kind of corruption.

2 Likes
4

Happened to me once too.
Linux tend to switch a disk to read-only mode when things start going wrong with the disk, to minimize the risks of making things worse…

So yeah, definitely worth checking the whole drive for errors!

I would stop the node too while investigating… But you probably did that already ^^

5

Yes indeed.
It is actually an old huge external disk of 650GB. I wanted to wait just another 1 month to be able to perform a graceful exit for this node since it appears to cost more than it generates revenues.