Hi, i’ve been running a node on v3 network (upgraded from v2). Everything worked great until mid-end June. After all these version upgrades and for the past 2 months my node stops working randomly and it doesn’t automatically restart when a problem occurs. The only error i see in the logs is the following: “error”: “ordersdb error: database is locked”, “errorVerbose”: "ordersdb error: database is locked\n\tstorj.io/storj/storagenode/storagenodedb.(*ordersDB). Would that be the reason for my node crashes ? or any suggestion on why the node stops working? I use docker CLI installation on ubuntu 18.

Welcome to the forum @Pon! How is your HDD connected ?

Hi @nerdatwork and thanks for the reply. I use an ext. hdd - usb 3.0 but i’ve installed my node on an old computer so the connection is usb 2.0 I’ve setup static mount via /etc/fstab to the ext. hdd As i’ve stated this configuration was working with no problems since end of June. I’ve been thinking of migrating the whole node to a bit newer machine but first i want to identify the source of the problem if possible.

Can you check if your HDD is SMR ? [image] PSA: Beware of HDD manufacturers submarining SMR technology in HDD's without any public mention Node Operators Recently there have been several reports of SNOs running into massive CPU wait issues. It seems the common thread has been the use of HDD’s in which manufacturers have quietly started using SMR technology. [2003549104] Because of these overlapping tracks, when data is written to the disk, the drive has to rewrite several additional tracks. This obviously leads to much slower writes. These HDD’s usually deal with that using large caches and rewriting the shingles tracks during quiet times. How…

The problems you see are common when systems are IO constraint, this is also why @nerdatwork is asking about SMR drives. Though I think in your case the use of a USB 2.0 port could also cause this. If it is an option, it’s always best to connect the HDD internally. If that’s not an option you could consider moving the nodes db files to internal storage. That’s a bit of a risky procedure though. You have to do it exactly right or risk losing the node. It may have been working well before, but it probably also had less data. The more data it has the more traffic it sees and things like that could expose IO bottlenecks.

Hi again and thank you both for your answers ! It seems like it might be the HDD cause sometimes the PC starts like “frozen” and after a bit it works normally, however the CPU utilization is always low. I’m going to check whether my HDD is SMR and see from there. If i want to put the disk internally via sata, in the same machine would it be so much trouble to reconfigure the same node without losing its reputation and data? I see that there is a migration guide to a new device, is there any for changing your hdd connection on the same machine?

Shouldn’t be a problem if you mount it to the same path then no changes would be needed. If you mount it to a different path you just need to change the data location in the run command. The migration procedure you are referring to is for when you actually need to physically move the data, but that doesn’t apply to your scenario.

Nice. Then is going to be two birds with one stone. I’ll make the ext. hdd internal and by doing that i’ll see the exact hdd model and check whether its SMR. So, if i got everything straight, PMR HDDs are preferred instead of SMR right?

Definitely! SMR has problems with sustained random writes, because it needs to rewrite multiple tracks every time. Since Storj involves a constant stream of small writes, those HDD’s tend to eventually croak.

Here is a good youtube clip witch explain difference between SMR and PMR/CMR. https://youtu.be/aztTf2gI55k

So just to let you know, the HDD is working fine. It seems the problem was originating from an old 3Com Lan Card. Possibly it got rekt somehow. As soon as i changed the lan card and ethernet cable (just to be sure) the node became stable again. Thank you all again for your help !!

Node stops working randomly

Node Operators troubleshooting

nerdatwork September 2, 2020, 8:04am 4

Can you check if your HDD is SMR ?