Node stops working randomly

Hi, i’ve been running a node on v3 network (upgraded from v2).
Everything worked great until mid-end June.
After all these version upgrades and for the past 2 months my node stops working randomly and it doesn’t automatically restart when a problem occurs.

The only error i see in the logs is the following:
“error”: “ordersdb error: database is locked”, “errorVerbose”: "ordersdb error: database is locked\n\tstorj.io/storj/storagenode/storagenodedb.(*ordersDB).

Would that be the reason for my node crashes ?
or any suggestion on why the node stops working?

I use docker CLI installation on ubuntu 18.

Welcome to the forum @Pon!

How is your HDD connected ?

Hi @nerdatwork and thanks for the reply.

I use an ext. hdd - usb 3.0 but i’ve installed my node on an old computer so the connection is usb 2.0
I’ve setup static mount via /etc/fstab to the ext. hdd

As i’ve stated this configuration was working with no problems since end of June.

I’ve been thinking of migrating the whole node to a bit newer machine but first i want to identify the source of the problem if possible.

Can you check if your HDD is SMR ?

1 Like

The problems you see are common when systems are IO constraint, this is also why @nerdatwork is asking about SMR drives. Though I think in your case the use of a USB 2.0 port could also cause this. If it is an option, it’s always best to connect the HDD internally. If that’s not an option you could consider moving the nodes db files to internal storage. That’s a bit of a risky procedure though. You have to do it exactly right or risk losing the node.

It may have been working well before, but it probably also had less data. The more data it has the more traffic it sees and things like that could expose IO bottlenecks.

1 Like

Hi again and thank you both for your answers !
It seems like it might be the HDD cause sometimes the PC starts like “frozen” and after a bit it works normally, however the CPU utilization is always low.

I’m going to check whether my HDD is SMR and see from there.

If i want to put the disk internally via sata, in the same machine would it be so much trouble to reconfigure the same node without losing its reputation and data?

I see that there is a migration guide to a new device, is there any for changing your hdd connection on the same machine?

Shouldn’t be a problem if you mount it to the same path then no changes would be needed. If you mount it to a different path you just need to change the data location in the run command.

The migration procedure you are referring to is for when you actually need to physically move the data, but that doesn’t apply to your scenario.

1 Like

Nice. Then is going to be two birds with one stone.
I’ll make the ext. hdd internal and by doing that i’ll see the exact hdd model and check whether its SMR.

So, if i got everything straight, PMR HDDs are preferred instead of SMR right?

Definitely! SMR has problems with sustained random writes, because it needs to rewrite multiple tracks every time. Since Storj involves a constant stream of small writes, those HDD’s tend to eventually croak.

1 Like

Here is a good youtube clip witch explain difference between SMR and PMR/CMR.
https://youtu.be/aztTf2gI55k

So just to let you know, the HDD is working fine.
It seems the problem was originating from an old 3Com Lan Card. Possibly it got rekt somehow.
As soon as i changed the lan card and ethernet cable (just to be sure) the node became stable again.

Thank you all again for your help !!

1 Like

Nice to hear that it worked out.

If anyone else has an external drive that they want to make internal, but lack the SATA connectors on the motherboard for it, here’s a tip: Go to your favorite online cheap electronics shopping site and search for SA3004. It’s a 4 port SATA PCIe extension card which can be had for less than $20.

I bought one about a month ago, when I ran out of integrated SATA ports on my motherboard. I set up my second node on a hard drive plugged into it. No driver required under Windows 10. Speed tests were OK, and it has been working reliably for me so far, knock on wood.