Can't restart storagenode docker container after host system reboot

Node Operators troubleshooting
, ,
1

I’m running storagenode, as a Docker on my Synology.
So far, so good.

Apparently, there was a power outage and Synology rebooted (I have UPS, not sure, why it didn’t help).
So, at some point today, I went to check the status and I see, that the storagenode is not running. By the way, the watchtower container was running.
So, I tried to start it and got:

docker: Error response from daemon: Conflict. The container name “/storagenode” is already in use by container "f6d49…

Here is my docker command:

sudo docker run -d \
--restart unless-stopped \
-p ***:***\
-p 14002:14002 \
-e WALLET="0x***" \
-e EMAIL="***" \
-e ADDRESS="***:***" \
-e BANDWIDTH="20TB" \
-e STORAGE="6TB" \
--mount type=bind,source="/volume1/storj/Identity/storagenode",destination=/app/identity \
--mount type=bind,source="/volume1/storj/data/",destination=/app/config \
--name storagenode \
storjlabs/storagenode:beta

I know, how to fix that (sudo docker rm storagenode fixed the issue).

But I’m wondering, why it did not successfully started by itself? Could watchtower do anything about it? Perhaps, remove dead container?

2

One more observation:
The node was offline for about 9 hours, but still reports 100% uptime.

You can for yourself:
http://slavikca.myds.me:14002/

3

Your storgaenode alredy running

image
image1065×524 40.5 KB

use docker ps to make sure.
If you would like to recreate your node
docker stop storagenode - stop running node
docker rm storagenode - remove docker container
docker run ..... - create new container

4

While it is possible to place your SN Dashboard to be available directly on the Internet via an unauthenticated web page… it is not recommended that SNOs do this.

5

My question is not HOW TO RESTART storagenode. I know how to do that manually.

My concern is that, when host system restart going to happen next time - storagenode would not be able to restart by itself, because of that issue.

And yes, I know the risk of exposing dashboard. I’m not concerned.

6

If you check the storagenode logs, do you see any errors from right after the synology came back online?

On a side note, the stop command should be docker stop -t 300 storagenode for a graceful shutdown.

7

any errors from right after the synology came back online?

When Synology came back - storagenode failed to start, so there was no storagenode logs right after the start.

I started it after ~9 hours, and I see few errors like this:

2019-12-12T01:45:03.450Z ERROR piecestore failed to add order {“error”: “ordersdb error: database is locked”, “errorVerbose”: “ordersdb error: database is locked\n\tstorj.io/storj/storagenode/storagenodedb.(*ordersDB).Enqueue:52\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).saveOrder:672\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:426\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Upload:223\n\tstorj.io/storj/pkg/pb.DRPCPiecestoreDescription.Method.func1:985\n\tstorj.io/drpc/drpcserver.(*Server).doHandle:175\n\tstorj.io/drpc/drpcserver.(*Server).HandleRPC:153\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:114\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:147\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51”}

8

This error is discussed there Ordersdb error: database is locked