Storagenode/watchtoweer won't start correctly

Node Operators troubleshooting
1

Hi
A couple of issues I’m facing:

  1. My nas that I’m using for storj unfortunately gets killed occasionally due to power fluctuations in the local area. It reboots automatically but the storagenode/watchtower containers regularly fail to restart despite both being started with the following options:

    docker run -d --restart unless-stopped …

As a result, both containers have been offline for a while. So the question here is what can I do to make this more resilient.

  1. The storagenode currently won’t start, it’s erroring with:

    Error: piecestore monitor: error verifying location and/or readability of storage directory: open config/storage/storage-dir-verification: no such file or directory.

I’m passing in the following mount option:

--mount type=bind,source="/var/storj/storage",destination=/app/config

And the directory contains the following:

ls /var/storj/storage/
config.yaml  kademlia  orders  revocations.db  storage  trust-cache.json

So the question here is how can I fix that?

Here’s the full command I’m using to start the storagenode:

#!/bin/bash
docker run -d --restart unless-stopped \
  -p 28967:28967 \
  -p 14002:14002 \
  -e WALLET="xxx" \
  -e EMAIL="xxxx" \
  -e ADDRESS="storj.example.com:28967" \
  -e BANDWIDTH="2.0TB" \
  -e STORAGE="2.5TB" \
  --mount type=bind,source="/var/storj/identify/storagenode",destination=/app/identity \
  --mount type=bind,source="/var/storj/storage",destination=/app/config \
  --name storagenode storjlabs/storagenode:latest

Many thanks

2

Are you sure it doesn’t spell as identity ?

3 Likes
3

Maybe it should but I’m quite sure:

ls /var/storj
audit_db.sh           storage                  storj-launch-command.sh.old
audits_satellites.sh  storj-dashboard.sh       storj-restore.sh
identify              storj-launch-command.sh  storj-watchtower-launch-command.sh
4

Hello @damacneil,
Welcome to the forum!

The file config/storage/storage-dir-verification should be created with the first setup. If the file doesn’t exist - the storagenode will fail to start, because you probably mounted a wrong disk or not mounted on startup. It’s usually happened with Unraid (they start a docker service before the mount of the disks).
If you think that your data folder is correct, then you can run setup once:

  1. Rename or delete config.yaml
  2. Run the command for setup: https://documentation.storj.io/setup/cli/storage-node#setting-up-the-storage-node
  3. If you renamed config.yaml - rename it back
  4. Run the storagenode as usual (without -e SETUP="true")
5

I really recommend only storing user data on Synology on a path on one of the defined volumes, like /volumes/storj. They have a tendency to wipe anything that isn’t inside those user locations on DSM updates.

Additionally, instead of trying to fix the restart you should consider a UPS to prevent the restarting to begin with. Those sudden restarts are a guarantee your data will get corrupted at some point.

1 Like
6

Thanks for helping.

@BrightSilence It’s not synology and whilst I whole heartedly agree UPS would be a wise investment I still have to be able to get things back to a sane point where I’m able to run the storagenode container.

@Alexey I think this covers what you recommend but still no luck

[root@srv storj]# rm storage/config.yaml
rm: cannot remove ‘storage/config.yaml’: No such file or directory
[root@srv storj]# cat storj-launch-command.sh
#!/bin/bash
#docker run -d --restart unless-stopped \
#  -p 28967:28967 \
#  -p 14002:14002 \
#  -e WALLET="xx" \
#  -e EMAIL="xx@gmail.com" \
#  -e ADDRESS="storj.xx.co.uk:28967" \
#  -e BANDWIDTH="2.0TB" \
#  -e STORAGE="2.5TB" \
#  -e SETUP="true" \
#  --mount type=bind,source="/var/storj/identify/storagenode",destination=/app/identity \
#  --mount type=bind,source="/var/storj/storage",destination=/app/config \
#  --name storagenode storjlabs/storagenode:latest
docker run --rm -e SETUP="true" --mount type=bind,source="/var/storj/identify/storagenode",destination=/app/identity --mount type=bind,source="/var/storj/storage",destination=/app/config --name storagenode storjlabs/storagenode:latest
[root@srv storj]# ./storj-launch-command.sh
[root@srv storj]# docker logs -f storagenode
Error: No such container: storagenode
[root@srv storj]# fg
vim storj-launch-command.sh

[1]+  Stopped                 vim storj-launch-command.sh
[root@srv storj]# cp /tmp/config.yaml storage/config.yaml
cp: overwrite ‘storage/config.yaml’? y
[root@srv storj]# cat storj-launch-command.sh
#!/bin/bash
docker run -d --restart unless-stopped \
  -p 28967:28967 \
  -p 14002:14002 \
  -e WALLET="xx" \
  -e EMAIL="xx@gmail.com" \
  -e ADDRESS="storj.xx.co.uk:28967" \
  -e BANDWIDTH="2.0TB" \
  -e STORAGE="2.5TB" \
  -e SETUP="true" \
  --mount type=bind,source="/var/storj/identify/storagenode",destination=/app/identity \
  --mount type=bind,source="/var/storj/storage",destination=/app/config \
  --name storagenode storjlabs/storagenode:latest
#docker run --rm -e SETUP="true" --mount type=bind,source="/var/storj/identify/storagenode",destination=/app/identity --mount type=bind,source="/var/storj/storage",destination=/app/config --name storagenode storjlabs/storagenode:latest
[root@srv storj]# ./storj-launch-command.sh
213b803e2faeb421df8b0c228fe5ee8a318c33534c7d610819b2ccd2356042d3
[root@srv storj]# docker logs -f storagenode
2020-12-22T09:57:00.260Z	INFO	Configuration loaded	{"Location": "/app/config/config.yaml"}
Error: storagenode configuration already exists (/app/config)
2020-12-22T09:57:00.912Z	INFO	Configuration loaded	{"Location": "/app/config/config.yaml"}
Error: storagenode configuration already exists (/app/config)
2020-12-22T09:57:01.592Z	INFO	Configuration loaded	{"Location": "/app/config/config.yaml"}
Error: storagenode configuration already exists (/app/config)
2020-12-22T09:57:02.504Z	INFO	Configuration loaded	{"Location": "/app/config/config.yaml"}
Error: storagenode configuration already exists (/app/config)
7

My apologies, I jumped to a bad conclusion there.

You need to remove this from your docker run command. This parameter should only be used in the one time setup command, not every time you run the node.

2 Likes
8

Brilliant, thanks I missed that and it’s running :slight_smile:

2 Likes