Problems setting up 2nd storagenode on same host

Node Operators troubleshooting
1

Hi Folks,

I’m trying to setup a 2nd storagenode on the same host (ubuntu 22.04 on raspberry pi). I run the setup command:

docker run --rm -e SETUP=“true”
–user $(id -u):$(id -g)
–mount type=bind,source=“/root/.local/share/storj/identity/storj2”,destination=/app/identity
–mount type=bind,source=“/storj2”,destination=/app/config
–name storj2 storjlabs/storagenode:latest

But I get this error on my node.log:

2022-05-31T01:54:48.129Z INFO Configuration loaded {“Process”: “storagenode”, “Location”: “/app/config/config.yaml”}
2022-05-31T01:54:48.130Z FATAL Unrecoverable error {“Process”: “storagenode”, “error”: “storagenode configuration already exists (/app/config)”}

I then try to run storj2:

docker run -d --restart unless-stopped --stop-timeout 300
-p 28968:28968/tcp
-p 28968:28968/udp
-p 14003:14003
-e WALLET=“0x…”
-e EMAIL="my@email.com"
-e ADDRESS=“myhost.mydomain.com:28968”
-e STORAGE=“2TB”
–user $(id -u):$(id -g)
–mount type=bind,source=“/root/.local/share/storj/identity/storj2”,destination=/app/identity
–mount type=bind,source=“/storj2”,destination=/app/config
–name storj2 storjlabs/storagenode:latest

I get this error in my node.log:

2022-05-31T02:02:55.217Z FATAL Unrecoverable error {“Process”: “storagenode”, “error”: “Error starting master database on storagenode: group:\n— stat config/storage/blobs: no such file or directory\n— stat config/storage/temp: no such file or directory\n— stat config/storage/garbage: no such file or directory\n— stat config/storage/trash: no such file or directory”, “errorVerbose”: “Error starting master database on storagenode: group:\n— stat config/storage/blobs: no such file or directory\n— stat config/storage/temp: no such file or directory\n— stat config/storage/garbage: no such file or directory\n— stat config/storage/trash: no such file or directory\n\tmain.cmdRun:195\n\tstorj.io/private/process.cleanup.func1.4:372\n\tstorj.io/private/process.cleanup.func1:390\n\tgithub.com/spf13/cobra.(*Command).execute:852\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:960\n\tgithub.com/spf13/cobra.(*Command).Execute:897\n\tstorj.io/private/process.ExecWithCustomConfigAndLogger:93\n\tmain.main:479\n\truntime.main:255”}

Any idea what the problem is?

$ docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
d1418e5a9069 storjlabs/storagenode:latest “/entrypoint” 5 minutes ago Up 6 seconds 14002/tcp, 0.0.0.0:14003->14003/tcp, :::14003->14003/tcp, 28967/tcp, 0.0.0.0:28968->28968/tcp, :::28968->28968/tcp, 0.0.0.0:28968->28968/udp, :::28968->28968/udp storj2
bf6e73bb45fa 3a4ccde00f5d “/entrypoint” 3 weeks ago Up 21 hours 0.0.0.0:14002->14002/tcp, :::14002->14002/tcp, 0.0.0.0:28967->28967/tcp, :::28967->28967/tcp, 0.0.0.0:28967->28967/udp, :::28967->28967/udp storj1

$ ls -l /storj2
total 268
-rwxr-xr-x 1 root root 9392 May 29 23:51 config.yaml
drwx------ 2 root root 16384 May 29 22:05 lost+found
-rw-r–r-- 1 root root 238383 May 30 22:11 node.log

$ docker logs --tail 20 /storj2
2022-05-31T02:14:56.808Z INFO Invalid configuration file key {“Process”: “storagenode-updater”, “Key”: “contact.external-address”}
2022-05-31T02:14:56.808Z INFO Invalid configuration file key {“Process”: “storagenode-updater”, “Key”: “operator.wallet-features.0”}
2022-05-31T02:14:56.808Z INFO Invalid configuration file key {“Process”: “storagenode-updater”, “Key”: “console.address”}
2022-05-31T02:14:56.808Z INFO Invalid configuration file key {“Process”: “storagenode-updater”, “Key”: “operator.wallet”}
2022-05-31T02:14:56.808Z INFO Invalid configuration file key {“Process”: “storagenode-updater”, “Key”: “storage.allocated-disk-space”}
2022-05-31T02:14:56.808Z INFO Invalid configuration file key {“Process”: “storagenode-updater”, “Key”: “server.private-address”}
2022-05-31T02:14:56.808Z INFO Invalid configuration file key {“Process”: “storagenode-updater”, “Key”: “operator.email”}
2022-05-31T02:14:56.808Z INFO Invalid configuration file value for key {“Process”: “storagenode-updater”, “Key”: “log.level”}
2022-05-31T02:14:56.811Z INFO Running on version {“Process”: “storagenode-updater”, “Service”: “storagenode-updater”, “Version”: “v1.55.1”}
2022-05-31T02:14:56.811Z INFO Downloading versions. {“Process”: “storagenode-updater”, “Server Address”: “https://version.storj.io”}
2022-05-31 02:14:57,038 INFO exited: storagenode (exit status 1; not expected)
2022-05-31T02:14:57.143Z INFO Current binary version {“Process”: “storagenode-updater”, “Service”: “storagenode”, “Version”: “v1.55.1”}
2022-05-31T02:14:57.143Z INFO Version is up to date {“Process”: “storagenode-updater”, “Service”: “storagenode”}
2022-05-31T02:14:57.174Z INFO Current binary version {“Process”: “storagenode-updater”, “Service”: “storagenode-updater”, “Version”: “v1.55.1”}
2022-05-31T02:14:57.174Z INFO Version is up to date {“Process”: “storagenode-updater”, “Service”: “storagenode-updater”}
2022-05-31 02:14:58,176 INFO success: processes-exit-eventlistener entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2022-05-31 02:14:58,185 INFO spawned: ‘storagenode’ with pid 44
2022-05-31 02:14:58,187 INFO success: storagenode-updater entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2022-05-31 02:14:58,373 INFO exited: storagenode (exit status 1; not expected)
2022-05-31 02:15:00,381 INFO spawned: ‘storagenode’ with pid 52

2

You may want to make sure /root/.local/share/storj/identity/storj2 and /storj2 are both created and empty before before running the setup command. This is how I’ve set up my storage nodes before running setup. e.g.

rm -r /root/.local/share/storj/identity/storj2 /storj2
mkdir -p /root/.local/share/storj/identity/storj2 /storj2
# copy in your identity files to /root/.local/share/storj/identity/storj2

docker run --rm -e SETUP="true" \
--user $(id -u):$(id -g) \
--mount type=bind,source="/root/.local/share/storj/identity/storj2",destination=/app/identity \
--mount type=bind,source="/storj2",destination=/app/config \
--name storj2 storjlabs/storagenode:latest

The other thing I was thinking about is the --user argument. I had a thought that perhaps the directories on the host are owned by a different user to the container user, so it could be causing some problems. If you’re running this all as root it’s probably not a problem though.

1 Like
3

I copied the config.yaml file from the 1st storagenode to put in the 2nd storagenode config directory, incrementing the ports in the config file +1.

This was causing the issue. Apparently the correct process is to start with an empty config directory when creating a new storagenode.

1 Like
4

You may want to run them as a service. If you use docker, it may have problems communicating to the outside world with it’s virtual ethernet adapters, should you use more than one /24 network on a separate network port. You have to route it all manually.

1 Like