2nd node in docker fail to start

lovaszl · January 7, 2024, 5:39pm

Hello

I thought it would be easy, but it wasn’t.

The disk is full on my first node running under synology docker. I requested an auth token for my existing email address and generated an identity.
I created port forwarding on the router and first ran the new container with setup true as storagenode2 with the mount parameters.

docker run --rm -e SETUP="true" \
    --mount type=bind,source="/volume2/shared2/storj2/identity",destination=/app/identity \
    --mount type=bind,source="/volume2/shared2/storj2",destination=/app/config \
    --name storagenode2 storjlabs/storagenode:latest

I create the new container, but it doesn’t seem to care about my port settings:

1. Node

docker run -d --restart unless-stopped --stop-timeout 300 
-p 28967:28967/tcp -p 28967:28967/udp -p 14002:14002 
-e WALLET="xyz" 
-e EMAIL="xyz@gmail.com" -e ADDRESS="xyz.no-ip.org:28967" 
-e STORAGE="14.2TB" --net=host 
--mount type=bind,source="/volume3/shared/storj/identity",destination=/app/identity 
--mount type=bind,source="/volume3/shared/storj",destination=/app/config 
--name storagenode storjlabs/storagenode:latest 

New 2. Node

docker run -d --restart unless-stopped --stop-timeout 300 
-p 28968:28967/tcp -p 28968:28967/udp -p 14004:14002 
-e WALLET="xyz" 
-e EMAIL="xyz@gmail.com" -e ADDRESS="xyz.no-ip.org:28968" 
-e STORAGE="4TB" --net=host 
--mount type=bind,source="/volume2/shared2/storj2/identity",destination=/app/identity 
--mount type=bind,source="/volume2/shared2/storj2",destination=/app/config 
--name storagenode2 storjlabs/storagenode:latest 

2024-01-07T17:19:54Z	INFO	Configuration loaded	{"process": "storagenode", "Location": "/app/config/config.yaml"}
2024-01-07T17:19:54Z	INFO	Anonymized tracing enabled	{"process": "storagenode"}
2024-01-07T17:19:54Z	INFO	Operator email	{"process": "storagenode", "Address": "@gmail.com"}
2024-01-07T17:19:54Z	INFO	Operator wallet	{"process": "storagenode", "Address": "xyz"}
2024-01-07T17:19:54Z	INFO	db	database does not exist	{"process": "storagenode", "database": "info"}
2024-01-07T17:19:54Z	INFO	db	database does not exist	{"process": "storagenode", "database": "bandwidth"}
2024-01-07T17:19:54Z	INFO	db	database does not exist	{"process": "storagenode", "database": "orders"}
2024-01-07T17:19:54Z	INFO	db	database does not exist	{"process": "storagenode", "database": "piece_expiration"}
2024-01-07T17:19:54Z	INFO	db	database does not exist	{"process": "storagenode", "database": "pieceinfo"}
2024-01-07T17:19:54Z	INFO	db	database does not exist	{"process": "storagenode", "database": "piece_spaced_used"}
2024-01-07T17:19:54Z	INFO	db	database does not exist	{"process": "storagenode", "database": "reputation"}
2024-01-07T17:19:54Z	INFO	db	database does not exist	{"process": "storagenode", "database": "storage_usage"}
2024-01-07T17:19:54Z	INFO	db	database does not exist	{"process": "storagenode", "database": "used_serial"}
2024-01-07T17:19:54Z	INFO	db	database does not exist	{"process": "storagenode", "database": "satellites"}
2024-01-07T17:19:54Z	INFO	db	database does not exist	{"process": "storagenode", "database": "notifications"}
2024-01-07T17:19:54Z	INFO	db	database does not exist	{"process": "storagenode", "database": "heldamount"}
2024-01-07T17:19:54Z	INFO	db	database does not exist	{"process": "storagenode", "database": "pricing"}
2024-01-07T17:19:54Z	INFO	db	database does not exist	{"process": "storagenode", "database": "secret"}
2024-01-07T17:19:54Z	INFO	server	kernel support for server-side tcp fast open remains disabled.	{"process": "storagenode"}
2024-01-07T17:19:54Z	INFO	server	enable with: sysctl -w net.ipv4.tcp_fastopen=3	{"process": "storagenode"}
2024-01-07T17:19:54Z	FATAL	Unrecoverable error	{"process": "storagenode", "error": "listen tcp 0.0.0.0:14002: bind: address already in use"}

What can I do wrong?

First, I accidentally generated the identity with the name storagenode (I hope it doesn’t destroy my old node), then I created a new one with the name storagenode2, but the error remains the same.

tylkomat · January 7, 2024, 6:01pm

Did you run the container in setup mode first?

docker run --rm -e SETUP="true" \
    --user $(id -u):$(id -g) \
    --mount type=bind,source="<identity-dir>",destination=/app/identity \
    --mount type=bind,source="<storage-dir>",destination=/app/config \
    --name storagenode storjlabs/storagenode:latest

lovaszl · January 7, 2024, 6:40pm

Thanks, I wrote that I did it.

nerdatwork · January 7, 2024, 7:35pm

/volume2/shared2/storj2/storage

What are the contents of folder ?

lovaszl · January 7, 2024, 7:44pm

snorkel · January 7, 2024, 8:56pm

Look at the last line in log; it sais the 14002 is in use. And you don’t need the -p parameters if you use network host mode.
Just use my ports, change only the WAN IP and use port forwarding rules in router only for 28981 and 28982, if you use my ports. Then you have the dashboards at local node IP:14011 and local node IP:14013.

https://forum.storj.io/t/my-docker-run-commands-for-multinodes-on-synology-nas/22034/7?u=snorkel

Dosen’t matter what name you use for identity generation. Use storagenode for every identity. You only need to use different paths for data and config, and different name for the container. And before generating a new identity, remove the old one from the generation folder and sync your PC’s clock.
You should read the first part of my post too, because it’s a comprehensive guide for multinode setup.

Alexey · January 8, 2024, 4:19am

this is mean that your node doesn’t use the docker’s NAT (the bridge), so you must provide all ports as unique ports for the host, include private and local ones, see an example for a binary nodes. This is also mean that -p options are not used, so you may remove them.

For your case the second node should be run either with additional arguments or with changed local addresses and ports in the config.yaml file, i.e.

docker run -d --restart unless-stopped --stop-timeout 300 \
-e WALLET="xyz" \
-e EMAIL="xyz@gmail.com" -e ADDRESS="xyz.no-ip.org:28968" \
-e STORAGE="4TB" --net=host \
--mount type=bind,source="/volume2/shared2/storj2/identity",destination=/app/identity \
--mount type=bind,source="/volume2/shared2/storj2",destination=/app/config \
--name storagenode2 storjlabs/storagenode:latest \
--server.address=:28968 \
--console.address=:14004 \
--server.private-address=127.0.0.1:7779

or

docker run -d --restart unless-stopped --stop-timeout 300 \
-e WALLET="xyz" \
-e EMAIL="xyz@gmail.com" -e ADDRESS="xyz.no-ip.org:28968" \
-e STORAGE="4TB" --net=host \
--mount type=bind,source="/volume2/shared2/storj2/identity",destination=/app/identity \
--mount type=bind,source="/volume2/shared2/storj2",destination=/app/config \
--name storagenode2 storjlabs/storagenode:latest \
--console.address=:14004

but with changes in the config.yaml:

# public address to listen on
server.address: :28968

# private address to listen on
server.private-address: 127.0.0.1:7779

Please note that option --console.address=:14004 is required in both cases, because the port number for the dashboard is defaulted to 14002 in the entrypoint script.
You may also use an environment variables in the format STORJ_UPPER_CASED_OPTION to specify these ports.

lovaszl · January 8, 2024, 9:27pm

Thank you all for your help, the second one is already running.
I don’t trust docker’s nat, so I usually use the host’s network. It’s strange, but it also ignored the config.yaml parameters this way. It was started only after the commands entered when the container was created.

--server.address=:28968 \
--console.address=:14004 \
--server.private-address=127.0.0.1:7779