My node is frequently misconfigured randomly as it was working fine since last 2 year

[iBhadresh]

I am facing node misconfiguration issue since last 3-4 month. It is automatically up and down randomly. I have not changed any configuration since last 2 year.

I am running following commands but nothing happen.

docker stop -t 300 storagenode

docker rm storagenode

docker rmi storjlabs/storagenode:latest

docker pull storjlabs/storagenode:latest

docker run -d --restart …rest of the command

…

I am not getting what exactly the issue is. I have checked port forwarding rules and all the stuffs and it is fine.

[nerdatwork]

What do the last 20 lines of the log show ?

[iBhadresh]

@nerdatwork

the last 20 lines of the log show ?

root@xxx:/home/xxx# docker stop -t 300 storagenode
storagenode
root@xxxx:/home/xxx# docker rm storagenode
storagenode
root@xxxx:/home/xxx# docker rmi storjlabs/storagenode:latest
Untagged: storjlabs/storagenode:latest
Untagged: storjlabs/storagenode@sha256:21ca0ac819d1d66240830d28097eeb693923e51383aa3bb07bf823f9fad1f6e9
Deleted: sha256:4c1e65b147c42b70a513cadd66d660871aa7f3d644fb2dde89a5210535d48c9d
Deleted: sha256:63ce7216dcb272318af1012deb945db3b00e61a9cb7fc361232ab261e10997a6
Deleted: sha256:3c926d2e409a98a88ff9e7a95f2ca436e64e15a2a9cb98d5feeb591485b195c0
Deleted: sha256:79803e8b35c11470c196a99da8aebc7879e41fe46f37b93b71a38aec5fff2a90
Deleted: sha256:2697da522ccbc8a64bd96cb3a1d5e10063cbee1ff0eb8d50719fec81812d8799
Deleted: sha256:912232c569eb02a6f41ac2feb8d379748ea83c381181d5a3bc5fd4ed3955e5e2
Deleted: sha256:37491d217800b372de5cb522c6bd758a73d6dd9c9a89a9b25aea5769f245b5da
Deleted: sha256:473e6db13411463cea9e81eb4b5c25824bbc88d26012547ccbf47b5b7ee85490
Deleted: sha256:850a31e7da89bcf120282e4f48be90ab4315a574609d053b22fda2348e1f5cc5
Deleted: sha256:7dfa50f06fccf924c8673e1a6db3851ce54b5b44bc8a409778e1e4775ced9665
Deleted: sha256:8d853c8add5d1e7b0aafc4b68a3d9fb8e7a0da27970c2acf831fe63be4a0cd2c
root@xxxx:/home/xxx# docker pull storjlabs/storagenode:latest
latest: Pulling from storjlabs/storagenode
302e3ee49805: Pull complete
85560c5f5016: Pull complete
9630d5e636d4: Pull complete
c2616dfe100a: Pull complete
819b67ea6dcc: Pull complete
6ef36da4c671: Pull complete
7186d708415f: Pull complete
a6d12fda6e63: Pull complete
93f309ebcc23: Pull complete
4f4fb700ef54: Pull complete
Digest: sha256:21ca0ac819d1d66240830d28097eeb693923e51383aa3bb07bf82fdfdfdfsfsf
Status: Downloaded newer image for storjlabs/storagenode:latest
docker.io/storjlabs/storagenode:latest
root@xxxx:/home/xxxx# docker run -d --restart unless-stopped --stop-timeout 300 -p 28967:28967/tcp -p 28967:28967/udp -p 127.0.0.1:14002:14002 -e WALLET=“0x1FEE62dxxxzcxc859054a7d87d0B9aC1B55D12” -e EMAIL="fvfdf@gmail.com" -e ADDRESS=“afsf.ddns.net:28967” -e STORAGE=“1.58TB” --user $(id -u):$(id -g) --mount type=bind,source=“/home/xx/.local/share/storj/identity/storagenode”,destination=/app/identity --mount type=bind,source=“/home/storage”,destination=/app/config --name storagenode storjlabs/storagenode:latest
1d0df09dc37f1f33343a3223c9f108022ca2c51b15bcvbdgz8v14782fa22e0c

[nerdatwork]

How do I check my logs? - Storj Docs

You posted your commands and its results. Check above link on how to view the logs.

[iBhadresh]

2024-10-28T12:12:47Z ERROR contact:service ping satellite failed {“Process”: “storagenode”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “attempts”: 7, “error”: “ping satellite: check-in ratelimit: node rate limited by id”, “errorVerbose”: “ping satellite: check-in ratelimit: node rate limited by id\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:202\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:156\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:99\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78”}

2024-10-28T12:12:51Z ERROR contact:service ping satellite failed {“Process”: “storagenode”, “Satellite ID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”, “attempts”: 8, “error”: “ping satellite: check-in ratelimit: node rate limited by id”, “errorVerbose”: “ping satellite: check-in ratelimit: node rate limited by id\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:202\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:156\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:99\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78”}

2024-10-28T12:12:58Z ERROR contact:service ping satellite failed {“Process”: “storagenode”, “Satellite ID”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”, “attempts”: 9, “error”: “ping satellite: failed to ping storage node, your node indicated error code: 0, rpc: tcp connector failed: rpc: dial tcp 43.250.157.159:28967: connect: connection refused”, “errorVerbose”: “ping satellite: failed to ping storage node, your node indicated error code: 0, rpc: tcp connector failed: rpc: dial tcp 43.250.157.159:28967: connect: connection refused\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:208\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:156\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:99\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78”}

2024-10-28T12:13:51Z ERROR contact:service ping satellite failed {“Process”: “storagenode”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “attempts”: 8, “error”: “ping satellite: check-in ratelimit: node rate limited by id”, “errorVerbose”: “ping satellite: check-in ratelimit: node rate limited by id\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:202\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:156\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:99\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78”}

2024-10-28T12:14:12Z ERROR contact:service ping satellite failed {“Process”: “storagenode”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “attempts”: 9, “error”: “ping satellite: failed to ping storage node, your node indicated error code: 0, rpc: tcp connector failed: rpc: dial tcp 43.250.157.159:28967: connect: connection refused”, “errorVerbose”: “ping satellite: failed to ping storage node, your node indicated error code: 0, rpc: tcp connector failed: rpc: dial tcp 43.250.157.159:28967: connect: connection refused\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:208\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:156\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:99\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78”}

2024-10-28T12:15:00Z ERROR contact:service ping satellite failed {“Process”: “storagenode”, “Satellite ID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”, “attempts”: 9, “error”: “ping satellite: check-in ratelimit: node rate limited by id”, “errorVerbose”: “ping satellite: check-in ratelimit: node rate limited by id\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:202\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:156\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:99\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78”}

2024-10-28T12:16:00Z ERROR contact:service ping satellite failed {“Process”: “storagenode”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “attempts”: 9, “error”: “ping satellite: check-in ratelimit: node rate limited by id”, “errorVerbose”: “ping satellite: check-in ratelimit: node rate limited by id\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:202\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:156\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:99\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78”}

2024-10-28T12:17:36Z ERROR contact:service ping satellite failed {“Process”: “storagenode”, “Satellite ID”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”, “attempts”: 10, “error”: “ping satellite: failed to ping storage node, your node indicated error code: 0, rpc: tcp connector failed: rpc: dial tcp 43.250.157.159:28967: connect: connection refused”, “errorVerbose”: “ping satellite: failed to ping storage node, your node indicated error code: 0, rpc: tcp connector failed: rpc: dial tcp 43.250.157.159:28967: connect: connection refused\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:208\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:156\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:99\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78”}

2024-10-28T12:19:06Z ERROR contact:service ping satellite failed {“Process”: “storagenode”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “attempts”: 10, “error”: “ping satellite: failed to ping storage node, your node indicated error code: 0, rpc: tcp connector failed: rpc: dial tcp 43.250.157.159:28967: connect: connection refused”, “errorVerbose”: “ping satellite: failed to ping storage node, your node indicated error code: 0, rpc: tcp connector failed: rpc: dial tcp 43.250.157.159:28967: connect: connection refused\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:208\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:156\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:99\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78”}

2024-10-28T12:19:22Z ERROR contact:service ping satellite failed {“Process”: “storagenode”, “Satellite ID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”, “attempts”: 10, “error”: “ping satellite: failed to ping storage node, your node indicated error code: 0, rpc: tcp connector failed: rpc: dial tcp 43.250.157.159:28967: connect: connection refused”, “errorVerbose”: “ping satellite: failed to ping storage node, your node indicated error code: 0, rpc: tcp connector failed: rpc: dial tcp 43.250.157.159:28967: connect: connection refused\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:208\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:156\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:99\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78”}

2024-10-28T12:20:23Z ERROR contact:service ping satellite failed {“Process”: “storagenode”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “attempts”: 10, “error”: “ping satellite: failed to ping storage node, your node indicated error code: 0, rpc: tcp connector failed: rpc: dial tcp 43.250.157.159:28967: connect: connection refused”, “errorVerbose”: “ping satellite: failed to ping storage node, your node indicated error code: 0, rpc: tcp connector failed: rpc: dial tcp 43.250.157.159:28967: connect: connection refused\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:208\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:156\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:99\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78”}

2024-10-28T12:22:11Z INFO Downloading versions. {“Process”: “storagenode-updater”, “Server Address”: “https://version.storj.io”}

2024-10-28T12:22:12Z INFO Current binary version {“Process”: “storagenode-updater”, “Service”: “storagenode”, “Version”: “v1.114.6”}

2024-10-28T12:22:12Z INFO Version is up to date {“Process”: “storagenode-updater”, “Service”: “storagenode”}

2024-10-28T12:22:12Z INFO Current binary version {“Process”: “storagenode-updater”, “Service”: “storagenode-updater”, “Version”: “v1.114.6”}

2024-10-28T12:22:12Z INFO Version is up to date {“Process”: “storagenode-updater”, “Service”: “storagenode-updater”}

2024-10-28T12:26:14Z ERROR contact:service ping satellite failed {“Process”: “storagenode”, “Satellite ID”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”, “attempts”: 11, “error”: “ping satellite: failed to ping storage node, your node indicated error code: 0, rpc: tcp connector failed: rpc: dial tcp 43.250.157.159:28967: connect: connection refused”, “errorVerbose”: “ping satellite: failed to ping storage node, your node indicated error code: 0, rpc: tcp connector failed: rpc: dial tcp 43.250.157.159:28967: connect: connection refused\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:208\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:156\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:99\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78”}

2024-10-28T12:27:53Z ERROR contact:service ping satellite failed {“Process”: “storagenode”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “attempts”: 11, “error”: “ping satellite: failed to ping storage node, your node indicated error code: 0, rpc: tcp connector failed: rpc: dial tcp 43.250.157.159:28967: connect: connection refused”, “errorVerbose”: “ping satellite: failed to ping storage node, your node indicated error code: 0, rpc: tcp connector failed: rpc: dial tcp 43.250.157.159:28967: connect: connection refused\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:208\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:156\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:99\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78”}

2024-10-28T12:28:17Z ERROR contact:service ping satellite failed {“Process”: “storagenode”, “Satellite ID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”, “attempts”: 11, “error”: “ping satellite: failed to ping storage node, your node indicated error code: 0, rpc: tcp connector failed: rpc: dial tcp 43.250.157.159:28967: connect: connection refused”, “errorVerbose”: “ping satellite: failed to ping storage node, your node indicated error code: 0, rpc: tcp connector failed: rpc: dial tcp 43.250.157.159:28967: connect: connection refused\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:208\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:156\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:99\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78”}

2024-10-28T12:29:17Z ERROR contact:service ping satellite failed {“Process”: “storagenode”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “attempts”: 11, “error”: “ping satellite: failed to ping storage node, your node indicated error code: 0, rpc: tcp connector failed: rpc: dial tcp 43.250.157.159:28967: connect: connection refused”, “errorVerbose”: “ping satellite: failed to ping storage node, your node indicated error code: 0, rpc: tcp connector failed: rpc: dial tcp 43.250.157.159:28967: connect: connection refused\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:208\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:156\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:99\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78”}

[nerdatwork]

Follow this checklist

My Node is OFFLINE for 2 hours

It should connect in seconds. There are a few things you should check. Start with checking the port. On https://www.yougetsignal.com/tools/open-ports/?port=28967 you can fill in the domain or IP you are using in the run command. Then check the port. If it’s closed, it could still be several things. Make sure the WAN IP you see in your router settings matches the IP shown on the above page. Make sure your port forward is set up correctly. That means no filter on incoming IPs, forwarding to th…

Make sure your DDNS resolves to your ip using

nslookup afsf.ddns.net 8.8.8.8

Henceforth whenever you are posting your logs please enclose your log lines between 3 back ticks ```

[iBhadresh]

I have followed this checklist and my port is open.

Also I have checked nslookup for my DDNS and it is pointing to same ip in the log.

root@xxxx:/home/xxx# nslookup afsf.ddns.net 8.8.8.8
Server:		8.8.8.8
Address:	8.8.8.8#53

Non-authoritative answer:
Name:	afsf.ddns.net
Address: 43.250.157.159

Also I have checked the command
docker exec -it storagenode ls -l /app/identity and it shows me 5 files

root@zxx:/home/xxx# docker exec -it storagenode ls -l /app/identity
total 20

-rw-r--r-- 1 1000 1000  558 Mar 24  2022 ca.164811118.cert
-rw-r--r-- 1 1000 1000 1088 Mar 24  2022 ca.cert
-rw-r--r-- 1 1000 1000 1100 Mar 24  2022 identity. 164811118.cert
-rw-r--r-- 1 1000 1000 1630 Mar 24  2022 identity.cert
-rw------- 1 1000 1000  241 Mar 24  2022 identity.key

Let me know if I am missing something.

[andrew2.hart]

@iBhadresh

node misconfiguration

Just checking … are you referring to the QUIC misconfigured warning?

[Alexey]

Your public IP and IP in DDNS are different. Please check your DDNS updater. It’s better to configure it directly on your router in the DDNS section. You may click there the Update button to check and assign a correct public IP.

Please also compare the IP from yougetsignal and WAN IP on your router, they must match, otherwise the port forwarding will not work (because your ISP placed you behind their NAT, it’s called CGNAT).
If they did so, you would need to contact them to rollback their change for your account and give you a public IP. It could be dynamic, but must be public (the WAN IP will match IP from yougetsignal).