Storage node offline - ERROR contact:service ping satellite failed

gacostarua · November 6, 2021, 7:42pm

After trying to update the config.yaml file my storage nodes will not go back online. I have tried to rename and re-setup the config.yaml file but no success. any advice? here is my log file entry.

ERROR contact:service ping satellite failed {“Satellite ID”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”, “attempts”: 9, “error”: “ping satellite: check-in ratelimit: node rate limited by id”, “errorVerbose”: “ping satellite: check-in ratelimit: node rate limited by id\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:138\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:95\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:92\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}

gacostarua · November 6, 2021, 7:48pm

I am running docker on ubuntu

Alexey · November 7, 2021, 8:32am

Hello @gacostarua ,
Welcome to the forum!

If you use a docker version, you should configure it through the docker run command, the parameters from the command are overrides parameters in config.yaml.
Please, use this check-list to fix the offline issue:

gacostarua · November 7, 2021, 2:09pm

Alexey,

Thanks for getting back to me. I have gone over the check list and had reviewed earlier post on similar topics. Everything checks out - except the 28967 port issue. This port appears and not open when tested against my Public IP address. However, I do not believe it is routing or network issue because when I check the same port on the internal IP address I get the same response that the port is not open. I believe it is something internal to the machine that is running the nodes. Its as if nothing is listening on port 29867 or the connection is being refused. I have the same issue on two different machines. This all started when I added the line in the config.yaml file for zkSync.

Also rebuilt the config.yaml file, but that did not work.

here are the contents of the identity folder:

Not sure what other info you need. but here is the log file as well:

2021-11-07T14:24:12.072Z ERROR contact:service ping satellite failed {“Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “attempts”: 12, “error”: “ping satellite: failed to dial storage node (ID: 1249844g8NxsZioYVUNA69jKiQx37mqSgJPNZ1EgsQxe2pqW8yb) at address 47.22.140.198:28967: rpc: dial tcp 47.22.140.198:28967: connect: connection refused”, “errorVerbose”: “ping satellite: failed to dial storage node (ID: 1249844g8NxsZioYVUNA69jKiQx37mqSgJPNZ1EgsQxe2pqW8yb) at address 47.22.140.198:28967: rpc: dial tcp 47.22.140.198:28967: connect: connection refused\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:141\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:95\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:152\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}
2021-11-07T14:24:42.942Z ERROR contact:service ping satellite failed {“Satellite ID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”, “attempts”: 12, “error”: “ping satellite: failed to dial storage node (ID: 1249844g8NxsZioYVUNA69jKiQx37mqSgJPNZ1EgsQxe2pqW8yb) at address 47.22.140.198:28967: rpc: dial tcp 47.22.140.198:28967: connect: connection refused”, “errorVerbose”: “ping satellite: failed to dial storage node (ID: 1249844g8NxsZioYVUNA69jKiQx37mqSgJPNZ1EgsQxe2pqW8yb) at address 47.22.140.198:28967: rpc: dial tcp 47.22.140.198:28967: connect: connection refused\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:141\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:95\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:152\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}
2021-11-07T14:24:45.175Z ERROR contact:service ping satellite failed {“Satellite ID”: “12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB”, “attempts”: 12, “error”: “ping satellite: failed to dial storage node (ID: 1249844g8NxsZioYVUNA69jKiQx37mqSgJPNZ1EgsQxe2pqW8yb) at address 47.22.140.198:28967: rpc: dial tcp 47.22.140.198:28967: connect: connection refused”, “errorVerbose”: “ping satellite: failed to dial storage node (ID: 1249844g8NxsZioYVUNA69jKiQx37mqSgJPNZ1EgsQxe2pqW8yb) at address 47.22.140.198:28967: rpc: dial tcp 47.22.140.198:28967: connect: connection refused\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:141\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:95\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:152\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}
2021-11-07T14:24:45.651Z ERROR contact:service ping satellite failed {“Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “attempts”: 12, “error”: “ping satellite: failed to dial storage node (ID: 1249844g8NxsZioYVUNA69jKiQx37mqSgJPNZ1EgsQxe2pqW8yb) at address 47.22.140.198:28967: rpc: dial tcp 47.22.140.198:28967: connect: connection refused”, “errorVerbose”: “ping satellite: failed to dial storage node (ID: 1249844g8NxsZioYVUNA69jKiQx37mqSgJPNZ1EgsQxe2pqW8yb) at address 47.22.140.198:28967: rpc: dial tcp 47.22.140.198:28967: connect: connection refused\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:141\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:95\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:152\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}
2021-11-07T14:25:22.925Z WARN console:service unable to get Satellite URL {“Satellite ID”: “118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW”, “error”: “console: trust: satellite “118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW” is untrusted”, “errorVerbose”: “console: trust: satellite “118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW” is untrusted\n\tstorj.io/storj/storagenode/trust.(*Pool).getInfo:238\n\tstorj.io/storj/storagenode/trust.(*Pool).GetNodeURL:177\n\tstorj.io/storj/storagenode/console.(*Service).GetDashboardData:174\n\tstorj.io/storj/storagenode/console/consoleapi.(*StorageNode).StorageNode:45\n\tnet/http.HandlerFunc.ServeHTTP:2046\n\tgithub.com/gorilla/mux.(*Router).ServeHTTP:210\n\tnet/http.serverHandler.ServeHTTP:2878\n\tnet/http.(*conn).serve:1929”}
2021-11-07T14:25:44.427Z ERROR contact:service ping satellite failed {“Satellite ID”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”, “attempts”: 12, “error”: “ping satellite: failed to dial storage node (ID: 1249844g8NxsZioYVUNA69jKiQx37mqSgJPNZ1EgsQxe2pqW8yb) at address 47.22.140.198:28967: rpc: dial tcp 47.22.140.198:28967: connect: connection refused”, “errorVerbose”: “ping satellite: failed to dial storage node (ID: 1249844g8NxsZioYVUNA69jKiQx37mqSgJPNZ1EgsQxe2pqW8yb) at address 47.22.140.198:28967: rpc: dial tcp 47.22.140.198:28967: connect: connection refused\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:141\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:95\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:152\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}

gacostarua · November 7, 2021, 2:36pm

Alexey,

Thanks for getting back to me. I have gone over the check list and had reviewed earlier post on similar topics. Everything checks out - except the 28967 port issue. This port appears and not open when tested against my Public IP address. However, I do not believe it is routing or network issue because when I check the same port on the internal IP address I get the same response that the port is not open. I believe it is something internal to the machine that is running the nodes. Its as if nothing is listening on port 29867 or the connection is being refused. I have the same issue on two different machines. This all started when I added the line in the config.yaml file for zkSync.

Also rebuilt the config.yaml file, but that did not work.

here are the contents of the identity folder:

Not sure what other info you need. but here is the log file as well:

2021-11-07T14:24:12.072Z ERROR contact:service ping satellite failed {“Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “attempts”: 12, “error”: “ping satellite: failed to dial storage node (ID: 1249844g8NxsZioYVUNA69jKiQx37mqSgJPNZ1EgsQxe2pqW8yb) at address 47.22.140.198:28967: rpc: dial tcp 47.22.140.198:28967: connect: connection refused”, “errorVerbose”: “ping satellite: failed to dial storage node (ID: 1249844g8NxsZioYVUNA69jKiQx37mqSgJPNZ1EgsQxe2pqW8yb) at address 47.22.140.198:28967: rpc: dial tcp 47.22.140.198:28967: connect: connection refused\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:141\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:95\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:152\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}
2021-11-07T14:24:42.942Z ERROR contact:service ping satellite failed {“Satellite ID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”, “attempts”: 12, “error”: “ping satellite: failed to dial storage node (ID: 1249844g8NxsZioYVUNA69jKiQx37mqSgJPNZ1EgsQxe2pqW8yb) at address 47.22.140.198:28967: rpc: dial tcp 47.22.140.198:28967: connect: connection refused”, “errorVerbose”: “ping satellite: failed to dial storage node (ID: 1249844g8NxsZioYVUNA69jKiQx37mqSgJPNZ1EgsQxe2pqW8yb) at address 47.22.140.198:28967: rpc: dial tcp 47.22.140.198:28967: connect: connection refused\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:141\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:95\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:152\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}
2021-11-07T14:24:45.175Z ERROR contact:service ping satellite failed {“Satellite ID”: “12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB”, “attempts”: 12, “error”: “ping satellite: failed to dial storage node (ID: 1249844g8NxsZioYVUNA69jKiQx37mqSgJPNZ1EgsQxe2pqW8yb) at address 47.22.140.198:28967: rpc: dial tcp 47.22.140.198:28967: connect: connection refused”, “errorVerbose”: “ping satellite: failed to dial storage node (ID: 1249844g8NxsZioYVUNA69jKiQx37mqSgJPNZ1EgsQxe2pqW8yb) at address 47.22.140.198:28967: rpc: dial tcp 47.22.140.198:28967: connect: connection refused\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:141\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:95\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:152\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}
2021-11-07T14:24:45.651Z ERROR contact:service ping satellite failed {“Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “attempts”: 12, “error”: “ping satellite: failed to dial storage node (ID: 1249844g8NxsZioYVUNA69jKiQx37mqSgJPNZ1EgsQxe2pqW8yb) at address 47.22.140.198:28967: rpc: dial tcp 47.22.140.198:28967: connect: connection refused”, “errorVerbose”: “ping satellite: failed to dial storage node (ID: 1249844g8NxsZioYVUNA69jKiQx37mqSgJPNZ1EgsQxe2pqW8yb) at address 47.22.140.198:28967: rpc: dial tcp 47.22.140.198:28967: connect: connection refused\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:141\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:95\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:152\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}
2021-11-07T14:25:22.925Z WARN console:service unable to get Satellite URL {“Satellite ID”: “118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW”, “error”: “console: trust: satellite “118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW” is untrusted”, “errorVerbose”: “console: trust: satellite “118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW” is untrusted\n\tstorj.io/storj/storagenode/trust.(*Pool).getInfo:238\n\tstorj.io/storj/storagenode/trust.(*Pool).GetNodeURL:177\n\tstorj.io/storj/storagenode/console.(*Service).GetDashboardData:174\n\tstorj.io/storj/storagenode/console/consoleapi.(*StorageNode).StorageNode:45\n\tnet/http.HandlerFunc.ServeHTTP:2046\n\tgithub.com/gorilla/mux.(*Router).ServeHTTP:210\n\tnet/http.serverHandler.ServeHTTP:2878\n\tnet/http.(*conn).serve:1929”}
2021-11-07T14:25:44.427Z ERROR contact:service ping satellite failed {“Satellite ID”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”, “attempts”: 12, “error”: “ping satellite: failed to dial storage node (ID: 1249844g8NxsZioYVUNA69jKiQx37mqSgJPNZ1EgsQxe2pqW8yb) at address 47.22.140.198:28967: rpc: dial tcp 47.22.140.198:28967: connect: connection refused”, “errorVerbose”: “ping satellite: failed to dial storage node (ID: 1249844g8NxsZioYVUNA69jKiQx37mqSgJPNZ1EgsQxe2pqW8yb) at address 47.22.140.198:28967: rpc: dial tcp 47.22.140.198:28967: connect: connection refused\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:141\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:95\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:152\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}

I can share with you my docker run command but nothing has changed there. same as it has always been.

Alexey · November 7, 2021, 6:36pm

The node should be running when you check this.
You can easily check the routing (or more like ISP) issue by comparing the WAN IP from the status page of your router with IP there: Open Port Check Tool - Test Port Forwarding on Your Router
These IPs must match, otherwise the port forwarding will not work.
If they do not match, then you have a few options:

call to your ISP and ask for public IP, it could be dynamic, but must be public (the WAN IP will match the IP on yougetsignal); you can say that you need an access to your security IP cam from anywhere to do not explain Storj;
switch the ISP to the one who offer a public IP;
use VPN services with port forwarding options such as portmap.io, ngrok, PIA, AirVPN, PureVPN, etc.

gacostarua · November 7, 2021, 7:09pm

The node is running and the IP address does match. This has never been a problem until I made the change to the config.yaml file.

I cannot even communicate with the docker node via the internal IP address on port 28967.

I have a static IP address and I even NATed it to my internal IP. Other ports such as 14002 are showing as open.

Do the logs tell you anything or is there any info I can DM you? I would prefer not to expose my IP addess.

Alexey · November 7, 2021, 7:57pm

The logs tell me that your node is offline. Please, double check that your external address on yougetsignal and WAN IP are match to

Check that local IP of your PC is match the IP in your port forwarding rule.
I would recommend to rollback your changes in config.yaml. As an edge case you can rename it and run the setup step again - it will re-create the config.yaml and you would use only parameters in your docker run command to configure your node.

gacostarua · November 7, 2021, 10:02pm

I have already deleted and recreated the config.yaml file multiple times. I have also moved to a different IP address (public and local). See images below. What am I missing?

deathlessdd · November 8, 2021, 12:58am

Is it possible that your local IP isnt the same?

peem · November 8, 2021, 5:12am

You have:

28967/tcp

It should be:

0.0.0.0:28967->28967/tcp

Something is missing from the “docker run…” command.
Please show it here (instead of wallet, email type xxx)

gacostarua · November 8, 2021, 1:16pm

docker run -d --restart unless-stopped -p 14002:14002 -e WALLET=0xXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX -e EMAIL=xxxxxxx@xxxxxx.com -e ADDRESS=47.22.140.198:28967 -e BANDWIDTH=50TB -e STORAGE=7.7TB --mount type=bind,source=/home/user1/.local/share/storj/identity/storagenode,destination=/app/identity --mount type=bind,source=/home/user1/STORJ,destination=/app/config --name storagenode storjlabs/storagenode:latest

CutieePie · November 8, 2021, 1:26pm

you are missing the docker container 28967 mapping, so the container will not expose port 28967 to the network…

add ;

-p 28967:28967/tcp -p 28967:28967/udp

so your command like ;

docker run -d --restart unless-stopped \
-p 14002:14002 \
-p 28967:28967/tcp \
-p 28967:28967/udp \
-e WALLET=0xXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX \
-e EMAIL=xxxxxxx@xxxxxx.com \
-e ADDRESS=47.22.140.198:28967 \
-e STORAGE=7.7TB \
--mount type=bind,source=/home/user1/.local/share/storj/identity/storagenode,destination=/app/identity  \
--mount type=bind,source=/home/user1/STORJ,destination=/app/config \
--name storagenode storjlabs/storagenode:latest

#sorry the forum is mucking up the command, so don’t copy and paste what I’ve put >< but you should get the idea

#edit : remove BANDWIDTH, it does nothing…

#edit: parameters like address, and storage should be enclosed in double quotes “”

#Edit from @Alexey : replaced ` to two lines with ``` to make the formatting better

gacostarua · November 8, 2021, 1:41pm

I implemented those changes but still OFFLINE. Here is my edited docker run script.

docker run -d --restart unless-stopped -p 14002:14002 -p 28967:28967/tcp -p 28967:28967/udp -e WALLET=“0xXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX” -e EMAIL="xxxxxx@xxxxx.com" -e ADDRESS=“47.22.140.198:28967” -e STORAGE=“7.7TB” --mount type=bind,source=/home/user1/.local/share/storj/identity/storagenode,destination=/app/identity --mount type=bind,source=/home/user1/STORJ,destination=/app/config --name storagenode storjlabs/storagenode:latest

peem · November 8, 2021, 1:55pm

docker logs storagenode --tail 50

gacostarua · November 8, 2021, 2:02pm

2021-11-08T13:39:22.560Z ERROR contact:service ping satellite failed {“Satellite ID”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”, “attempts”: 6, “error”: “ping satellite: failed to dial storage node (ID: 1249844g8NxsZioYVUNA69jKiQx37mqSgJPNZ1EgsQxe2pqW8yb) at address 47.22.140.198:28967: rpc: dial tcp 47.22.140.198:28967: connect: connection refused”, “errorVerbose”: “ping satellite: failed to dial storage node (ID: 1249844g8NxsZioYVUNA69jKiQx37mqSgJPNZ1EgsQxe2pqW8yb) at address 47.22.140.198:28967: rpc: dial tcp 47.22.140.198:28967: connect: connection refused\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:141\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:95\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:92\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}
2021-11-08T13:39:24.258Z ERROR contact:service ping satellite failed {“Satellite ID”: “12tRQrMTWUWwzwGh18i7Fqs67kmdhH9t6aToeiwbo5mfS2rUmo”, “attempts”: 6, “error”: “ping satellite: failed to dial storage node (ID: 1249844g8NxsZioYVUNA69jKiQx37mqSgJPNZ1EgsQxe2pqW8yb) at address 47.22.140.198:28967: rpc: dial tcp 47.22.140.198:28967: connect: connection refused”, “errorVerbose”: “ping satellite: failed to dial storage node (ID: 1249844g8NxsZioYVUNA69jKiQx37mqSgJPNZ1EgsQxe2pqW8yb) at address 47.22.140.198:28967: rpc: dial tcp 47.22.140.198:28967: connect: connection refused\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:141\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:95\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:92\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}
2021-11-08T13:39:24.750Z ERROR contact:service ping satellite failed {“Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “attempts”: 6, “error”: “ping satellite: failed to dial storage node (ID: 1249844g8NxsZioYVUNA69jKiQx37mqSgJPNZ1EgsQxe2pqW8yb) at address 47.22.140.198:28967: rpc: dial tcp 47.22.140.198:28967: connect: connection refused”, “errorVerbose”: “ping satellite: failed to dial storage node (ID: 1249844g8NxsZioYVUNA69jKiQx37mqSgJPNZ1EgsQxe2pqW8yb) at address 47.22.140.198:28967: rpc: dial tcp 47.22.140.198:28967: connect: connection refused\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:141\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:95\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:92\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}
2021-11-08T13:39:25.073Z ERROR contact:service ping satellite failed {“Satellite ID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”, “attempts”: 6, “error”: “ping satellite: check-in ratelimit: node rate limited by id”, “errorVerbose”: “ping satellite: check-in ratelimit: node rate limited by id\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:138\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:95\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:92\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}
2021-11-08T13:39:25.399Z ERROR contact:service ping satellite failed {“Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “attempts”: 6, “error”: “ping satellite: failed to dial storage node (ID: 1249844g8NxsZioYVUNA69jKiQx37mqSgJPNZ1EgsQxe2pqW8yb) at address 47.22.140.198:28967: rpc: dial tcp 47.22.140.198:28967: connect: connection refused”, “errorVerbose”: “ping satellite: failed to dial storage node (ID: 1249844g8NxsZioYVUNA69jKiQx37mqSgJPNZ1EgsQxe2pqW8yb) at address 47.22.140.198:28967: rpc: dial tcp 47.22.140.198:28967: connect: connection refused\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:141\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:95\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:92\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}
2021-11-08T13:39:25.747Z ERROR contact:service ping satellite failed {“Satellite ID”: “12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB”, “attempts”: 6, “error”: “ping satellite: check-in ratelimit: node rate limited by id”, “errorVerbose”: “ping satellite: check-in ratelimit: node rate limited by id\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:138\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:95\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:92\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}
2021-11-08T13:39:55.282Z ERROR contact:service ping satellite failed {“Satellite ID”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”, “attempts”: 7, “error”: “ping satellite: failed to dial storage node (ID: 1249844g8NxsZioYVUNA69jKiQx37mqSgJPNZ1EgsQxe2pqW8yb) at address 47.22.140.198:28967: rpc: dial tcp 47.22.140.198:28967: connect: connection refused”, “errorVerbose”: “ping satellite: failed to dial storage node (ID: 1249844g8NxsZioYVUNA69jKiQx37mqSgJPNZ1EgsQxe2pqW8yb) at address 47.22.140.198:28967: rpc: dial tcp 47.22.140.198:28967: connect: connection refused\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:141\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:95\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:92\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}
2021-11-08T13:39:56.423Z ERROR contact:service ping satellite failed {“Satellite ID”: “12tRQrMTWUWwzwGh18i7Fqs67kmdhH9t6aToeiwbo5mfS2rUmo”, “attempts”: 7, “error”: “ping satellite: check-in ratelimit: node rate limited by id”, “errorVerbose”: “ping satellite: check-in ratelimit: node rate limited by id\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:138\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:95\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:92\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}
2021-11-08T13:39:57.038Z ERROR contact:service ping satellite failed {“Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “attempts”: 7, “error”: “ping satellite: failed to dial storage node (ID: 1249844g8NxsZioYVUNA69jKiQx37mqSgJPNZ1EgsQxe2pqW8yb) at address 47.22.140.198:28967: rpc: dial tcp 47.22.140.198:28967: connect: connection refused”, “errorVerbose”: “ping satellite: failed to dial storage node (ID: 1249844g8NxsZioYVUNA69jKiQx37mqSgJPNZ1EgsQxe2pqW8yb) at address 47.22.140.198:28967: rpc: dial tcp 47.22.140.198:28967: connect: connection refused\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:141\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:95\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:92\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}
2021-11-08T13:39:57.402Z ERROR contact:service ping satellite failed {“Satellite ID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”, “attempts”: 7, “error”: “ping satellite: check-in ratelimit: node rate limited by id”, “errorVerbose”: “ping satellite: check-in ratelimit: node rate limited by id\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:138\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:95\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:92\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}
2021-11-08T13:39:57.847Z ERROR contact:service ping satellite failed {“Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “attempts”: 7, “error”: “ping satellite: failed to dial storage node (ID: 1249844g8NxsZioYVUNA69jKiQx37mqSgJPNZ1EgsQxe2pqW8yb) at address 47.22.140.198:28967: rpc: dial tcp 47.22.140.198:28967: connect: connection refused”, “errorVerbose”: “ping satellite: failed to dial storage node (ID: 1249844g8NxsZioYVUNA69jKiQx37mqSgJPNZ1EgsQxe2pqW8yb) at address 47.22.140.198:28967: rpc: dial tcp 47.22.140.198:28967: connect: connection refused\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:141\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:95\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:92\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}
2021-11-08T13:39:58.164Z ERROR contact:service ping satellite failed {“Satellite ID”: “12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB”, “attempts”: 7, “error”: “ping satellite: check-in ratelimit: node rate limited by id”, “errorVerbose”: “ping satellite: check-in ratelimit: node rate limited by id\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:138\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:95\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:92\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}
2021-11-08T13:41:00.215Z ERROR contact:service ping satellite failed {“Satellite ID”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”, “attempts”: 8, “error”: “ping satellite: failed to dial storage node (ID: 1249844g8NxsZioYVUNA69jKiQx37mqSgJPNZ1EgsQxe2pqW8yb) at address 47.22.140.198:28967: rpc: dial tcp 47.22.140.198:28967: connect: connection refused”, “errorVerbose”: “ping satellite: failed to dial storage node (ID: 1249844g8NxsZioYVUNA69jKiQx37mqSgJPNZ1EgsQxe2pqW8yb) at address 47.22.140.198:28967: rpc: dial tcp 47.22.140.198:28967: connect: connection refused\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:141\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:95\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:92\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}
2021-11-08T13:41:00.534Z ERROR contact:service ping satellite failed {“Satellite ID”: “12tRQrMTWUWwzwGh18i7Fqs67kmdhH9t6aToeiwbo5mfS2rUmo”, “attempts”: 8, “error”: “ping satellite: check-in ratelimit: node rate limited by id”, “errorVerbose”: “ping satellite: check-in ratelimit: node rate limited by id\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:138\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:95\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:92\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}
2021-11-08T13:41:01.241Z ERROR contact:service ping satellite failed {“Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “attempts”: 8, “error”: “ping satellite: failed to dial storage node (ID: 1249844g8NxsZioYVUNA69jKiQx37mqSgJPNZ1EgsQxe2pqW8yb) at address 47.22.140.198:28967: rpc: dial tcp 47.22.140.198:28967: connect: connection refused”, “errorVerbose”: “ping satellite: failed to dial storage node (ID: 1249844g8NxsZioYVUNA69jKiQx37mqSgJPNZ1EgsQxe2pqW8yb) at address 47.22.140.198:28967: rpc: dial tcp 47.22.140.198:28967: connect: connection refused\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:141\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:95\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:92\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}
2021-11-08T13:41:01.800Z ERROR contact:service ping satellite failed {“Satellite ID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”, “attempts”: 8, “error”: “ping satellite: failed to dial storage node (ID: 1249844g8NxsZioYVUNA69jKiQx37mqSgJPNZ1EgsQxe2pqW8yb) at address 47.22.140.198:28967: rpc: dial tcp 47.22.140.198:28967: connect: connection refused”, “errorVerbose”: “ping satellite: failed to dial storage node (ID: 1249844g8NxsZioYVUNA69jKiQx37mqSgJPNZ1EgsQxe2pqW8yb) at address 47.22.140.198:28967: rpc: dial tcp 47.22.140.198:28967: connect: connection refused\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:141\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:95\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:92\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}
2021-11-08T13:41:02.249Z ERROR contact:service ping satellite failed {“Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “attempts”: 8, “error”: “ping satellite: failed to dial storage node (ID: 1249844g8NxsZioYVUNA69jKiQx37mqSgJPNZ1EgsQxe2pqW8yb) at address 47.22.140.198:28967: rpc: dial tcp 47.22.140.198:28967: connect: connection refused”, “errorVerbose”: “ping satellite: failed to dial storage node (ID: 1249844g8NxsZioYVUNA69jKiQx37mqSgJPNZ1EgsQxe2pqW8yb) at address 47.22.140.198:28967: rpc: dial tcp 47.22.140.198:28967: connect: connection refused\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:141\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:95\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:92\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}
2021-11-08T13:41:02.720Z ERROR contact:service ping satellite failed {“Satellite ID”: “12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB”, “attempts”: 8, “error”: “ping satellite: failed to dial storage node (ID: 1249844g8NxsZioYVUNA69jKiQx37mqSgJPNZ1EgsQxe2pqW8yb) at address 47.22.140.198:28967: rpc: dial tcp 47.22.140.198:28967: connect: connection refused”, “errorVerbose”: “ping satellite: failed to dial storage node (ID: 1249844g8NxsZioYVUNA69jKiQx37mqSgJPNZ1EgsQxe2pqW8yb) at address 47.22.140.198:28967: rpc: dial tcp 47.22.140.198:28967: connect: connection refused\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:141\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:95\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:92\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}
2021-11-08T13:43:13.663Z ERROR contact:service ping satellite failed {“Satellite ID”: “12tRQrMTWUWwzwGh18i7Fqs67kmdhH9t6aToeiwbo5mfS2rUmo”, “attempts”: 9, “error”: “ping satellite: check-in ratelimit: node rate limited by id”, “errorVerbose”: “ping satellite: check-in ratelimit: node rate limited by id\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:138\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:95\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:92\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}
2021-11-08T13:43:14.171Z ERROR contact:service ping satellite failed {“Satellite ID”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”, “attempts”: 9, “error”: “ping satellite: failed to dial storage node (ID: 1249844g8NxsZioYVUNA69jKiQx37mqSgJPNZ1EgsQxe2pqW8yb) at address 47.22.140.198:28967: rpc: dial tcp 47.22.140.198:28967: connect: connection refused”, “errorVerbose”: “ping satellite: failed to dial storage node (ID: 1249844g8NxsZioYVUNA69jKiQx37mqSgJPNZ1EgsQxe2pqW8yb) at address 47.22.140.198:28967: rpc: dial tcp 47.22.140.198:28967: connect: connection refused\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:141\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:95\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:92\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}
2021-11-08T13:43:14.495Z ERROR contact:service ping satellite failed {“Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “attempts”: 9, “error”: “ping satellite: failed to dial storage node (ID: 1249844g8NxsZioYVUNA69jKiQx37mqSgJPNZ1EgsQxe2pqW8yb) at address 47.22.140.198:28967: rpc: dial tcp 47.22.140.198:28967: connect: connection refused”, “errorVerbose”: “ping satellite: failed to dial storage node (ID: 1249844g8NxsZioYVUNA69jKiQx37mqSgJPNZ1EgsQxe2pqW8yb) at address 47.22.140.198:28967: rpc: dial tcp 47.22.140.198:28967: connect: connection refused\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:141\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:95\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:92\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}
2021-11-08T13:43:15.216Z ERROR contact:service ping satellite failed {“Satellite ID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”, “attempts”: 9, “error”: “ping satellite: failed to dial storage node (ID: 1249844g8NxsZioYVUNA69jKiQx37mqSgJPNZ1EgsQxe2pqW8yb) at address 47.22.140.198:28967: rpc: dial tcp 47.22.140.198:28967: connect: connection refused”, “errorVerbose”: “ping satellite: failed to dial storage node (ID: 1249844g8NxsZioYVUNA69jKiQx37mqSgJPNZ1EgsQxe2pqW8yb) at address 47.22.140.198:28967: rpc: dial tcp 47.22.140.198:28967: connect: connection refused\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:141\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:95\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:92\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}
2021-11-08T13:43:15.693Z ERROR contact:service ping satellite failed {“Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “attempts”: 9, “error”: “ping satellite: failed to dial storage node (ID: 1249844g8NxsZioYVUNA69jKiQx37mqSgJPNZ1EgsQxe2pqW8yb) at address 47.22.140.198:28967: rpc: dial tcp 47.22.140.198:28967: connect: connection refused”, “errorVerbose”: “ping satellite: failed to dial storage node (ID: 1249844g8NxsZioYVUNA69jKiQx37mqSgJPNZ1EgsQxe2pqW8yb) at address 47.22.140.198:28967: rpc: dial tcp 47.22.140.198:28967: connect: connection refused\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:141\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:95\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:92\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}
2021-11-08T13:43:16.137Z ERROR contact:service ping satellite failed {“Satellite ID”: “12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB”, “attempts”: 9, “error”: “ping satellite: check-in ratelimit: node rate limited by id”, “errorVerbose”: “ping satellite: check-in ratelimit: node rate limited by id\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:138\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:95\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:92\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}
2021-11-08T13:45:53.037Z WARN console:service unable to get Satellite URL {“Satellite ID”: “118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW”, “error”: “console: trust: satellite “118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW” is untrusted”, “errorVerbose”: “console: trust: satellite “118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW” is untrusted\n\tstorj.io/storj/storagenode/trust.(*Pool).getInfo:238\n\tstorj.io/storj/storagenode/trust.(*Pool).GetNodeURL:177\n\tstorj.io/storj/storagenode/console.(*Service).GetDashboardData:174\n\tstorj.io/storj/storagenode/console/consoleapi.(*StorageNode).StorageNode:45\n\tnet/http.HandlerFunc.ServeHTTP:2046\n\tgithub.com/gorilla/mux.(*Router).ServeHTTP:210\n\tnet/http.serverHandler.ServeHTTP:2878\n\tnet/http.(*conn).serve:1929”}
2021-11-08T13:47:29.945Z ERROR contact:service ping satellite failed {“Satellite ID”: “12tRQrMTWUWwzwGh18i7Fqs67kmdhH9t6aToeiwbo5mfS2rUmo”, “attempts”: 10, “error”: “ping satellite: failed to dial storage node (ID: 1249844g8NxsZioYVUNA69jKiQx37mqSgJPNZ1EgsQxe2pqW8yb) at address 47.22.140.198:28967: rpc: dial tcp 47.22.140.198:28967: connect: connection refused”, “errorVerbose”: “ping satellite: failed to dial storage node (ID: 1249844g8NxsZioYVUNA69jKiQx37mqSgJPNZ1EgsQxe2pqW8yb) at address 47.22.140.198:28967: rpc: dial tcp 47.22.140.198:28967: connect: connection refused\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:141\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:95\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:92\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}
2021-11-08T13:47:30.766Z ERROR contact:service ping satellite failed {“Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “attempts”: 10, “error”: “ping satellite: failed to dial storage node (ID: 1249844g8NxsZioYVUNA69jKiQx37mqSgJPNZ1EgsQxe2pqW8yb) at address 47.22.140.198:28967: rpc: dial tcp 47.22.140.198:28967: connect: connection refused”, “errorVerbose”: “ping satellite: failed to dial storage node (ID: 1249844g8NxsZioYVUNA69jKiQx37mqSgJPNZ1EgsQxe2pqW8yb) at address 47.22.140.198:28967: rpc: dial tcp 47.22.140.198:28967: connect: connection refused\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:141\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:95\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:92\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}
2021-11-08T13:47:31.128Z ERROR contact:service ping satellite failed {“Satellite ID”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”, “attempts”: 10, “error”: “ping satellite: failed to dial storage node (ID: 1249844g8NxsZioYVUNA69jKiQx37mqSgJPNZ1EgsQxe2pqW8yb) at address 47.22.140.198:28967: rpc: dial tcp 47.22.140.198:28967: connect: connection refused”, “errorVerbose”: “ping satellite: failed to dial storage node (ID: 1249844g8NxsZioYVUNA69jKiQx37mqSgJPNZ1EgsQxe2pqW8yb) at address 47.22.140.198:28967: rpc: dial tcp 47.22.140.198:28967: connect: connection refused\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:141\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:95\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:92\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}
2021-11-08T13:47:31.529Z ERROR contact:service ping satellite failed {“Satellite ID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”, “attempts”: 10, “error”: “ping satellite: check-in ratelimit: node rate limited by id”, “errorVerbose”: “ping satellite: check-in ratelimit: node rate limited by id\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:138\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:95\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:92\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}
2021-11-08T13:47:32.155Z ERROR contact:service ping satellite failed {“Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “attempts”: 10, “error”: “ping satellite: failed to dial storage node (ID: 1249844g8NxsZioYVUNA69jKiQx37mqSgJPNZ1EgsQxe2pqW8yb) at address 47.22.140.198:28967: rpc: dial tcp 47.22.140.198:28967: connect: connection refused”, “errorVerbose”: “ping satellite: failed to dial storage node (ID: 1249844g8NxsZioYVUNA69jKiQx37mqSgJPNZ1EgsQxe2pqW8yb) at address 47.22.140.198:28967: rpc: dial tcp 47.22.140.198:28967: connect: connection refused\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:141\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:95\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:92\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}
2021-11-08T13:47:32.692Z ERROR contact:service ping satellite failed {“Satellite ID”: “12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB”, “attempts”: 10, “error”: “ping satellite: failed to dial storage node (ID: 1249844g8NxsZioYVUNA69jKiQx37mqSgJPNZ1EgsQxe2pqW8yb) at address 47.22.140.198:28967: rpc: dial tcp 47.22.140.198:28967: connect: connection refused”, “errorVerbose”: “ping satellite: failed to dial storage node (ID: 1249844g8NxsZioYVUNA69jKiQx37mqSgJPNZ1EgsQxe2pqW8yb) at address 47.22.140.198:28967: rpc: dial tcp 47.22.140.198:28967: connect: connection refused\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:141\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:95\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:92\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}
2021-11-08T13:56:02.128Z ERROR contact:service ping satellite failed {“Satellite ID”: “12tRQrMTWUWwzwGh18i7Fqs67kmdhH9t6aToeiwbo5mfS2rUmo”, “attempts”: 11, “error”: “ping satellite: failed to dial storage node (ID: 1249844g8NxsZioYVUNA69jKiQx37mqSgJPNZ1EgsQxe2pqW8yb) at address 47.22.140.198:28967: rpc: dial tcp 47.22.140.198:28967: connect: connection refused”, “errorVerbose”: “ping satellite: failed to dial storage node (ID: 1249844g8NxsZioYVUNA69jKiQx37mqSgJPNZ1EgsQxe2pqW8yb) at address 47.22.140.198:28967: rpc: dial tcp 47.22.140.198:28967: connect: connection refused\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:141\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:95\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:92\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}
2021-11-08T13:56:03.298Z ERROR contact:service ping satellite failed {“Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “attempts”: 11, “error”: “ping satellite: failed to dial storage node (ID: 1249844g8NxsZioYVUNA69jKiQx37mqSgJPNZ1EgsQxe2pqW8yb) at address 47.22.140.198:28967: rpc: dial tcp 47.22.140.198:28967: connect: connection refused”, “errorVerbose”: “ping satellite: failed to dial storage node (ID: 1249844g8NxsZioYVUNA69jKiQx37mqSgJPNZ1EgsQxe2pqW8yb) at address 47.22.140.198:28967: rpc: dial tcp 47.22.140.198:28967: connect: connection refused\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:141\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:95\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:92\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}
2021-11-08T13:56:04.059Z ERROR contact:service ping satellite failed {“Satellite ID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”, “attempts”: 11, “error”: “ping satellite: failed to dial storage node (ID: 1249844g8NxsZioYVUNA69jKiQx37mqSgJPNZ1EgsQxe2pqW8yb) at address 47.22.140.198:28967: rpc: dial tcp 47.22.140.198:28967: connect: connection refused”, “errorVerbose”: “ping satellite: failed to dial storage node (ID: 1249844g8NxsZioYVUNA69jKiQx37mqSgJPNZ1EgsQxe2pqW8yb) at address 47.22.140.198:28967: rpc: dial tcp 47.22.140.198:28967: connect: connection refused\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:141\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:95\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:92\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}
2021-11-08T13:56:04.127Z ERROR contact:service ping satellite failed {“Satellite ID”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”, “attempts”: 11, “error”: “ping satellite: failed to dial storage node (ID: 1249844g8NxsZioYVUNA69jKiQx37mqSgJPNZ1EgsQxe2pqW8yb) at address 47.22.140.198:28967: rpc: dial tcp 47.22.140.198:28967: connect: connection refused”, “errorVerbose”: “ping satellite: failed to dial storage node (ID: 1249844g8NxsZioYVUNA69jKiQx37mqSgJPNZ1EgsQxe2pqW8yb) at address 47.22.140.198:28967: rpc: dial tcp 47.22.140.198:28967: connect: connection refused\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:141\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:95\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:92\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}
2021-11-08T13:56:04.569Z ERROR contact:service ping satellite failed {“Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “attempts”: 11, “error”: “ping satellite: failed to dial storage node (ID: 1249844g8NxsZioYVUNA69jKiQx37mqSgJPNZ1EgsQxe2pqW8yb) at address 47.22.140.198:28967: rpc: dial tcp 47.22.140.198:28967: connect: connection refused”, “errorVerbose”: “ping satellite: failed to dial storage node (ID: 1249844g8NxsZioYVUNA69jKiQx37mqSgJPNZ1EgsQxe2pqW8yb) at address 47.22.140.198:28967: rpc: dial tcp 47.22.140.198:28967: connect: connection refused\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:141\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:95\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:92\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}
2021-11-08T13:56:05.235Z ERROR contact:service ping satellite failed {“Satellite ID”: “12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB”, “attempts”: 11, “error”: “ping satellite: failed to dial storage node (ID: 1249844g8NxsZioYVUNA69jKiQx37mqSgJPNZ1EgsQxe2pqW8yb) at address 47.22.140.198:28967: rpc: dial tcp 47.22.140.198:28967: connect: connection refused”, “errorVerbose”: “ping satellite: failed to dial storage node (ID: 1249844g8NxsZioYVUNA69jKiQx37mqSgJPNZ1EgsQxe2pqW8yb) at address 47.22.140.198:28967: rpc: dial tcp 47.22.140.198:28967: connect: connection refused\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:141\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:95\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:92\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}
user1@SJ1-PowerEdge-T30:~$

gacostarua · November 8, 2021, 2:15pm

HEY - I got it fixed! Your config changes were correct. My last issue was the config.yaml file was missing from my debugging. Nodes are back online. THANK YOU!