Node is OFFLINE Suddenly

raftoral · September 24, 2020, 4:20pm

Hi!

Up to yesterday, I was running a node in a Synology NAS with no problems. But now it is offline. I’ve tried to stop the container an rerun it, reboot the NAS, reboot the router but no success. Checking the logs, I see the same error:

2020-09-24T16:13:59.375Z ERROR contact:service ping satellite failed {“Satellite ID”: “118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW”, “attempts”: 11, “error”: “ping satellite error: rpccompat: dial tcp 78.94.240.189:7777: connect: connection refused”, “errorVerbose”: “ping satellite error: rpccompat: dial tcp 78.94.240.189:7777: connect: connection refused\n\tstorj.io/common/rpc.Dialer.dialTransport:211\n\tstorj.io/common/rpc.Dialer.dial:188\n\tstorj.io/common/rpc.Dialer.DialNodeURL:148\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:124\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:95\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:92\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}

Any knows where it is the problem?

Thanks

nerdatwork · September 24, 2020, 4:50pm

You can ignore that error since that satellite has shutdown. Can you show more of your log to get a better picture?

raftoral · September 24, 2020, 4:58pm

Thanks for the answer. Here it’s the log (I deleted operator email an wallet values):

2020-09-24T15:56:51.435Z 2020-09-24T15:56:51.470Z 2020-09-24T15:56:51.470Z 2020-09-24T15:56:53.050Z 2020-09-24T15:56:53.112Z 2020-09-24T15:56:54.518Z 2020-09-24T15:56:54.604Z 2020-09-24T15:56:55.345Z 2020-09-24T15:56:55.346Z 2020-09-24T15:56:55.346Z 2020-09-24T15:56:55.346Z 2020-09-24T15:56:55.346Z 2020-09-24T15:56:55.347Z 2020-09-24T15:56:55.553Z 2020-09-24T15:56:56.638Z 2020-09-24T15:56:58.710Z 2020-09-24T15:57:01.335Z 2020-09-24T15:57:02.794Z 2020-09-24T15:57:10.876Z 2020-09-24T15:57:26.970Z 2020-09-24T15:57:59.050Z 2020-09-24T15:59:03.131Z 2020-09-24T15:59:23.940Z 2020-09-24T15:59:24.031Z 2020-09-24T15:59:28.072Z 2020-09-24T16:00:58.709Z 2020-09-24T16:01:11.216Z 2020-09-24T16:05:27.297Z 2020-09-24T16:13:59.375Z 2020-09-24T16:31:03.465Z INFO Configuration loaded {“Location”: “/app/config/config.yaml”}
INFO Operator email {“Address”: “"}
INFO Operator wallet {“Address”: "”}
INFO Telemetry enabled
INFO db.migration Database Version {“version”: 45}
INFO preflight:localtime start checking local system clock with trusted satellites’ system clock.
ERROR preflight:localtime unable to get satellite system time {“Satellite ID”: “118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW”, “error”: “rpccompat: dial tcp 78.94.240.189:7777: connect: connection refused”, “errorVerbose”: “rpccompat: dial tcp 78.94.240.189:7777: connect: connection refused\n\tstorj.io/common/rpc.Dialer.dialTransport:211\n\tstorj.io/common/rpc.Dialer.dial:188\n\tstorj.io/common/rpc.Dialer.DialNodeURL:148\n\tstorj.io/storj/storagenode/preflight.(*LocalTime).getSatelliteTime:110\n\tstorj.io/storj/storagenode/preflight.(*LocalTime).Check.func1:67\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}
INFO preflight:localtime local system clock is in sync with trusted satellites’ system clock.
INFO bandwidth Performing bandwidth usage rollups
INFO Node 12oV5bdzCtFrxTequKDkkTBjH8Pd4Cf62sEfDNBPAJUUDuTT5zo started
INFO Public server started on [::]:28967
INFO Private server started on 127.0.0.1:7778
INFO trust Scheduling next refresh {“after”: “8h37m53.113494219s”}
ERROR contact:service ping satellite failed {“Satellite ID”: “118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW”, “attempts”: 1, “error”: “ping satellite error: rpccompat: dial tcp 78.94.240.189:7777: connect: connection refused”, “errorVerbose”: “ping satellite error: rpccompat: dial tcp 78.94.240.189:7777: connect: connection refused\n\tstorj.io/common/rpc.Dialer.dialTransport:211\n\tstorj.io/common/rpc.Dialer.dial:188\n\tstorj.io/common/rpc.Dialer.DialNodeURL:148\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:124\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:95\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:92\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}
ERROR contact:service ping satellite failed {“Satellite ID”: “118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW”, “attempts”: 2, “error”: “ping satellite error: rpccompat: dial tcp 78.94.240.189:7777: connect: connection refused”, “errorVerbose”: “ping satellite error: rpccompat: dial tcp 78.94.240.189:7777: connect: connection refused\n\tstorj.io/common/rpc.Dialer.dialTransport:211\n\tstorj.io/common/rpc.Dialer.dial:188\n\tstorj.io/common/rpc.Dialer.DialNodeURL:148\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:124\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:95\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:92\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}
ERROR contact:service ping satellite failed {“Satellite ID”: “118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW”, “attempts”: 3, “error”: “ping satellite error: rpccompat: dial tcp 78.94.240.189:7777: connect: connection refused”, “errorVerbose”: “ping satellite error: rpccompat: dial tcp 78.94.240.189:7777: connect: connection refused\n\tstorj.io/common/rpc.Dialer.dialTransport:211\n\tstorj.io/common/rpc.Dialer.dial:188\n\tstorj.io/common/rpc.Dialer.DialNodeURL:148\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:124\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:95\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:92\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}
ERROR nodestats:cache Get pricing-model/join date failed {“error”: “heldamount service error: unable to connect to the satellite 118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW: rpccompat: dial tcp 78.94.240.189:7777: connect: connection refused”, “errorVerbose”: “heldamount service error: unable to connect to the satellite 118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW: rpccompat: dial tcp 78.94.240.189:7777: connect: connection refused\n\tstorj.io/storj/storagenode/heldamount.(*Endpoint).dial:229\n\tstorj.io/storj/storagenode/heldamount.(*Endpoint).GetAllPaystubs:105\n\tstorj.io/storj/storagenode/nodestats.(*Cache).Run.func1:80\n\tstorj.io/storj/storagenode/nodestats.(*Cache).satelliteLoop:259\n\tstorj.io/storj/storagenode/nodestats.(*Cache).Run:79\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func1:56\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}
ERROR contact:service ping satellite failed {“Satellite ID”: “118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW”, “attempts”: 4, “error”: “ping satellite error: rpccompat: dial tcp 78.94.240.189:7777: connect: connection refused”, “errorVerbose”: “ping satellite error: rpccompat: dial tcp 78.94.240.189:7777: connect: connection refused\n\tstorj.io/common/rpc.Dialer.dialTransport:211\n\tstorj.io/common/rpc.Dialer.dial:188\n\tstorj.io/common/rpc.Dialer.DialNodeURL:148\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:124\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:95\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:92\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}
ERROR contact:service ping satellite failed {“Satellite ID”: “118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW”, “attempts”: 5, “error”: “ping satellite error: rpccompat: dial tcp 78.94.240.189:7777: connect: connection refused”, “errorVerbose”: “ping satellite error: rpccompat: dial tcp 78.94.240.189:7777: connect: connection refused\n\tstorj.io/common/rpc.Dialer.dialTransport:211\n\tstorj.io/common/rpc.Dialer.dial:188\n\tstorj.io/common/rpc.Dialer.DialNodeURL:148\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:124\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:95\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:92\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}
ERROR contact:service ping satellite failed {“Satellite ID”: “118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW”, “attempts”: 6, “error”: “ping satellite error: rpccompat: dial tcp 78.94.240.189:7777: connect: connection refused”, “errorVerbose”: “ping satellite error: rpccompat: dial tcp 78.94.240.189:7777: connect: connection refused\n\tstorj.io/common/rpc.Dialer.dialTransport:211\n\tstorj.io/common/rpc.Dialer.dial:188\n\tstorj.io/common/rpc.Dialer.DialNodeURL:148\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:124\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:95\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:92\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}
ERROR contact:service ping satellite failed {“Satellite ID”: “118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW”, “attempts”: 7, “error”: “ping satellite error: rpccompat: dial tcp 78.94.240.189:7777: connect: connection refused”, “errorVerbose”: “ping satellite error: rpccompat: dial tcp 78.94.240.189:7777: connect: connection refused\n\tstorj.io/common/rpc.Dialer.dialTransport:211\n\tstorj.io/common/rpc.Dialer.dial:188\n\tstorj.io/common/rpc.Dialer.DialNodeURL:148\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:124\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:95\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:92\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}
ERROR contact:service ping satellite failed {“Satellite ID”: “118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW”, “attempts”: 8, “error”: “ping satellite error: rpccompat: dial tcp 78.94.240.189:7777: connect: connection refused”, “errorVerbose”: “ping satellite error: rpccompat: dial tcp 78.94.240.189:7777: connect: connection refused\n\tstorj.io/common/rpc.Dialer.dialTransport:211\n\tstorj.io/common/rpc.Dialer.dial:188\n\tstorj.io/common/rpc.Dialer.DialNodeURL:148\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:124\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:95\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:92\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}
ERROR nodestats:cache Get disk space usage query failed {“error”: “node stats service error: unable to connect to the satellite 118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW: rpccompat: dial tcp 78.94.240.189:7777: connect: connection refused”, “errorVerbose”: “node stats service error: unable to connect to the satellite 118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW: rpccompat: dial tcp 78.94.240.189:7777: connect: connection refused\n\tstorj.io/storj/storagenode/nodestats.(*Service).dial:159\n\tstorj.io/storj/storagenode/nodestats.(*Service).GetDailyStorageUsage:110\n\tstorj.io/storj/storagenode/nodestats.(*Cache).CacheSpaceUsage.func1:176\n\tstorj.io/storj/storagenode/nodestats.(*Cache).satelliteLoop:259\n\tstorj.io/storj/storagenode/nodestats.(*Cache).CacheSpaceUsage:175\n\tstorj.io/storj/storagenode/nodestats.(*Cache).Run.func3:131\n\tstorj.io/common/sync2.(*Cycle).Run:92\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}
ERROR nodestats:cache heldamount err {“satellite”: “118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW”}
ERROR nodestats:cache Get held amount query failed {“error”: “heldamount service error: unable to connect to the satellite 118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW: rpccompat: dial tcp 78.94.240.189:7777: connect: connection refused”, “errorVerbose”: “heldamount service error: unable to connect to the satellite 118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW: rpccompat: dial tcp 78.94.240.189:7777: connect: connection refused\n\tstorj.io/storj/storagenode/heldamount.(*Endpoint).dial:229\n\tstorj.io/storj/storagenode/heldamount.(*Endpoint).GetPaystub:56\n\tstorj.io/storj/storagenode/nodestats.(*Cache).CacheHeldAmount.func1:202\n\tstorj.io/storj/storagenode/nodestats.(*Cache).satelliteLoop:259\n\tstorj.io/storj/storagenode/nodestats.(*Cache).CacheHeldAmount:194\n\tstorj.io/storj/storagenode/nodestats.(*Cache).Run.func3:136\n\tstorj.io/common/sync2.(*Cycle).Run:92\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}
ERROR nodestats:cache Get stats query failed {“error”: “node stats service error: unable to connect to the satellite 118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW: rpccompat: dial tcp 78.94.240.189:7777: connect: connection refused”, “errorVerbose”: “node stats service error: unable to connect to the satellite 118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW: rpccompat: dial tcp 78.94.240.189:7777: connect: connection refused\n\tstorj.io/storj/storagenode/nodestats.(*Service).dial:159\n\tstorj.io/storj/storagenode/nodestats.(*Service).GetReputationStats:66\n\tstorj.io/storj/storagenode/nodestats.(*Cache).CacheReputationStats.func1:153\n\tstorj.io/storj/storagenode/nodestats.(*Cache).satelliteLoop:259\n\tstorj.io/storj/storagenode/nodestats.(*Cache).CacheReputationStats:152\n\tstorj.io/storj/storagenode/nodestats.(*Cache).Run.func2:119\n\tstorj.io/common/sync2.(*Cycle).Run:92\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}
ERROR contact:service ping satellite failed {“Satellite ID”: “118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW”, “attempts”: 9, “error”: “ping satellite error: rpccompat: dial tcp 78.94.240.189:7777: connect: connection refused”, “errorVerbose”: “ping satellite error: rpccompat: dial tcp 78.94.240.189:7777: connect: connection refused\n\tstorj.io/common/rpc.Dialer.dialTransport:211\n\tstorj.io/common/rpc.Dialer.dial:188\n\tstorj.io/common/rpc.Dialer.DialNodeURL:148\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:124\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:95\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:92\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}
ERROR contact:service ping satellite failed {“Satellite ID”: “118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW”, “attempts”: 10, “error”: “ping satellite error: rpccompat: dial tcp 78.94.240.189:7777: connect: connection refused”, “errorVerbose”: “ping satellite error: rpccompat: dial tcp 78.94.240.189:7777: connect: connection refused\n\tstorj.io/common/rpc.Dialer.dialTransport:211\n\tstorj.io/common/rpc.Dialer.dial:188\n\tstorj.io/common/rpc.Dialer.DialNodeURL:148\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:124\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:95\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:92\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}
ERROR contact:service ping satellite failed {“Satellite ID”: “118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW”, “attempts”: 11, “error”: “ping satellite error: rpccompat: dial tcp 78.94.240.189:7777: connect: connection refused”, “errorVerbose”: “ping satellite error: rpccompat: dial tcp 78.94.240.189:7777: connect: connection refused\n\tstorj.io/common/rpc.Dialer.dialTransport:211\n\tstorj.io/common/rpc.Dialer.dial:188\n\tstorj.io/common/rpc.Dialer.DialNodeURL:148\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:124\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:95\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:92\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}
ERROR contact:service ping satellite failed {“Satellite ID”: “118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW”, “attempts”: 12, “error”: “ping satellite error: rpccompat: dial tcp 78.94.240.189:7777: connect: connection refused”, “errorVerbose”: "ping satellite error: rpccompat: dial tcp 78.94.240.189:7777: connect: connection refused\n\tstorj.io/common/rpc.Dialer.dialTransport:211\n\tstorj.io/common/rpc.Dialer.dial:188\n\tstorj.io/common/rpc.Dialer.DialNodeURL:148\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:124\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:95\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:92\n\tstorj.io/common/sync2.

raftoral · September 24, 2020, 5:15pm

Hey @nerdatwork! Now it’s online. Do you know why? Apparently, I haven’t changed anything. Thanks.

nerdatwork · September 24, 2020, 5:57pm

Satellite 118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW has shutdown so the errors related to this can be safely ignored. Unfortunately your log does not show any signs of what could have caused it to go offline. You can try this checklist in the future.