Node OFFLINE -- what's happening?

My node seems to have gone OFFLINE sometime over the past 24hrs. I am not sure exactly when. I have tried stopping/starting in multiple times but it still shows as offline. This is what I see from the log:

2020-10-28T21:59:14.720205494Z 2020-10-28T21:59:14.719Z INFO contact:service context cancelled {“Satellite ID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”}

2020-10-28T21:59:14.720732468Z 2020-10-28T21:59:14.720Z INFO contact:service context cancelled {“Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”}

2020-10-28T21:59:14.721212482Z 2020-10-28T21:59:14.720Z INFO contact:service context cancelled {“Satellite ID”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”}

2020-10-28T21:59:14.721738623Z 2020-10-28T21:59:14.721Z INFO contact:service context cancelled {“Satellite ID”: “12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB”}

2020-10-28T21:59:15.253765115Z 2020-10-28T21:59:15.253Z ERROR servers unexpected shutdown of a runner {“name”: “debug”, “error”: “debug: http: Server closed”, “errorVerbose”: “debug: http: Server closed\n\tstorj.io/private/debug.(*Server).Run.func2:108\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}

2020-10-28T21:59:15.842534466Z Error: debug: http: Server closed

2020-10-28T21:59:25.421365811Z 2020-10-28T21:59:25.417Z INFO Configuration loaded {“Location”: “/app/config/config.yaml”}

2020-10-28T21:59:25.436223796Z 2020-10-28T21:59:25.435Z INFO Operator email {“Address”: “galewis@yaddatech.com”}

2020-10-28T21:59:25.436428968Z 2020-10-28T21:59:25.436Z INFO Operator wallet {“Address”: “0x00000000000000000000000000000000000000000”}

2020-10-28T21:59:27.073321203Z 2020-10-28T21:59:27.072Z INFO Telemetry enabled

2020-10-28T21:59:27.232647289Z 2020-10-28T21:59:27.232Z INFO db.migration Database Version {“version”: 45}

2020-10-28T21:59:29.249790364Z 2020-10-28T21:59:29.249Z INFO preflight:localtime start checking local system clock with trusted satellites’ system clock.

2020-10-28T21:59:32.502132112Z 2020-10-28T21:59:32.501Z INFO preflight:localtime local system clock is in sync with trusted satellites’ system clock.

2020-10-28T21:59:32.502528540Z 2020-10-28T21:59:32.502Z INFO bandwidth Performing bandwidth usage rollups

2020-10-28T21:59:32.504717855Z 2020-10-28T21:59:32.504Z INFO Node 12mYxtBsxSKpbZrh1bwZ9kMrrPx2W9SxDu62qrJWaFHcsCb4xkV started

2020-10-28T21:59:32.504880735Z 2020-10-28T21:59:32.504Z INFO Public server started on [::]:28967

2020-10-28T21:59:32.505416751Z 2020-10-28T21:59:32.504Z INFO Private server started on 127.0.0.1:7778

2020-10-28T21:59:32.507267515Z 2020-10-28T21:59:32.506Z INFO trust Scheduling next refresh {“after”: “1h45m10.870182702s”}

2020-10-28T22:07:54.568100241Z 2020-10-28T22:07:54.567Z WARN console:service unable to get Satellite URL {“Satellite ID”: “118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW”, “error”: “storage node dashboard service error: trust: satellite "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW" is untrusted”, “errorVerbose”: “storage node dashboard service error: trust: satellite "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW" is untrusted\n\tstorj.io/storj/storagenode/trust.(*Pool).getInfo:228\n\tstorj.io/storj/storagenode/trust.(*Pool).GetNodeURL:167\n\tstorj.io/storj/storagenode/console.(*Service).GetDashboardData:168\n\tstorj.io/storj/storagenode/console/consoleapi.(*StorageNode).StorageNode:44\n\tnet/http.HandlerFunc.ServeHTTP:2042\n\tgithub.com/gorilla/mux.(*Router).ServeHTTP:210\n\tnet/http.serverHandler.ServeHTTP:2843\n\tnet/http.(*conn).serve:1925”}

2020-10-28T22:08:11.158614190Z 2020-10-28T22:08:11.157Z WARN console:service unable to get Satellite URL {“Satellite ID”: “118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW”, “error”: “storage node dashboard service error: trust: satellite "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW" is untrusted”, “errorVerbose”: “storage node dashboard service error: trust: satellite "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW" is untrusted\n\tstorj.io/storj/storagenode/trust.(*Pool).getInfo:228\n\tstorj.io/storj/storagenode/trust.(*Pool).GetNodeURL:167\n\tstorj.io/storj/storagenode/console.(*Service).GetDashboardData:168\n\tstorj.io/storj/storagenode/console/consoleapi.(*StorageNode).StorageNode:44\n\tnet/http.HandlerFunc.ServeHTTP:2042\n\tgithub.com/gorilla/mux.(*Router).ServeHTTP:210\n\tnet/http.serverHandler.ServeHTTP:2843\n\tnet/http.(*conn).serve:1925”}

It never seems to get past the “Scheduling next refresh…” message. I’ve checked my email to see whether I’ve been placed in the penalty box but I don’t see any emails regarding this so I am not sure why this is happening. I’ve also verified that my port is open.

Try to go through a check list

Hi @Alexey,

Yes, I already checked and the port is open and as I mentioned it was working a day ago (last night when I went to bed). I just noticed this a few hrs ago but I am not sure when this started. I know the WAN is up since I can ping storj.io and it responds.

The ping have nothing to do with changes of your WAN IP.
Please, make sure, that the port is open for the external address specified in the ADDRESS option (in case of docker) or in the contact.external-address: parameter in the "C:\Program Files\Storj\Storage Node\config.yaml" (in case of Windows).
Please, check it on https://www.yougetsignal.com/tools/open-ports/

Thx @Alexey, looks like they changed my WAN IP address underneath me in the last 24hrs.

1 Like

You are welcome!
I would like to recommend to use a DDNS hostname instead of public IP:
https://documentation.storj.io/dependencies/port-forwarding#setup-dynamic-dns-service-hostname-configuration