Both nodes remain OFFLINE

Node Operators troubleshooting
1

This is a mess. I’ve got two storage nodes that responded to the update exactly the same.

The bottom line is that neither storage node will go ONLINE. They both remain OFFLINE.

And, of course, I already tried restarting docker, rebooting, etc. All of the standard troubleshooting procedures…ports are open.

2020-01-24T06:45:40.610Z INFO Configuration loaded from: /app/config/config.yaml
2020-01-24T06:45:40.626Z DEBUG debug server listening on 127.0.0.1:39139
2020-01-24T06:45:40.677Z INFO Operator email: xxxxxx.xxxxxxx@hotmail.com
2020-01-24T06:45:40.678Z INFO operator wallet: 0xC00dB75C207258b3e8860d4DB320EC78648208C1
2020-01-24T06:45:40.764Z DEBUG Binary Version: v0.30.5 with CommitHash 72805d55ca6dcbc5dd32883e64538d77fcb35b8a, built at 2020-01-21 15:53:01 +0000 UTC as Release true
2020-01-24T06:45:41.194Z DEBUG version allowed minimum version from control server is: v0.29.0
2020-01-24T06:45:41.194Z INFO version running on version v0.30.5
2020-01-24T06:45:41.198Z DEBUG telemetry Initialized batcher with id = “12tyV5u2PeEdwGg7zhu9V4qm51LC3BT6cohf4LRKY8R6emgec3B”
2020-01-24T06:45:41.298Z INFO db.migration Database Version {“version”: 28}
2020-01-24T06:45:41.868Z DEBUG trust Fetched URLs from source; updating cache {“source”: “https://tardigrade.io/trusted-satellites”, “count”: 4}
2020-01-24T06:45:41.912Z DEBUG trust Satellite is trusted {“id”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”}
2020-01-24T06:45:41.912Z DEBUG trust Satellite is trusted {“id”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”}
2020-01-24T06:45:41.913Z DEBUG trust Satellite is trusted {“id”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”}
2020-01-24T06:45:41.913Z DEBUG trust Satellite is trusted {“id”: “118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW”}
2020-01-24T06:45:41.913Z INFO preflight:localtime start checking local system clock with trusted satellites’ system clock.
2020-01-24T06:45:42.565Z INFO preflight:localtime local system clock is in sync with trusted satellites’ system clock.
2020-01-24T06:45:42.565Z DEBUG gracefulexit:chore checking pending exits
2020-01-24T06:45:42.565Z DEBUG gracefulexit:chore no satellites found
2020-01-24T06:45:42.566Z DEBUG pieces:trash starting emptying trash
2020-01-24T06:45:42.566Z INFO bandwidth Performing bandwidth usage rollups
2020-01-24T06:45:42.567Z INFO trust Scheduling next refresh {“after”: “7h34m4.912222422s”}
2020-01-24T06:45:42.567Z INFO Node 12tyV5u2PeEdwGg7zhu9V4qm51LC3BT6cohf4LRKY8R6emgec3B started
2020-01-24T06:45:42.567Z INFO Public server started on [::]:28968
2020-01-24T06:45:42.567Z INFO Private server started on 127.0.0.1:7778
2020-01-24T06:45:42.617Z INFO piecestore:monitor Remaining Bandwidth {“bytes”: 99974345163264}
2020-01-24T06:45:42.618Z DEBUG version allowed minimum version from control server is: v0.29.0
2020-01-24T06:45:42.618Z INFO version running on version v0.30.5
2020-01-24T06:45:42.622Z DEBUG contact:chore Starting cycle {“Satellite ID”: “118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW”}
2020-01-24T06:45:42.629Z DEBUG contact:chore Starting cycle {“Satellite ID”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”}
2020-01-24T06:45:42.630Z DEBUG contact:chore Starting cycle {“Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”}
2020-01-24T06:45:42.630Z DEBUG contact:chore Starting cycle {“Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”}
2020-01-24T06:46:25.424Z DEBUG orders sending
2020-01-24T06:46:25.425Z DEBUG orders no orders to send
2020-01-24T06:46:42.350Z INFO Got a signal from the OS: “terminated”
2020-01-24T06:52:38.925Z INFO Configuration loaded from: /app/config/config.yaml
2020-01-24T06:52:38.963Z DEBUG debug server listening on 127.0.0.1:43333
2020-01-24T06:52:38.988Z INFO Operator email: xxxxxx.xxxxxxxx@hotmail.com
2020-01-24T06:52:38.988Z INFO operator wallet: 0xC00dB75C207258b3e8860d4DB320EC78648208C1
2020-01-24T06:52:39.101Z DEBUG Binary Version: v0.30.5 with CommitHash 72805d55ca6dcbc5dd32883e64538d77fcb35b8a, built at 2020-01-21 15:53:01 +0000 UTC as Release true
2020-01-24T06:52:41.502Z DEBUG version allowed minimum version from control server is: v0.29.0
2020-01-24T06:52:41.502Z INFO version running on version v0.30.5
2020-01-24T06:52:41.510Z DEBUG telemetry Initialized batcher with id = “12tyV5u2PeEdwGg7zhu9V4qm51LC3BT6cohf4LRKY8R6emgec3B”
2020-01-24T06:52:41.687Z INFO db.migration Database Version {“version”: 28}
2020-01-24T06:52:42.953Z DEBUG trust Fetched URLs from source; updating cache {“source”: “https://tardigrade.io/trusted-satellites”, “count”: 4}
2020-01-24T06:52:42.968Z DEBUG trust Satellite is trusted {“id”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”}
2020-01-24T06:52:42.968Z DEBUG trust Satellite is trusted {“id”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”}
2020-01-24T06:52:42.968Z DEBUG trust Satellite is trusted {“id”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”}
2020-01-24T06:52:42.968Z DEBUG trust Satellite is trusted {“id”: “118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW”}
2020-01-24T06:52:42.968Z INFO preflight:localtime start checking local system clock with trusted satellites’ system clock.
2020-01-24T06:52:44.036Z INFO preflight:localtime local system clock is in sync with trusted satellites’ system clock.
2020-01-24T06:52:44.036Z DEBUG gracefulexit:chore checking pending exits
2020-01-24T06:52:44.037Z DEBUG gracefulexit:chore no satellites found
2020-01-24T06:52:44.037Z DEBUG pieces:trash starting emptying trash
2020-01-24T06:52:44.037Z INFO bandwidth Performing bandwidth usage rollups
2020-01-24T06:52:44.037Z INFO trust Scheduling next refresh {“after”: “4h49m15.571198169s”}
2020-01-24T06:52:44.037Z INFO Node 12tyV5u2PeEdwGg7zhu9V4qm51LC3BT6cohf4LRKY8R6emgec3B started
2020-01-24T06:52:44.037Z INFO Public server started on [::]:28968
2020-01-24T06:52:44.037Z INFO Private server started on 127.0.0.1:7778
2020-01-24T06:52:44.089Z INFO piecestore:monitor Remaining Bandwidth {“bytes”: 99974345163264}
2020-01-24T06:52:44.092Z DEBUG contact:chore Starting cycle {“Satellite ID”: “118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW”}
2020-01-24T06:52:44.092Z DEBUG contact:chore Starting cycle {“Satellite ID”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”}
2020-01-24T06:52:44.092Z DEBUG contact:chore Starting cycle {“Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”}
2020-01-24T06:52:44.092Z DEBUG contact:chore Starting cycle {“Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”}
2020-01-24T06:52:44.103Z DEBUG version allowed minimum version from control server is: v0.29.0
2020-01-24T06:52:44.103Z INFO version running on version v0.30.5
2020-01-24T06:54:29.104Z DEBUG orders sending
2020-01-24T06:54:29.104Z DEBUG orders no orders to send
2020-01-24T06:55:17.237Z DEBUG orders cleaning
2020-01-24T06:55:17.316Z DEBUG orders cleanup finished {“items deleted”: 393}
2020-01-24T07:01:40.945Z INFO Got a signal from the OS: “terminated”
2020-01-24T07:02:54.351Z INFO Configuration loaded from: /app/config/config.yaml
2020-01-24T07:02:54.357Z DEBUG debug server listening on 127.0.0.1:32787
2020-01-24T07:02:54.388Z INFO Operator email: xxxxxx.xxxxxxxx@hotmail.com
2020-01-24T07:02:54.389Z INFO operator wallet: 0xC00dB75C207258b3e8860d4DB320EC78648208C1
2020-01-24T07:02:54.441Z DEBUG Binary Version: v0.30.5 with CommitHash 72805d55ca6dcbc5dd32883e64538d77fcb35b8a, built at 2020-01-21 15:53:01 +0000 UTC as Release true
2020-01-24T07:02:54.840Z DEBUG version allowed minimum version from control server is: v0.29.0
2020-01-24T07:02:54.840Z INFO version running on version v0.30.5
2020-01-24T07:02:54.843Z DEBUG telemetry Initialized batcher with id = “12tyV5u2PeEdwGg7zhu9V4qm51LC3BT6cohf4LRKY8R6emgec3B”
2020-01-24T07:02:54.892Z INFO db.migration Database Version {“version”: 28}
2020-01-24T07:02:55.413Z DEBUG trust Fetched URLs from source; updating cache {“source”: “https://tardigrade.io/trusted-satellites”, “count”: 4}
2020-01-24T07:02:55.428Z DEBUG trust Satellite is trusted {“id”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”}
2020-01-24T07:02:55.428Z DEBUG trust Satellite is trusted {“id”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”}
2020-01-24T07:02:55.428Z DEBUG trust Satellite is trusted {“id”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”}
2020-01-24T07:02:55.428Z DEBUG trust Satellite is trusted {“id”: “118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW”}
2020-01-24T07:02:55.428Z INFO preflight:localtime start checking local system clock with trusted satellites’ system clock.
2020-01-24T07:02:56.060Z INFO preflight:localtime local system clock is in sync with trusted satellites’ system clock.
2020-01-24T07:02:56.060Z DEBUG gracefulexit:chore checking pending exits
2020-01-24T07:02:56.060Z DEBUG pieces:trash starting emptying trash
2020-01-24T07:02:56.060Z DEBUG gracefulexit:chore no satellites found
2020-01-24T07:02:56.061Z INFO bandwidth Performing bandwidth usage rollups
2020-01-24T07:02:56.061Z INFO trust Scheduling next refresh {“after”: “5h49m28.882416702s”}
2020-01-24T07:02:56.061Z INFO Node 12tyV5u2PeEdwGg7zhu9V4qm51LC3BT6cohf4LRKY8R6emgec3B started
2020-01-24T07:02:56.061Z INFO Public server started on [::]:28968
2020-01-24T07:02:56.061Z INFO Private server started on 127.0.0.1:7778
2020-01-24T07:02:56.074Z INFO piecestore:monitor Remaining Bandwidth {“bytes”: 99974345163264}
2020-01-24T07:02:56.078Z DEBUG contact:chore Starting cycle {“Satellite ID”: “118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW”}
2020-01-24T07:02:56.079Z DEBUG contact:chore Starting cycle {“Satellite ID”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”}
2020-01-24T07:02:56.079Z DEBUG contact:chore Starting cycle {“Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”}
2020-01-24T07:02:56.079Z DEBUG contact:chore Starting cycle {“Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”}
2020-01-24T07:02:56.110Z DEBUG version allowed minimum version from control server is: v0.29.0
2020-01-24T07:02:56.110Z INFO version running on version v0.30.5
2020-01-24T07:05:12.556Z DEBUG orders sending
2020-01-24T07:05:12.556Z DEBUG orders no orders to send
2020-01-24T07:07:26.638Z DEBUG orders cleaning
2020-01-24T07:07:26.648Z DEBUG orders cleanup finished {“items deleted”: 85}

2

it looks like you activated debud log level in config
I am no see any error here.

3

First turn off debug mode by changing log.level: debug to log.level: info
Also this should be a separate thread about your issue.

After changing log level to info, show your last 20 log entries docker logs --tail 20 storagenode

4

Got up this morning and restarted both storage nodes. BAM! They instantly came online. Weird. I didn’t make any changes. This incident is now closed and will remain in the x-files. :slight_smile: