Node crash since december 2023

[Aniel]

Hello here,

Since december my node crash and crash raspberry pi as the same time. I have no idea why but it since decembre 2023.

docker logs storagenode 2>&1 | grep GET_AUDIT | grep failed | grep -v open -c
0
docker logs storagenode 2>&1 | grep GET_AUDIT | grep downloaded -c
0
docker logs storagenode 2>&1 | grep ‘“GET”’ | grep failed -c
0
docker logs storagenode 2>&1 | grep ‘“GET”’ | grep downloaded -c
0
docker logs storagenode 2>&1 | grep ‘“PUT”’ | grep failed -c
0
docker logs storagenode 2>&1 | grep ‘“PUT”’ | grep uploaded -c
0
docker logs storagenode 2>&1 | grep GET_REPAIR | grep failed -c
0
docker logs storagenode 2>&1 | grep GET_REPAIR | grep downloaded -c
0
docker logs storagenode 2>&1 | grep PUT_REPAIR | grep failed -c
0
docker logs storagenode 2>&1 | grep PUT_REPAIR | grep uploaded -c
0

in logs I see some errors on Invalid configuration file key {“Process”: “storagenode-updater”, “Key”:…
WARN received SIGTERM indicating exit request
2024-01-20 16:15:33,958 INFO stopped: storagenode-updater (exit status 0)
2024-01-20 16:15:35,707 INFO stopped: storagenode (exit status 0)
2024-01-20 16:15:35,709 INFO stopped: processes-exit-eventlistener (terminated by SIGTERM)

and I think that is just before crash :

2024-01-19 20:11:34,981 INFO exited: storagenode (exit status 1; not expected)
2024-01-19 20:11:35,984 INFO spawned: ‘storagenode’ with pid 43
2024-01-19 20:11:35,984 WARN received SIGQUIT indicating exit request
2024-01-19 20:11:35,985 INFO waiting for storagenode, processes-exit-eventlistener, storagenode-updater to die
2024-01-19T20:11:35Z INFO Got a signal from the OS: “terminated” {“Process”: “storagenode-updater”}
2024-01-19 20:11:35,986 INFO stopped: storagenode-updater (exit status 0)
2024-01-20 06:16:08,977 WARN killing ‘storagenode’ (43) with SIGKILL

[daki82]

These are just the logs from the updater, do you have the logs redirected from the node?
does the raspi reboot or something else?

Please describe the setup a bit, hardware, cluster size, filesystem, disk, raspberry? size of ram etc.

i guess you have to dig in the logs from the raspi as well.
(boot from sd card?) maybe corrupted? Awsome- @Alexey i summon you!

(here ends my linux/raspi knowledge)

[Alexey]

Hello @Aniel,
Welcome back!

Please search for FATAL errors in the logs:

docker logs storagenode 2>&1 | grep FATAL

If you redirected logs to the file, then you need to search for errors there, @daki82 is right, your logs seems contains only output from the storagenode-updater service, not from the node. This is usually mean, that you redirected logs to the file.

[Aniel]

I chack on my storagenode by running “docker inspect storagenode” log are aon “LogPath”: “/var/lib/docker/containers/”
I have this:
{“log”:“2024-01-21 01:36:39,174 INFO success: storagenode-updater entered RUNNING state, process has stayed up for \u003e than 1 seconds (startsecs)\n”,“stream”:“stdout”,“time”:“2024-01-21T01:36:39.174079857Z”}
{“log”:“2024-01-21 01:36:39,706 INFO exited: storagenode (exit status 1; not expected)\n”,“stream”:“stdout”,“time”:“2024-01-21T01:36:39.706915246Z”}
{“log”:“2024-01-21 01:36:40,709 INFO spawned: ‘storagenode’ with pid 43\n”,“stream”:“stdout”,“time”:“2024-01-21T01:36:40.710352689Z”}
{“log”:“2024-01-21 01:36:40,710 WARN received SIGQUIT indicating exit request\n”,“stream”:“stdout”,“time”:“2024-01-21T01:36:40.710658152Z”}
{“log”:“2024-01-21 01:36:40,710 INFO waiting for storagenode, processes-exit-eventlistener, storagenode-updater to die\n”,“stream”:“stdout”,“time”:“2024-01-21T01:36:40.711197115Z”}
{“log”:“2024-01-21T01:36:40Z\u0009INFO\u0009Got a signal from the OS: "terminated"\u0009{"Process": "storagenode-updater"}\n”,“stream”:“stdout”,“time”:“2024-01-21T01:36:40.711348986Z”}
{“log”:“2024-01-21 01:36:40,711 INFO stopped: storagenode-updater (exit status 0)\n”,“stream”:“stdout”,“time”:“2024-01-21T01:36:40.711989782Z”}
{“log”:“2024-01-21 01:36:42,806 INFO stopped: storagenode (exit status 1)\n”,“stream”:“stdout”,“time”:“2024-01-21T01:36:42.807621929Z”}
{“log”:“2024-01-21 01:36:42,807 INFO stopped: processes-exit-eventlistener (terminated by SIGTERM)\n”,“stream”:“stdout”,“time”:“2024-01-21T01:36:42.807661559Z”}
{“log”:“2024-01-21 01:36:43,592 INFO Set uid to user 0 succeeded\n”,“stream”:“stdout”,“time”:“2024-01-21T01:36:43.59285154Z”}
{“log”:“2024-01-21 01:36:43,598 INFO RPC interface ‘supervisor’ initialized\n”,“stream”:“stdout”,“time”:“2024-01-21T01:36:43.598740854Z”}
{“log”:“2024-01-21 01:36:43,598 INFO supervisord started with pid 1\n”,“stream”:“stdout”,“time”:“2024-01-21T01:36:43.598893762Z”}
{“log”:“2024-01-21 01:36:44,601 INFO spawned: ‘processes-exit-eventlistener’ with pid 11\n”,“stream”:“stdout”,“time”:“2024-01-21T01:36:44.601590872Z”}
{“log”:“2024-01-21 01:36:44,603 INFO spawned: ‘storagenode’ with pid 12\n”,“stream”:“stdout”,“time”:“2024-01-21T01:36:44.604458261Z”}
{“log”:“2024-01-21 01:36:44,606 INFO spawned: ‘storagenode-updater’ with pid 13\n”,“stream”:“stdout”,“time”:“2024-01-21T01:36:44.608793076Z”}
{“log”:“2024-01-21T01:36:44Z\u0009INFO\u0009Configuration loaded\u0009{"Process": "storagenode-updater", "Location": "/app/config/config.yaml"}\n”,“stream”:“stdout”,“time”:“2024-01-21T01:36:44.617828261Z”}
{“log”:“2024-01-21T01:36:44Z\u0009INFO\u0009Invalid configuration file key\u0009{"Process": "storagenode-updater", "Key": "server.address"}\n”,“stream”:“stdout”,“time”:“2024-01-21T01:36:44.617883224Z”}
{“log”:“2024-01-21T01:36:44Z\u0009INFO\u0009Invalid configuration file key\u0009{"Process": "storagenode-updater", "Key": "server.debug-log-traffic"}\n”,“stream”:“stdout”,“time”:“2024-01-21T01:36:44.617891743Z”}
{“log”:“2024-01-21T01:36:44Z\u0009INFO\u0009Invalid configuration file key\u0009{"Process": "storagenode-updater", "Key": "server.private-address"}\n”,“stream”:“stdout”,“time”:“2024-01-21T01:36:44.61789715Z”}
{“log”:“2024-01-21T01:36:44Z\u0009INFO\u0009Invalid configuration file key\u0009{"Process": "storagenode-updater", "Key": "contact.external-address"}\n”,“stream”:“stdout”,“time”:“2024-01-21T01:36:44.617902298Z”}
{“log”:“2024-01-21T01:36:44Z\u0009INFO\u0009Invalid configuration file key\u0009{"Process": "storagenode-updater", "Key": "storage.allocated-bandwidth"}\n”,“stream”:“stdout”,“time”:“2024-01-21T01:36:44.617907354Z”}
{“log”:“2024-01-21T01:36:44Z\u0009INFO\u0009Invalid configuration file key\u0009{"Process": "storagenode-updater", "Key": "operator.email"}\n”,“stream”:“stdout”,“time”:“2024-01-21T01:36:44.617918558Z”}
{“log”:“2024-01-21T01:36:44Z\u0009INFO\u0009Invalid configuration file key\u0009{"Process": "storagenode-updater", "Key": "storage2.database-dir"}\n”,“stream”:“stdout”,“time”:“2024-01-21T01:36:44.617924335Z”}
{“log”:“2024-01-21T01:36:44Z\u0009INFO\u0009Invalid configuration file key\u0009{"Process": "storagenode-updater", "Key": "operator.wallet"}\n”,“stream”:“stdout”,“time”:“2024-01-21T01:36:44.61793528Z”}
{“log”:“2024-01-21T01:36:44Z\u0009INFO\u0009Invalid configuration file key\u0009{"Process": "storagenode-updater", "Key": "storage.allocated-disk-space"}\n”,“stream”:“stdout”,“time”:“2024-01-21T01:36:44.617940558Z”}
{“log”:“2024-01-21T01:36:44Z\u0009INFO\u0009Invalid configuration file value for key\u0009{"Process": "storagenode-updater", "Key": "log.output"}\n”,“stream”:“stdout”,“time”:“2024-01-21T01:36:44.61794652Z”}
{“log”:“2024-01-21T01:36:44Z\u0009INFO\u0009Anonymized tracing enabled\u0009{"Process": "storagenode-updater"}\n”,“stream”:“stdout”,“time”:“2024-01-21T01:36:44.618296835Z”}
{“log”:“2024-01-21T01:36:44Z\u0009INFO\u0009Running on version\u0009{"Process": "storagenode-updater", "Service": "storagenode-updater", "Version": "v1.94.2"}\n”,“stream”:“stdout”,“time”:“2024-01-21T01:36:44.627536076Z”}
{“log”:“2024-01-21T01:36:44Z\u0009INFO\u0009Downloading versions.\u0009{"Process": "storagenode-updater", "Server Address": "https://version.storj.io"}\n”,“stream”:“stdout”,“time”:“2024-01-21T01:36:44.627951983Z”}
{“log”:“2024-01-21T01:36:45Z\u0009INFO\u0009Current binary version\u0009{"Process": "storagenode-updater", "Service": "storagenode", "Version": "v1.94.2"}\n”,“stream”:“stdout”,“time”:“2024-01-21T01:36:45.002597594Z”}
{“log”:“2024-01-21T01:36:45Z\u0009INFO\u0009New version is being rolled out but hasn’t made it to this node yet\u0009{"Process": "storagenode-updater", "Service": "storagenode"}\n”,“stream”:“stdout”,“time”:“2024-01-21T01:36:45.002659835Z”}
{“log”:“2024-01-21T01:36:45Z\u0009INFO\u0009Current binary version\u0009{"Process": "storagenode-updater", "Service": "storagenode-updater", "Version": "v1.94.2"}\n”,“stream”:“stdout”,“time”:“2024-01-21T01:36:45.007806446Z”}
{“log”:“2024-01-21T01:36:45Z\u0009INFO\u0009New version is being rolled out but hasn’t made it to this node yet\u0009{"Process": "storagenode-updater", "Service": "storagenode-updater"}\n”,“stream”:“stdout”,“time”:“2024-01-21T01:36:45.007849446Z”}
{“log”:“2024-01-21 01:36:46,009 INFO success: processes-exit-eventlistener entered RUNNING state, process has stayed up for \u003e than 1 seconds (startsecs)\n”,“stream”:“stdout”,“time”:“2024-01-21T01:36:46.009171353Z”}
{“log”:“2024-01-21 01:36:46,009 INFO success: storagenode entered RUNNING state, process has stayed up for \u003e than 1 seconds (startsecs)\n”,“stream”:“stdout”,“time”:“2024-01-21T01:36:46.009295872Z”}
{“log”:“2024-01-21 01:36:46,009 INFO success: storagenode-updater entered RUNNING state, process has stayed up for \u003e than 1 seconds (startsecs)\n”,“stream”:“stdout”,“time”:“2024-01-21T01:36:46.009384205Z”}
{“log”:“2024-01-21 05:56:55,606 INFO exited: storagenode (exit status 1; not expected)\n”,“stream”:“stdout”,“time”:“2024-01-21T05:56:55.6066183Z”}
{“log”:“2024-01-21 05:56:56,609 INFO spawned: ‘storagenode’ with pid 43\n”,“stream”:“stdout”,“time”:“2024-01-21T05:56:56.609701911Z”}
{“log”:“2024-01-21 05:56:56,609 WARN received SIGQUIT indicating exit request\n”,“stream”:“stdout”,“time”:“2024-01-21T05:56:56.609797744Z”}
{“log”:“2024-01-21 05:56:56,610 INFO waiting for storagenode, processes-exit-eventlistener, storagenode-updater to die\n”,“stream”:“stdout”,“time”:“2024-01-21T05:56:56.610370355Z”}
{“log”:“2024-01-21T05:56:56Z\u0009INFO\u0009Got a signal from the OS: "terminated"\u0009{"Process": "storagenode-updater"}\n”,“stream”:“stdout”,“time”:“2024-01-21T05:56:56.6105503Z”}
{“log”:“2024-01-21 05:56:56,611 INFO stopped: storagenode-updater (exit status 0)\n”,“stream”:“stdout”,“time”:“2024-01-21T05:56:56.611165114Z”}
{“log”:“2024-01-21 05:56:59,614 INFO waiting for storagenode, processes-exit-eventlistener to die\n”,“stream”:“stdout”,“time”:“2024-01-21T05:56:59.615096779Z”}
{“log”:“2024-01-21 05:57:02,618 INFO waiting for storagenode, processes-exit-eventlistener to die\n”,“stream”:“stdout”,“time”:“2024-01-21T05:57:02.618812092Z”}
{“log”:“2024-01-21 05:57:05,621 INFO waiting for storagenode, processes-exit-eventlistener to die\n”,“stream”:“stdout”,“time”:“2024-01-21T05:57:05.621594387Z”}
{“log”:“2024-01-21 05:57:06,621 WARN killing ‘storagenode’ (43) with SIGKILL\n”,“stream”:“stdout”,“time”:“2024-01-21T05:57:06.622104497Z”}
{“log”:“2024-01-21 05:57:08,624 INFO waiting for storagenode, processes-exit-eventlistener to die\n”,“stream”:“stdout”,“time”:“2024-01-21T05:57:08.624834811Z”}
{“log”:“2024-01-21 05:57:09,217 INFO stopped: storagenode (terminated by SIGKILL)\n”,“stream”:“stdout”,“time”:“2024-01-21T05:57:09.217841366Z”}
{“log”:“2024-01-21 05:57:09,218 INFO stopped: processes-exit-eventlistener (terminated by SIGTERM)\n”,“stream”:“stdout”,“time”:“2024-01-21T05:57:09.220216107Z”}
{“log”:“2024-01-21 05:57:10,275 INFO Set uid to user 0 succeeded\n”,“stream”:“stdout”,“time”:“2024-01-21T05:57:10.275948847Z”}
{“log”:“2024-01-21 05:57:10,282 INFO RPC interface ‘supervisor’ initialized\n”,“stream”:“stdout”,“time”:“2024-01-21T05:57:10.282338551Z”}
{“log”:“2024-01-21 05:57:10,282 INFO supervisord started with pid 1\n”,“stream”:“stdout”,“time”:“2024-01-21T05:57:10.282576328Z”}
{“log”:“2024-01-21 05:57:11,284 INFO spawned: ‘processes-exit-eventlistener’ with pid 11\n”,“stream”:“stdout”,“time”:“2024-01-21T05:57:11.285794161Z”}
{“log”:“2024-01-21 05:57:11,287 INFO spawned: ‘storagenode’ with pid 12\n”,“stream”:“stdout”,“time”:“2024-01-21T05:57:11.28974055Z”}
{“log”:“2024-01-21 05:57:11,290 INFO spawned: ‘storagenode-updater’ with pid 13\n”,“stream”:“stdout”,“time”:“2024-01-21T05:57:11.291685254Z”}
{“log”:“2024-01-21T05:57:11Z\u0009INFO\u0009Configuration loaded\u0009{"Process": "storagenode-updater", "Location": "/app/config/config.yaml"}\n”,“stream”:“stdout”,“time”:“2024-01-21T05:57:11.301597587Z”}
{“log”:“2024-01-21T05:57:11Z\u0009INFO\u0009Invalid configuration file key\u0009{"Process": "storagenode-updater", "Key": "server.address"}\n”,“stream”:“stdout”,“time”:“2024-01-21T05:57:11.301653328Z”}
{“log”:“2024-01-21T05:57:11Z\u0009INFO\u0009Invalid configuration file key\u0009{"Process": "storagenode-updater", "Key": "contact.external-address"}\n”,“stream”:“stdout”,“time”:“2024-01-21T05:57:11.301663439Z”}
{“log”:“2024-01-21T05:57:11Z\u0009INFO\u0009Invalid configuration file key\u0009{"Process": "storagenode-updater", "Key": "storage.allocated-bandwidth"}\n”,“stream”:“stdout”,“time”:“2024-01-21T05:57:11.301670235Z”}
{“log”:“2024-01-21T05:57:11Z\u0009INFO\u0009Invalid configuration file key\u0009{"Process": "storagenode-updater", "Key": "server.private-address"}\n”,“stream”:“stdout”,“time”:“2024-01-21T05:57:11.301676606Z”}
{“log”:“2024-01-21T05:57:11Z\u0009INFO\u0009Invalid configuration file key\u0009{"Process": "storagenode-updater", "Key": "operator.email"}\n”,“stream”:“stdout”,“time”:“2024-01-21T05:57:11.301682328Z”}
{“log”:“2024-01-21T05:57:11Z\u0009INFO\u0009Invalid configuration file key\u0009{"Process": "storagenode-updater", "Key": "storage2.database-dir"}\n”,“stream”:“stdout”,“time”:“2024-01-21T05:57:11.301687587Z”}
{“log”:“2024-01-21T05:57:11Z\u0009INFO\u0009Invalid configuration file key\u0009{"Process": "storagenode-updater", "Key": "operator.wallet"}\n”,“stream”:“stdout”,“time”:“2024-01-21T05:57:11.301693291Z”}
{“log”:“2024-01-21T05:57:11Z\u0009INFO\u0009Invalid configuration file key\u0009{"Process": "storagenode-updater", "Key": "storage.allocated-disk-space"}\n”,“stream”:“stdout”,“time”:“2024-01-21T05:57:11.301698735Z”}
{“log”:“2024-01-21T05:57:11Z\u0009INFO\u0009Invalid configuration file key\u0009{"Process": "storagenode-updater", "Key": "server.debug-log-traffic"}\n”,“stream”:“stdout”,“time”:“2024-01-21T05:57:11.301704087Z”}
{“log”:“2024-01-21T05:57:11Z\u0009INFO\u0009Invalid configuration file value for key\u0009{"Process": "storagenode-updater", "Key": "log.level"}\n”,“stream”:“stdout”,“time”:“2024-01-21T05:57:11.301709161Z”}
{“log”:“2024-01-21T05:57:11Z\u0009INFO\u0009Invalid configuration file value for key\u0009{"Process": "storagenode-updater", "Key": "log.output"}\n”,“stream”:“stdout”,“time”:“2024-01-21T05:57:11.301722217Z”}
{“log”:“2024-01-21T05:57:11Z\u0009INFO\u0009Invalid configuration file value for key\u0009{"Process": "storagenode-updater", "Key": "log.development"}\n”,“stream”:“stdout”,“time”:“2024-01-21T05:57:11.301728106Z”}

[Aniel]

just sorry because I have no explication about this problem. before december 2023 this node node working well from 2020 (and before) and december2023. no modification before problem.

[Alexey]

Please try:

docker stop -t 300 storagenode
docker rm storagenode
docker rmi storjlabs/storagenode:latest
docker pull storjlabs/storagenode:latest

then execute your full docker run command and check logs

docker logs --tail 20 storagenode

Also please give the result of the command:

docker -it exec storagenode grep log config/config.yaml

[Aniel]

@Alexey

docker stop -t 300 storagenode
docker rm storagenode
docker rmi storjlabs/storagenode:latest
docker pull storjlabs/storagenode:latest

then execute your full docker run command and check logs

docker logs --tail 20 storagenode

Also please give the result of the command:

docker -it exec storagenode grep log config/config.yaml

Hello , sorry for delay:
for docker logs --tail 20 storagenode_13To:

docker logs --tail 20 storagenode_13To
2024-01-27T11:50:34Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "operator.wallet"}
2024-01-27T11:50:34Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "storage.allocated-disk-space"}
2024-01-27T11:50:34Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "server.address"}
2024-01-27T11:50:34Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "storage2.database-dir"}
2024-01-27T11:50:34Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "storage.allocated-bandwidth"}
2024-01-27T11:50:34Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "server.private-address"}
2024-01-27T11:50:34Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "server.debug-log-traffic"}
2024-01-27T11:50:34Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "contact.external-address"}
2024-01-27T11:50:34Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "operator.email"}
2024-01-27T11:50:34Z    INFO    Invalid configuration file value for key        {"Process": "storagenode-updater", "Key": "log.output"}
2024-01-27T11:50:34Z    INFO    Anonymized tracing enabled      {"Process": "storagenode-updater"}
2024-01-27T11:50:34Z    INFO    Running on version      {"Process": "storagenode-updater", "Service": "storagenode-updater", "Version": "v1.95.1"}
2024-01-27T11:50:34Z    INFO    Downloading versions.   {"Process": "storagenode-updater", "Server Address": "https://version.storj.io"}
2024-01-27T11:50:35Z    INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode", "Version": "v1.95.1"}
2024-01-27T11:50:35Z    INFO    Version is up to date   {"Process": "storagenode-updater", "Service": "storagenode"}
2024-01-27T11:50:35Z    INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode-updater", "Version": "v1.95.1"}
2024-01-27T11:50:35Z    INFO    Version is up to date   {"Process": "storagenode-updater", "Service": "storagenode-updater"}
2024-01-27 11:50:36,279 INFO success: processes-exit-eventlistener entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2024-01-27 11:50:36,279 INFO success: storagenode entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2024-01-27 11:50:36,279 INFO success: storagenode-updater entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)

docker exec storagenode_13To grep log config/config.yaml
# if true, log function filename and line number
# log.caller: false
# if true, set logging to development mode
# log.development: false
# configures log encoding. can either be 'console' or 'json'
# log.encoding: console
# the minimum log level to log
log.level: info
log.output: "/app/config/node.log"
# if true, log stack traces
# log.stack: false
# path to log for oom notices
# monkit.hw.oomlog: /var/log/kern.log
# log all GRPC traffic to zap logger
server.debug-log-traffic: false

==> option -it not working

[Alexey]

@Aniel

log.output: "/app/config/node.log"

Aha! So you redirected logs to the file. The file node.log is placed in the data location, so you need to check it instead of docker logs.
Check the path, specified in the mapping --mount type=bind as a source for the destination=/app/config on your host.

[Aniel]

@Alexey

docker stop -t 300 storagenode

for the moment no problem.
But I create another ssd with last raspbian bookworm and install last stable docker + copy all Identity folder and DB folder.
I restart all and for the moment it’s clear.

I deleted node.log and restarted node. I’ll check log if node bug again. but now all install is clear.

(And in order to reboot when I’m away I bought a smart plug. just in case of)

[Alexey]

@Aniel

• copy all Identity folder and DB folder.

It’s better to move the identity to the disk with data, they are useless without each other anyway, and update the source path to the new identity location in your docker run command.

[Aniel]

Data disk is HDD, system with DB and identity is SSD.

Today that crash again but I have log

2024-01-29T10:22:18Z INFO Telemetry enabled {"process": "storagenode", "instance ID": "1CDV4vNy8wzdt5MV1CZRqa9op52AewZ4GCW4z3YXwrGSTNU2S1"}
2024-01-29T10:22:18Z INFO Event collection enabled {"process": "storagenode", "instance ID": "1CDV4vNy8wzdt5MV1CZRqa9op52AewZ4GCW4z3YXwrGSTNU2S1"}
2024-01-29T10:22:18Z INFO db.migration Database Version {"process": "storagenode", "version": 54}
2024-01-29T10:22:18Z INFO preflight:localtime start checking local system clock with trusted satellites' system clock. {"process": "storagenode"}
2024-01-29T10:22:18Z ERROR preflight:localtime system clock is out of sync with satellite {"process": "storagenode", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "error": "system clock is out of sync: clock off by 285.258631 minutes", "errorVerbose": "system clock is out of sync: clock off by 285.258631 minutes\n\tstorj.io/storj/storagenode/preflight.(*LocalTime).checkSatelliteTime:132\n\tstorj.io/storj/storagenode/preflight.(*LocalTime).Check.func1:74\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75"}
2024-01-29T10:22:18Z INFO Got a signal from the OS: "terminated" {"process": "storagenode"}
2024-01-29T10:22:18Z ERROR preflight:localtime unable to get satellite system time {"process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "error": "rpc: tcp connector failed: rpc: context canceled", "errorVerbose": "rpc: tcp connector failed: rpc: context canceled\n\tstorj.io/common/rpc.HybridConnector.DialContext.func1:190"}
2024-01-29T10:22:18Z ERROR preflight:localtime unable to get satellite system time {"process": "storagenode", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "error": "rpc: tcp connector failed: rpc: context canceled", "errorVerbose": "rpc: tcp connector failed: rpc: context canceled\n\tstorj.io/common/rpc.HybridConnector.DialContext.func1:190"}
2024-01-29T10:22:18Z ERROR preflight:localtime unable to get satellite system time {"process": "storagenode", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "error": "context canceled"}
2024-01-29T10:22:18Z ERROR Failed preflight check. {"process": "storagenode", "error": "system clock is out of sync: system clock is out of sync with all trusted satellites", "errorVerbose": "system clock is out of sync: system clock is out of sync with all trusted satellites\n\tstorj.io/storj/storagenode/preflight.(*LocalTime).Check:96\n\tstorj.io/storj/storagenode.(*Peer).Run:890\n\tmain.cmdRun:118\n\tmain.newRunCmd.func1:32\n\tstorj.io/private/process.cleanup.func1.4:393\n\tstorj.io/private/process.cleanup.func1:411\n\tgithub.com/spf13/cobra.(*Command).execute:852\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:960\n\tgithub.com/spf13/cobra.(*Command).Execute:897\n\tstorj.io/private/process.ExecWithCustomOptions:112\n\tmain.main:30\n\truntime.main:267"}
2024-01-29T10:22:18Z FATAL Unrecoverable error {"process": "storagenode", "error": "system clock is out of sync: system clock is out of sync with all trusted satellites", "errorVerbose": "system clock is out of sync: system clock is out of sync with all trusted satellites\n\tstorj.io/storj/storagenode/preflight.(*LocalTime).Check:96\n\tstorj.io/storj/storagenode.(*Peer).Run:890\n\tmain.cmdRun:118\n\tmain.newRunCmd.func1:32\n\tstorj.io/private/process.cleanup.func1.4:393\n\tstorj.io/private/process.cleanup.func1:411\n\tgithub.com/spf13/cobra.(*Command).execute:852\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:960\n\tgithub.com/spf13/cobra.(*Command).Execute:897\n\tstorj.io/private/process.ExecWithCustomOptions:112\n\tmain.main:30\n\truntime.main:267"}
2024-01-29T10:22:20Z INFO Configuration loaded {"process": "storagenode", "Location": "/app/config/config.yaml"}
2024-01-29T10:22:20Z INFO Anonymized tracing enabled {"process": "storagenode"}2024-01-29T10:22:20Z INFO Operator email {"process": "storagenode", "Address": "angenoir0001@free.fr"}
2024-01-29T10:22:20Z INFO Operator wallet {"process": "storagenode", "Address": "0xf522e7ae1ae881937C8e72c641E2d0177FD2c0c5"}
2024-01-29T10:22:22Z INFO server kernel support for server-side tcp fast open remains disabled. {"process": "storagenode"}
2024-01-29T10:22:22Z INFO server enable with: sysctl -w net.ipv4.tcp_fastopen=3 {"process": "storagenode"}
2024-01-29T15:07:38Z INFO Telemetry enabled {"process": "storagenode", "instance ID": "1CDV4vNy8wzdt5MV1CZRqa9op52AewZ4GCW4z3YXwrGSTNU2S1"}
2024-01-29T15:07:38Z INFO Event collection enabled {"process": "storagenode", "instance ID": "1CDV4vNy8wzdt5MV1CZRqa9op52AewZ4GCW4z3YXwrGSTNU2S1"}
2024-01-29T15:07:38Z INFO db.migration Database Version {"process": "storagenode", "version": 54}
2024-01-29T15:07:38Z INFO preflight:localtime start checking local system clock with trusted satellites' system clock. {"process": "storagenode"}
2024-01-29T15:07:39Z INFO preflight:localtime local system clock is in sync with trusted satellites' system clock. {"process": "storagenode"}
2024-01-29T15:07:39Z INFO bandwidth Performing bandwidth usage rollups {"process": "storagenode"}
2024-01-29T15:07:39Z INFO pieces:trash emptying trash started {"process": "storagenode", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs"}
2024-01-29T15:07:39Z INFO trust Scheduling next refresh {"process": "storagenode", "after": "5h56m31.643538121s"}
2024-01-29T15:07:39Z INFO Node 1CDV4vNy8wzdt5MV1CZRqa9op52AewZ4GCW4z3YXwrGSTNU2S1 started {"process": "storagenode"}
2024-01-29T15:07:39Z INFO Public server started on [::]:28967 {"process": "storagenode"}

==> “system clock is out of sync: system clock is out of sync with all trusted satellites” ?!?

[Aniel]

So I search on forum and found your post:

Node stopped responding overnight. Still shows OFFLINE after restart

ok, let check it together. WAN IP on the main page of the router. Please, compare it with IP there: https://www.yougetsignal.com/tools/open-ports/ , they should match Local IP in the port forwarding rule, it should be the same as IP of your PC with docker nslookup your.ddns.address 8.8.8.8 should return the same IP as p.1 Firewall. It should have an incoming rule to allow any traffic from the any source to the TCP 28967 port and your PC as a destination. If you have any outgoing rule, then y…

1. I check port , it’s OK and open
2. Ok too
3. Yes I have same IP from 1 to Nslookup
4. ok for port forwarding
5. identity , I have 2 and 3 respectively.

[Alexey]

@Aniel

2024-01-29T10:22:18Z FATAL Unrecoverable error {"process": "storagenode", "error": "system clock is out of sync: system clock is out of sync with all trusted satellites", "errorVerbose": "system clock is out of sync: system clock is out of sync with all trusted satellites\n\tstorj.io/storj/storagenode/preflight.(*LocalTime).Check:96\n\tstorj.io/storj/storagenode.(*Peer).Run:890\n\tmain.cmdRun:118\n\tmain.newRunCmd.func1:32\n\tstorj.io/private/process.cleanup.func1.4:393\n\tstorj.io/private/process.cleanup.func1:411\n\tgithub.com/spf13/cobra.(*Command).execute:852\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:960\n\tgithub.com/spf13/cobra.(*Command).Execute:897\n\tstorj.io/private/process.ExecWithCustomOptions:112\n\tmain.main:30\n\truntime.main:267"}

You need to sync clock on your device: Sync Time from Internet - Raspberry Pi Forums

[Aniel]

Exact 33h without problem for the moment