Node doesn't work anymore

Node Operators troubleshooting
1

Hi all,

I am running a node on a raspberry pi 3 (latest official OS, 64-bit) and since a couple of days ago everything ran smoothly, then it suddenly stopped working and these are my logs:

2025-06-13T07:03:27Z ERROR Error retrieving version info. {“Process”: “storagenode-updater”, “error”: “version checker client: Get "https://version.storj.io": dial tcp: lookup version.storj.io on 149.112.112.112:53: read udp 172.17.0.2:58620->149.112.112.112:53: i/o timeout”, “errorVerbose”: “version checker client: Get "https://version.storj.io": dial tcp: lookup version.storj.io on 149.112.112.112:53: read udp 172.17.0.2:58620->149.112.112.112:53: i/o timeout\n\tstorj.io/storj/private/version/checker.(*Client).All:68\n\tmain.loopFunc:20\n\tstorj.io/common/sync2.(*Cycle).Run:102\n\tmain.cmdRun:139\n\tstorj.io/common/process.cleanup.func1.4:402\n\tstorj.io/common/process.cleanup.func1:420\n\tgithub.com/spf13/cobra.(*Command).execute:985\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1117\n\tgithub.com/spf13/cobra.(*Command).Execute:1041\n\tstorj.io/common/process.ExecWithCustomOptions:116\n\tstorj.io/common/process.ExecWithCustomConfigAndLogger:81\n\tmain.main:22\n\truntime.main:283”}
2025-06-13T07:03:27Z ERROR version failed to get process version info {“Process”: “storagenode”, “error”: “version checker client: Get "https://version.storj.io": dial tcp: lookup version.storj.io on 149.112.112.112:53: read udp 172.17.0.2:41652->149.112.112.112:53: i/o timeout”, “errorVerbose”: “version checker client: Get "https://version.storj.io": dial tcp: lookup version.storj.io on 149.112.112.112:53: read udp 172.17.0.2:41652->149.112.112.112:53: i/o timeout\n\tstorj.io/storj/private/version/checker.(*Client).All:68\n\tstorj.io/storj/private/version/checker.(*Client).Process:89\n\tstorj.io/storj/private/version/checker.(*Service).checkVersion:104\n\tstorj.io/storj/private/version/checker.(*Service).CheckVersion:78\n\tmain.cmdRun:91\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.cleanup.func1.4:402\n\tstorj.io/common/process.cleanup.func1:420\n\tgithub.com/spf13/cobra.(*Command).execute:985\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1117\n\tgithub.com/spf13/cobra.(*Command).Execute:1041\n\tstorj.io/common/process.ExecWithCustomOptions:116\n\tmain.main:34\n\truntime.main:283”}
2025-06-13T07:03:27Z INFO Telemetry enabled {“Process”: “storagenode”, “instance ID”: “1tTK8aJUueVbxetvpmHwuRAgGgSgfUJ8R3YQM3L8qSzHFLUvhe”}
2025-06-13T07:03:27Z INFO Event collection enabled {“Process”: “storagenode”, “instance ID”: “1tTK8aJUueVbxetvpmHwuRAgGgSgfUJ8R3YQM3L8qSzHFLUvhe”}
2025-06-13T07:03:27Z INFO db.migration Database Version {“Process”: “storagenode”, “version”: 62}
2025-06-13T07:03:48Z WARN trust Failed to fetch URLs from source; used cache {“Process”: “storagenode”, “source”: “https://static.storj.io/dcs-satellites”, “error”: “HTTP source: Get "https://static.storj.io/dcs-satellites\”: dial tcp: lookup static.storj.io on 149.112.112.112:53: read udp 172.17.0.2:33128->149.112.112.112:53: i/o timeout", “errorVerbose”: “HTTP source: Get "https://static.storj.io/dcs-satellites\”: dial tcp: lookup static.storj.io on 149.112.112.112:53: read udp 172.17.0.2:33128->149.112.112.112:53: i/o timeout\n\tstorj.io/storj/storagenode/trust.(*HTTPSource).FetchEntries:68\n\tstorj.io/storj/storagenode/trust.(*List).fetchEntries:90\n\tstorj.io/storj/storagenode/trust.(*List).FetchURLs:49\n\tstorj.io/storj/storagenode/trust.(*Pool).fetchURLs:326\n\tstorj.io/storj/storagenode/trust.(*Pool).Refresh:209\n\tstorj.io/storj/storagenode.(*Peer).Run:1085\n\tmain.cmdRun:127\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.cleanup.func1.4:402\n\tstorj.io/common/process.cleanup.func1:420\n\tgithub.com/spf13/cobra.(*Command).execute:985\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1117\n\tgithub.com/spf13/cobra.(*Command).Execute:1041\n\tstorj.io/common/process.ExecWithCustomOptions:116\n\tmain.main:34\n\truntime.main:283"}
2025-06-13T07:03:48Z INFO preflight:localtime start checking local system clock with trusted satellites’ system clock. {“Process”: “storagenode”}
2025-06-13T07:04:08Z ERROR preflight:localtime unable to get satellite system time {“Process”: “storagenode”, “Satellite ID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”, “error”: “rpc: tcp connector failed: rpc: dial tcp: lookup saltlake.tardigrade.io on 149.112.112.112:53: read udp 172.17.0.2:42835->149.112.112.112:53: i/o timeout”, “errorVerbose”: “rpc: tcp connector failed: rpc: dial tcp: lookup saltlake.tardigrade.io on 149.112.112.112:53: read udp 172.17.0.2:42835->149.112.112.112:53: i/o timeout\n\tstorj.io/common/rpc.HybridConnector.DialContext.func1:190”}
2025-06-13T07:04:08Z ERROR preflight:localtime unable to get satellite system time {“Process”: “storagenode”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “error”: “rpc: tcp connector failed: rpc: dial tcp: lookup us1.storj.io on 149.112.112.112:53: read udp 172.17.0.2:50825->149.112.112.112:53: i/o timeout”, “errorVerbose”: “rpc: tcp connector failed: rpc: dial tcp: lookup us1.storj.io on 149.112.112.112:53: read udp 172.17.0.2:50825->149.112.112.112:53: i/o timeout\n\tstorj.io/common/rpc.HybridConnector.DialContext.func1:190”}
2025-06-13T07:04:08Z ERROR preflight:localtime unable to get satellite system time {“Process”: “storagenode”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “error”: “rpc: tcp connector failed: rpc: dial tcp: lookup eu1.storj.io on 149.112.112.112:53: read udp 172.17.0.2:36875->149.112.112.112:53: i/o timeout”, “errorVerbose”: “rpc: tcp connector failed: rpc: dial tcp: lookup eu1.storj.io on 149.112.112.112:53: read udp 172.17.0.2:36875->149.112.112.112:53: i/o timeout\n\tstorj.io/common/rpc.HybridConnector.DialContext.func1:190”}
2025-06-13T07:04:08Z ERROR preflight:localtime unable to get satellite system time {“Process”: “storagenode”, “Satellite ID”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”, “error”: “rpc: tcp connector failed: rpc: dial tcp: lookup ap1.storj.io on 149.112.112.112:53: read udp 172.17.0.2:49670->149.112.112.112:53: i/o timeout”, “errorVerbose”: “rpc: tcp connector failed: rpc: dial tcp: lookup ap1.storj.io on 149.112.112.112:53: read udp 172.17.0.2:49670->149.112.112.112:53: i/o timeout\n\tstorj.io/common/rpc.HybridConnector.DialContext.func1:190”}
2025-06-13T07:04:08Z ERROR Failed preflight check. {“Process”: “storagenode”, “error”: “system clock is out of sync: system clock is out of sync with all trusted satellites”, “errorVerbose”: “system clock is out of sync: system clock is out of sync with all trusted satellites\n\tstorj.io/storj/storagenode/preflight.(*LocalTime).Check:96\n\tstorj.io/storj/storagenode.(*Peer).Run:1089\n\tmain.cmdRun:127\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.cleanup.func1.4:402\n\tstorj.io/common/process.cleanup.func1:420\n\tgithub.com/spf13/cobra.(*Command).execute:985\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1117\n\tgithub.com/spf13/cobra.(*Command).Execute:1041\n\tstorj.io/common/process.ExecWithCustomOptions:116\n\tmain.main:34\n\truntime.main:283”}
2025-06-13T07:04:09Z ERROR failure during run {“Process”: “storagenode”, “error”: “system clock is out of sync: system clock is out of sync with all trusted satellites”, “errorVerbose”: “system clock is out of sync: system clock is out of sync with all trusted satellites\n\tstorj.io/storj/storagenode/preflight.(*LocalTime).Check:96\n\tstorj.io/storj/storagenode.(*Peer).Run:1089\n\tmain.cmdRun:127\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.cleanup.func1.4:402\n\tstorj.io/common/process.cleanup.func1:420\n\tgithub.com/spf13/cobra.(*Command).execute:985\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1117\n\tgithub.com/spf13/cobra.(*Command).Execute:1041\n\tstorj.io/common/process.ExecWithCustomOptions:116\n\tmain.main:34\n\truntime.main:283”}
Error: system clock is out of sync: system clock is out of sync with all trusted satellites
2025-06-13 07:04:09,560 WARN exited: storagenode (exit status 1; not expected)
2025-06-13 07:04:10,570 INFO spawned: ‘storagenode’ with pid 54
2025-06-13 07:04:10,572 WARN received SIGQUIT indicating exit request
2025-06-13 07:04:10,574 INFO waiting for processes-exit-eventlistener, storagenode, storagenode-updater to die
2025-06-13T07:04:10Z INFO Got a signal from the OS: “terminated” {“Process”: “storagenode-updater”}
2025-06-13 07:04:10,584 INFO stopped: storagenode-updater (exit status 0)
2025-06-13 07:04:10,589 WARN stopped: storagenode (terminated by SIGTERM)
2025-06-13 07:04:10,591 WARN stopped: processes-exit-eventlistener (terminated by SIGTERM)

On the dashboard it says QUICK “misconfigured” even though I did not change anything and it always used to be “OK”

2

It looks like trouble connecting to internet, but it’s doing Dns lookups ok.

Maybe this?

1 Like
3

Hello @cobra,
Welcome back!

Usually means that the DNS is not working, you may try to use 8.8.8.8 on your device or on the router as a DNS server.

4

Ok that’s interesting. I didn’t restart the pi in a very long while and after restarting everything is working well.

5

When I was using RasPis, I almost always had them on a crontab job that rebooted them once a month. Cleared out sooooo many issues, I did not want to debug.