I just realized that one of my nodes was down for almost 3 days (seems uptimerobot didn’t detect it) and is restarting all time.
I’ve checked the logs (docker logs --tail 20) and seem that can’t get the system time of any satellite
I’ve tried to edit the system clock updater with this settings
[Time]
NTP=ur.ntp.srv
FallbackNTP=ur.fallbackntp.srv
and restarted the service but seems not working
any help?
Hi @camulodunum,
Please let us know more about your setup. It sounds like you need to run a time sync if the system time is incorrect.
a raspberry pi 4 with 2gb of ram running ubuntu 20.04 lts 64 bits,a wireguard VPN in order to do remote support.
the storj is in an external 6tb hdd from WD connected by usb. the conection is 1gbps simmetrical fiber optic
and don’t know what more can I say about my setup that could help to solve this
So a ubuntu install… run “ntpq -p” or “timedatectl”
Time looks to be correct, even though it says “synchronized: no”.
Can you show last 20 lines of the storagenode log ?
Unable to get satellite system time means you can’t reach them. If you would be able to reach the satellite but your clock would be off you would see a different message including the offset.
any tough why could this be?
I have the tcp 28967 port open both in the ubuntu firewall and in the router
Missing port forwarding should trigger a different error message as well.
I bet on identity not signed. The satellite will terminate the connection.
¿how can I check that?
Identity files look fine and it’s unusual for them to stop working or disappear if the node was working fine.
It could be a DNS issue, please can you try to ping us2.storj.io
edit - It’s not a DNS issue as the satellite IP details are in the log entries
edit2 - next idea - please can you show more log lines, there may be errors before the “unable to get satellite system time” entries.
Seems to b just doing this in loop
2021-10-19T09:08:57.327Z INFO Configuration loaded {"Location": "-----------------"}
2021-10-19T09:08:57.329Z INFO Operator email {"Address": "---------@-------.-----"}
2021-10-19T09:08:57.329Z INFO Operator wallet {"Address": "0x------------------------------"}
2021-10-19T09:08:58.094Z INFO Telemetry enabled {"instance ID": "------------------------------------------"}
2021-10-19T09:08:58.194Z INFO db.migration Database Version {"version": 53}
2021-10-19T09:08:58.516Z INFO preflight:localtime start checking local system clock with trusted satellites' system clock.
2021-10-19T09:09:18.520Z ERROR preflight:localtime unable to get satellite system time {"Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "error": "rpc: dial tcp 34.94.230.169:7777: i/o timeout", "errorVerbose": "rpc: dial tcp 34.94.230.169:7777: i/o timeout\n\tstorj.io/common/rpc.TCPConnector.DialContextUnencrypted:107\n\tstorj.io/common/rpc.TCPConnector.DialContext:71\n\tstorj.io/common/rpc.Dialer.dialEncryptedConn:220\n\tstorj.io/common/rpc.Dialer.DialNodeURL.func1:110\n\tstorj.io/common/rpc/rpcpool.(*Pool).get:105\n\tstorj.io/common/rpc/rpcpool.(*Pool).Get:128\n\tstorj.io/common/rpc.Dialer.dialPool:186\n\tstorj.io/common/rpc.Dialer.DialNodeURL:109\n\tstorj.io/storj/storagenode/preflight.(*LocalTime).getSatelliteTime:110\n\tstorj.io/storj/storagenode/preflight.(*LocalTime).Check.func1:67\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
2021-10-19T09:09:18.520Z ERROR preflight:localtime unable to get satellite system time {"Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "error": "rpc: dial tcp 35.241.93.116:7777: i/o timeout", "errorVerbose": "rpc: dial tcp 35.241.93.116:7777: i/o timeout\n\tstorj.io/common/rpc.TCPConnector.DialContextUnencrypted:107\n\tstorj.io/common/rpc.TCPConnector.DialContext:71\n\tstorj.io/common/rpc.Dialer.dialEncryptedConn:220\n\tstorj.io/common/rpc.Dialer.DialNodeURL.func1:110\n\tstorj.io/common/rpc/rpcpool.(*Pool).get:105\n\tstorj.io/common/rpc/rpcpool.(*Pool).Get:128\n\tstorj.io/common/rpc.Dialer.dialPool:186\n\tstorj.io/common/rpc.Dialer.DialNodeURL:109\n\tstorj.io/storj/storagenode/preflight.(*LocalTime).getSatelliteTime:110\n\tstorj.io/storj/storagenode/preflight.(*LocalTime).Check.func1:67\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
2021-10-19T09:09:18.520Z ERROR preflight:localtime unable to get satellite system time {"Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "error": "rpc: dial tcp 34.86.126.23:7777: i/o timeout", "errorVerbose": "rpc: dial tcp 34.86.126.23:7777: i/o timeout\n\tstorj.io/common/rpc.TCPConnector.DialContextUnencrypted:107\n\tstorj.io/common/rpc.TCPConnector.DialContext:71\n\tstorj.io/common/rpc.Dialer.dialEncryptedConn:220\n\tstorj.io/common/rpc.Dialer.DialNodeURL.func1:110\n\tstorj.io/common/rpc/rpcpool.(*Pool).get:105\n\tstorj.io/common/rpc/rpcpool.(*Pool).Get:128\n\tstorj.io/common/rpc.Dialer.dialPool:186\n\tstorj.io/common/rpc.Dialer.DialNodeURL:109\n\tstorj.io/storj/storagenode/preflight.(*LocalTime).getSatelliteTime:110\n\tstorj.io/storj/storagenode/preflight.(*LocalTime).Check.func1:67\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
2021-10-19T09:09:18.520Z ERROR preflight:localtime unable to get satellite system time {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "error": "rpc: dial tcp 34.107.91.2:7777: i/o timeout", "errorVerbose": "rpc: dial tcp 34.107.91.2:7777: i/o timeout\n\tstorj.io/common/rpc.TCPConnector.DialContextUnencrypted:107\n\tstorj.io/common/rpc.TCPConnector.DialContext:71\n\tstorj.io/common/rpc.Dialer.dialEncryptedConn:220\n\tstorj.io/common/rpc.Dialer.DialNodeURL.func1:110\n\tstorj.io/common/rpc/rpcpool.(*Pool).get:105\n\tstorj.io/common/rpc/rpcpool.(*Pool).Get:128\n\tstorj.io/common/rpc.Dialer.dialPool:186\n\tstorj.io/common/rpc.Dialer.DialNodeURL:109\n\tstorj.io/storj/storagenode/preflight.(*LocalTime).getSatelliteTime:110\n\tstorj.io/storj/storagenode/preflight.(*LocalTime).Check.func1:67\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
2021-10-19T09:09:18.521Z ERROR preflight:localtime unable to get satellite system time {"Satellite ID": "12tRQrMTWUWwzwGh18i7Fqs67kmdhH9t6aToeiwbo5mfS2rUmo", "error": "rpc: dial tcp 34.86.70.247:7777: i/o timeout", "errorVerbose": "rpc: dial tcp 34.86.70.247:7777: i/o timeout\n\tstorj.io/common/rpc.TCPConnector.DialContextUnencrypted:107\n\tstorj.io/common/rpc.TCPConnector.DialContext:71\n\tstorj.io/common/rpc.Dialer.dialEncryptedConn:220\n\tstorj.io/common/rpc.Dialer.DialNodeURL.func1:110\n\tstorj.io/common/rpc/rpcpool.(*Pool).get:105\n\tstorj.io/common/rpc/rpcpool.(*Pool).Get:128\n\tstorj.io/common/rpc.Dialer.dialPool:186\n\tstorj.io/common/rpc.Dialer.DialNodeURL:109\n\tstorj.io/storj/storagenode/preflight.(*LocalTime).getSatelliteTime:110\n\tstorj.io/storj/storagenode/preflight.(*LocalTime).Check.func1:67\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
2021-10-19T09:09:18.521Z ERROR preflight:localtime unable to get satellite system time {"Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "error": "rpc: dial tcp 35.228.118.14:7777: i/o timeout", "errorVerbose": "rpc: dial tcp 35.228.118.14:7777: i/o timeout\n\tstorj.io/common/rpc.TCPConnector.DialContextUnencrypted:107\n\tstorj.io/common/rpc.TCPConnector.DialContext:71\n\tstorj.io/common/rpc.Dialer.dialEncryptedConn:220\n\tstorj.io/common/rpc.Dialer.DialNodeURL.func1:110\n\tstorj.io/common/rpc/rpcpool.(*Pool).get:105\n\tstorj.io/common/rpc/rpcpool.(*Pool).Get:128\n\tstorj.io/common/rpc.Dialer.dialPool:186\n\tstorj.io/common/rpc.Dialer.DialNodeURL:109\n\tstorj.io/storj/storagenode/preflight.(*LocalTime).getSatelliteTime:110\n\tstorj.io/storj/storagenode/preflight.(*LocalTime).Check.func1:67\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
2021-10-19T09:09:18.521Z ERROR Failed preflight check. {"error": "system clock is out of sync: system clock is out of sync with all trusted satellites", "errorVerbose": "system clock is out of sync: system clock is out of sync with all trusted satellites\n\tstorj.io/storj/storagenode/preflight.(*LocalTime).Check:96\n\tstorj.io/storj/storagenode.(*Peer).Run:846\n\tmain.cmdRun:218\n\tstorj.io/private/process.cleanup.func1.4:363\n\tstorj.io/private/process.cleanup.func1:381\n\tgithub.com/spf13/cobra.(*Command).execute:852\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:960\n\tgithub.com/spf13/cobra.(*Command).Execute:897\n\tstorj.io/private/process.ExecWithCustomConfig:88\n\tstorj.io/private/process.ExecCustomDebug:70\n\tmain.main:385\n\truntime.main:255"}
Error: system clock is out of sync: system clock is out of sync with all trusted satellites
2021-10-19T09:09:21.712Z INFO Configuration loaded {"-----------"}
2021-10-19T09:09:21.714Z INFO Operator email {"Address": "-----------------"}
2021-10-19T09:09:21.714Z INFO Operator wallet {"Address": "-------------------------------"}
2021-10-19T09:09:22.501Z INFO Telemetry enabled {"instance ID": "---------------------------------------"}
2021-10-19T09:09:22.717Z INFO db.migration Database Version {"version": 53}
2021-10-19T09:09:22.967Z INFO preflight:localtime start checking local system clock with trusted satellites' system clock.
tooks it like 20 sconds for each reboot
This is not a common error, but the other times this has occurred SNO’s have found a local network issue with port 7777 traffic (the one used for satellite communications) or ended up reinstalling the node (with the same identity and data).
just reinstalled this node like a month ago with a fresh new ubuntu because the last install was just freezing itself and the whole machine for no reason.
¿any way I can test the 7777 port as client? ¿anyplace where I can manually send a ping or something like that trought an specific port?
thanks for the help,I’m triying to avoid another reinstall as posible.
you could try telnet to us2.storj.io:7777 see if it connects - FYI there’s no response it should sit waiting
Ok seems that I’m not able to get a response,so seems my node can’t connect to us2.storj.io neither port 7777 or default ping port
triyed ping from my laptop in a different network location and seems to be working,with the same dns
I’m using google dns 8.8.8.8 as primary dns and cloudflare 1.1.1.1 as the secondary one, is strange to have some dns issues with this two
¿any way I can enforce the dns to the raspberry itself? is supposed to be taken by the DHCP config frm the router but maybe the ubuntu machine has misplaced it or is using another dns
if you ping us2 from the pi, do you get an IP resolve? I think you should, which shows dns is working. It is something blocking port 7777.