System clock is out of sync after reboot

nerdatwork · October 2, 2024, 6:33am

There is an upcoming change in the URL.

storagenode/trust: Update trusted satellites URL

committed 10:33PM - 19 Sep 24 UTC

Update storagenode config to use static.storj.io/dcs-satellites instead of www.s…torj.io/dcs-satellites The www subdomain will continue to serve dcs-satellites for now, but eventually it will redirect to the static.storj.io subdomain. A storagenode is still able to fetch the file with the redirect in place, but it's good to change the config now so that a redirect is not needed in the future. Change-Id: I7b59d7383c3def1dc26d2d5761780edef7b38210

Alexey · October 2, 2024, 6:37am

Yes, but both should work, until all nodes would migrate. There could be a bug in the not yet released 1.114.x

Vasabi · October 2, 2024, 7:01am

Да это работает, интересно можно все узлы перевести на 1,114,3?

MilinDV · October 2, 2024, 6:36pm

Hi.
After updating three of nodes manually (I do this to control the start of the service, automatic update failed several times in the past) from version v1.113.x to versions v1.114.3, the Windows service does not start, the logs show problem with NTP.

NTP Windows is available, synchronized both from the OS and using utilities.
Internet channels are available, different from different providers and in different countries, constant remote access to the nodes, replacing the Internet channel with a VPN does not change anything.
There is a fourth node on version v1.113.4, it is scary to update.

The problem is on all three, at the same time, immediately after the update, the nodes are down, what to do?

/go/pkg/mod/storj.io/common@v0.0.0-20240913104706-8fd6fa0286a3/process/exec_conf.go:62

main.(*service).Execute.func1
/go/src/storj.io/storj/cmd/storagenode/service_windows.go:107
golang.org/x/sync/errgroup.(*Group).Go.func1
/go/pkg/mod/golang.org/x/sync@v0.8.0/errgroup/errgroup.go:78

*2024-10-02T21:30:58+03:00 ERROR storagenode/cmd_run.go:50 failure during run {“error”: “system clock is out of sync: system clock is out of sync with all trusted satellites”, “errorVerbose”: "system clock is out of sync: system clock is out of sync with all trusted satellites\n\tstorj.io/storj/storagenode/preflight.(LocalTime).Check:96\n\tstorj.io/storj/storagenode.

(*Peer).Run:1009\n\tmain.cmdRun:125\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.cleanup.func1.4:392\n\tstorj.io/common/process.cleanup.func1:410\n\tgithub.com/spf13/cobra.(*Command).execute:983\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1115\n\tgithub.com/spf13/cobra.(*Command).Execute:1039\n\tstorj.io/common/process.ExecWithCustomOptions:112\n\tstorj.io/common/process.ExecWithCustomConfigAndLogger:77\n\tstorj.io/common/process.ExecWithCustomConfig:72\n\tstorj.io/common/process.Exec:62\n\tmain.(*service).Execute.func1:107\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78"}
main.cmdRun.func1
/go/src/storj.io/storj/cmd/storagenode/cmd_run.go:50
main.cmdRun
/go/src/storj.io/storj/cmd/storagenode/cmd_run.go:128
main.newRunCmd.func1

MilinDV · October 2, 2024, 6:49pm

After reading the forum I realized that there is an error in all 114 releases, tell me how to roll back to 113 correctly?

Alexey · October 3, 2024, 7:28am

Hello @MilinDV,
Welcome to the forum!

This is a mistake. You must NOT update to the dev versions on a production node.

Alexey · October 3, 2024, 7:29am

No, it’s better to follow https://version.storj.io

на русском

Нет. Рекомендуется использовать версии только из https://version.storj.io

VidocQ · October 8, 2024, 10:39pm

I have a problem, apparently the node has automatically updated to 1.114.50:

time to error!?wtf))

2024-10-09T01:37:04+03:00 ERROR storagenode/peer.go:1010 Failed preflight check. {“error”: “system clock is out of sync: system clock is out of sync with all trusted satellites”, “errorVerbose”: “system clock is out of sync: system clock is out of sync with all trusted satellites\n\tstorj.io/storj/storagenode/preflight.(*LocalTime).Check:96\n\tstorj.io/storj/storagenode.(*Peer).Run:1009\n\tmain.cmdRun:125\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.cleanup.func1.4:392\n\tstorj.io/common/process.cleanup.func1:410\n\tgithub.com/spf13/cobra.(*Command).execute:983\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1115\n\tgithub.com/spf13/cobra.(*Command).Execute:1039\n\tstorj.io/common/process.ExecWithCustomOptions:112\n\tstorj.io/common/process.ExecWithCustomConfigAndLogger:77\n\tstorj.io/common/process.ExecWithCustomConfig:72\n\tstorj.io/common/process.Exec:62\n\tmain.(*service).Execute.func1:107\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78”}
storj.io/storj/storagenode.(*Peer).Run
/go/src/storj.io/storj/storagenode/peer.go:1010
main.cmdRun
/go/src/storj.io/storj/cmd/storagenode/cmd_run.go:125
main.newRunCmd.func1
/go/src/storj.io/storj/cmd/storagenode/cmd_run.go:33
storj.io/common/process.cleanup.func1.4
/go/pkg/mod/storj.io/common@v0.0.0-20240913104706-8fd6fa0286a3/process/exec_conf.go:392
storj.io/common/process.cleanup.func1
/go/pkg/mod/storj.io/common@v0.0.0-20240913104706-8fd6fa0286a3/process/exec_conf.go:410
github.com/spf13/cobra.(*Command).execute
/go/pkg/mod/github.com/spf13/cobra@v1.8.0/command.go:983
github.com/spf13/cobra.(*Command).ExecuteC
/go/pkg/mod/github.com/spf13/cobra@v1.8.0/command.go:1115
github.com/spf13/cobra.(*Command).Execute
/go/pkg/mod/github.com/spf13/cobra@v1.8.0/command.go:1039
storj.io/common/process.ExecWithCustomOptions
/go/pkg/mod/storj.io/common@v0.0.0-20240913104706-8fd6fa0286a3/process/exec_conf.go:112
storj.io/common/process.ExecWithCustomConfigAndLogger
/go/pkg/mod/storj.io/common@v0.0.0-20240913104706-8fd6fa0286a3/process/exec_conf.go:77
storj.io/common/process.ExecWithCustomConfig
/go/pkg/mod/storj.io/common@v0.0.0-20240913104706-8fd6fa0286a3/process/exec_conf.go:72
storj.io/common/process.Exec
/go/pkg/mod/storj.io/common@v0.0.0-20240913104706-8fd6fa0286a3/process/exec_conf.go:62
main.(*service).Execute.func1
/go/src/storj.io/storj/cmd/storagenode/service_windows.go:107
The Go Programming Language
/go/pkg/mod/golang.org/x/sync@v0.8.0/errgroup/errgroup.go:78
2024-10-09T01:37:04+03:00 ERROR storagenode/cmd_run.go:50 failure during run {“error”: “system clock is out of sync: system clock is out of sync with all trusted satellites”, “errorVerbose”: “system clock is out of sync: system clock is out of sync with all trusted satellites\n\tstorj.io/storj/storagenode/preflight.(*LocalTime).Check:96\n\tstorj.io/storj/storagenode.(*Peer).Run:1009\n\tmain.cmdRun:125\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.cleanup.func1.4:392\n\tstorj.io/common/process.cleanup.func1:410\n\tgithub.com/spf13/cobra.(*Command).execute:983\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1115\n\tgithub.com/spf13/cobra.(*Command).Execute:1039\n\tstorj.io/common/process.ExecWithCustomOptions:112\n\tstorj.io/common/process.ExecWithCustomConfigAndLogger:77\n\tstorj.io/common/process.ExecWithCustomConfig:72\n\tstorj.io/common/process.Exec:62\n\tmain.(*service).Execute.func1:107\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78”}
main.cmdRun.func1
/go/src/storj.io/storj/cmd/storagenode/cmd_run.go:50
main.cmdRun
/go/src/storj.io/storj/cmd/storagenode/cmd_run.go:128
main.newRunCmd.func1
/go/src/storj.io/storj/cmd/storagenode/cmd_run.go:33
storj.io/common/process.cleanup.func1.4
/go/pkg/mod/storj.io/common@v0.0.0-20240913104706-8fd6fa0286a3/process/exec_conf.go:392
storj.io/common/process.cleanup.func1
/go/pkg/mod/storj.io/common@v0.0.0-20240913104706-8fd6fa0286a3/process/exec_conf.go:410
github.com/spf13/cobra.(*Command).execute
/go/pkg/mod/github.com/spf13/cobra@v1.8.0/command.go:983
github.com/spf13/cobra.(*Command).ExecuteC
/go/pkg/mod/github.com/spf13/cobra@v1.8.0/command.go:1115
github.com/spf13/cobra.(*Command).Execute
/go/pkg/mod/github.com/spf13/cobra@v1.8.0/command.go:1039
storj.io/common/process.ExecWithCustomOptions
/go/pkg/mod/storj.io/common@v0.0.0-20240913104706-8fd6fa0286a3/process/exec_conf.go:112
storj.io/common/process.ExecWithCustomConfigAndLogger
/go/pkg/mod/storj.io/common@v0.0.0-20240913104706-8fd6fa0286a3/process/exec_conf.go:77
storj.io/common/process.ExecWithCustomConfig
/go/pkg/mod/storj.io/common@v0.0.0-20240913104706-8fd6fa0286a3/process/exec_conf.go:72
storj.io/common/process.Exec
/go/pkg/mod/storj.io/common@v0.0.0-20240913104706-8fd6fa0286a3/process/exec_conf.go:62
main.(*service).Execute.func1
/go/src/storj.io/storj/cmd/storagenode/service_windows.go:107
The Go Programming Language
/go/pkg/mod/golang.org/x/sync@v0.8.0/errgroup/errgroup.go:78
2024-10-09T01:37:04+03:00 FATAL process/exec_conf.go:428 Unrecoverable error {“error”: “system clock is out of sync: system clock is out of sync with all trusted satellites”, “errorVerbose”: “system clock is out of sync: system clock is out of sync with all trusted satellites\n\tstorj.io/storj/storagenode/preflight.(*LocalTime).Check:96\n\tstorj.io/storj/storagenode.(*Peer).Run:1009\n\tmain.cmdRun:125\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.cleanup.func1.4:392\n\tstorj.io/common/process.cleanup.func1:410\n\tgithub.com/spf13/cobra.(*Command).execute:983\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1115\n\tgithub.com/spf13/cobra.(*Command).Execute:1039\n\tstorj.io/common/process.ExecWithCustomOptions:112\n\tstorj.io/common/process.ExecWithCustomConfigAndLogger:77\n\tstorj.io/common/process.ExecWithCustomConfig:72\n\tstorj.io/common/process.Exec:62\n\tmain.(*service).Execute.func1:107\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78”}
storj.io/common/process.cleanup.func1
/go/pkg/mod/storj.io/common@v0.0.0-20240913104706-8fd6fa0286a3/process/exec_conf.go:428
github.com/spf13/cobra.(*Command).execute
/go/pkg/mod/github.com/spf13/cobra@v1.8.0/command.go:983
github.com/spf13/cobra.(*Command).ExecuteC
/go/pkg/mod/github.com/spf13/cobra@v1.8.0/command.go:1115
github.com/spf13/cobra.(*Command).Execute
/go/pkg/mod/github.com/spf13/cobra@v1.8.0/command.go:1039
storj.io/common/process.ExecWithCustomOptions
/go/pkg/mod/storj.io/common@v0.0.0-20240913104706-8fd6fa0286a3/process/exec_conf.go:112
storj.io/common/process.ExecWithCustomConfigAndLogger
/go/pkg/mod/storj.io/common@v0.0.0-20240913104706-8fd6fa0286a3/process/exec_conf.go:77
storj.io/common/process.ExecWithCustomConfig
/go/pkg/mod/storj.io/common@v0.0.0-20240913104706-8fd6fa0286a3/process/exec_conf.go:72
storj.io/common/process.Exec
/go/pkg/mod/storj.io/common@v0.0.0-20240913104706-8fd6fa0286a3/process/exec_conf.go:62
main.(*service).Execute.func1
/go/src/storj.io/storj/cmd/storagenode/service_windows.go:107
The Go Programming Language
/go/pkg/mod/golang.org/x/sync@v0.8.0/errgroup/errgroup.go:78

VidocQ · October 8, 2024, 11:17pm

I had to switch to version 1.113.4, deleted the database and everything worked and I went to sleep.

jtolio · October 9, 2024, 12:24am

We should maybe merge this thread with 🔥 V1.114.5 broken on FreeBSD 13.3-RELEASE - #42 by thepaul

Summary: there are many broken things, not just the trust.sources configuration option. More detail in the above link about why and what, but the solution is to get v1.114.6 which is rolling out now.

VidocQ · October 9, 2024, 12:51am

I use Windows 2012 r2, version 1.114.5 broke on it too, so it will no longer be supported? I also had the client automatically update from 1.113.2 to 1.114.5 and got a time sync error, although over time everything is fine. I had to return the version and delete the databases ((((

jtolio · October 9, 2024, 1:04am

We are continuing to support the same versions of Windows 1.113 did. 1.114.6 should have everything fixed.

Please don’t delete your databases and downgrade, you will likely get disqualified if you do that to the wrong databases.

VidocQ · October 9, 2024, 9:22am

I’m using win 2012 r2 and now it works on 1.114.6. Question about the minimum OS requirements, how long ago the requirements in the section were updated. I must have missed this news.
Thank for answer

Alexey · October 10, 2024, 3:48am

here:

because of

However, later our developers implemented a hack described above, so the binaries are still can work on Windows Server 2012, so no announcements yet. But I would highly recommend to upgrade the OS or migrate to the other OS.