🔥 V1.114.5 broken on FreeBSD 13.3-RELEASE

arrogantrabbit · October 8, 2024, 5:54pm

v1.114.5 is broken on FreeBSD.

2024-10-08T10:52:34-07:00       ERROR   storagenode/peer.go:1010        Failed preflight check. {"Process": "storagenode", "error": "system clock is out of sync: system clock is out of sync with all trusted satellites", "errorVerbose": "system clock is out of sync: system clock is out of sync with all trusted satellites\n\tstorj.io/storj/storagenode/preflight.(*LocalTime).Check:96\n\tstorj.io/storj/storagenode.(*Peer).Run:1009\n\tmain.cmdRun:125\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.cleanup.func1.4:392\n\tstorj.io/common/process.cleanup.func1:410\n\tgithub.com/spf13/cobra.(*Command).execute:983\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1115\n\tgithub.com/spf13/cobra.(*Command).Execute:1039\n\tstorj.io/common/process.ExecWithCustomOptions:112\n\tmain.main:34\n\truntime.main:271"}
storj.io/storj/storagenode.(*Peer).Run
        /go/src/storj.io/storj/storagenode/peer.go:1010
main.cmdRun
        /go/src/storj.io/storj/cmd/storagenode/cmd_run.go:125
main.newRunCmd.func1
        /go/src/storj.io/storj/cmd/storagenode/cmd_run.go:33
storj.io/common/process.cleanup.func1.4
        /go/pkg/mod/storj.io/common@v0.0.0-20240913104706-8fd6fa0286a3/process/exec_conf.go:392
storj.io/common/process.cleanup.func1
        /go/pkg/mod/storj.io/common@v0.0.0-20240913104706-8fd6fa0286a3/process/exec_conf.go:410
github.com/spf13/cobra.(*Command).execute
        /go/pkg/mod/github.com/spf13/cobra@v1.8.0/command.go:983
github.com/spf13/cobra.(*Command).ExecuteC
        /go/pkg/mod/github.com/spf13/cobra@v1.8.0/command.go:1115
github.com/spf13/cobra.(*Command).Execute
        /go/pkg/mod/github.com/spf13/cobra@v1.8.0/command.go:1039
storj.io/common/process.ExecWithCustomOptions
        /go/pkg/mod/storj.io/common@v0.0.0-20240913104706-8fd6fa0286a3/process/exec_conf.go:112
main.main
        /go/src/storj.io/storj/cmd/storagenode/main.go:34
runtime.main
        /usr/local/go/src/runtime/proc.go:271
2024-10-08T10:52:34-07:00       ERROR   storagenode/cmd_run.go:50       failure during run      {"Process": "storagenode", "error": "system clock is out of sync: system clock is out of sync with all trusted satellites", "errorVerbose": "system clock is out of sync: system clock is out of sync with all trusted satellites\n\tstorj.io/storj/storagenode/preflight.(*LocalTime).Check:96\n\tstorj.io/storj/storagenode.(*Peer).Run:1009\n\tmain.cmdRun:125\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.cleanup.func1.4:392\n\tstorj.io/common/process.cleanup.func1:410\n\tgithub.com/spf13/cobra.(*Command).execute:983\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1115\n\tgithub.com/spf13/cobra.(*Command).Execute:1039\n\tstorj.io/common/process.ExecWithCustomOptions:112\n\tmain.main:34\n\truntime.main:271"}
main.cmdRun.func1
        /go/src/storj.io/storj/cmd/storagenode/cmd_run.go:50
main.cmdRun
        /go/src/storj.io/storj/cmd/storagenode/cmd_run.go:128
main.newRunCmd.func1
        /go/src/storj.io/storj/cmd/storagenode/cmd_run.go:33
storj.io/common/process.cleanup.func1.4
        /go/pkg/mod/storj.io/common@v0.0.0-20240913104706-8fd6fa0286a3/process/exec_conf.go:392
storj.io/common/process.cleanup.func1
        /go/pkg/mod/storj.io/common@v0.0.0-20240913104706-8fd6fa0286a3/process/exec_conf.go:410
github.com/spf13/cobra.(*Command).execute
        /go/pkg/mod/github.com/spf13/cobra@v1.8.0/command.go:983
github.com/spf13/cobra.(*Command).ExecuteC
        /go/pkg/mod/github.com/spf13/cobra@v1.8.0/command.go:1115
github.com/spf13/cobra.(*Command).Execute
        /go/pkg/mod/github.com/spf13/cobra@v1.8.0/command.go:1039
storj.io/common/process.ExecWithCustomOptions
        /go/pkg/mod/storj.io/common@v0.0.0-20240913104706-8fd6fa0286a3/process/exec_conf.go:112
main.main
        /go/src/storj.io/storj/cmd/storagenode/main.go:34
runtime.main
        /usr/local/go/src/runtime/proc.go:271
Error: system clock is out of sync: system clock is out of sync with all trusted satellites

I assure you, clock is fine, and version v1.113.2 works fine.

Pinging some folks who may be able to stop rollout: @bre @Knowledge @Alexey @littleskunk

Morisey · October 8, 2024, 6:04pm

its same on windoze , Alex told us to always check versions on version.storj.io and not to upgrade prematurely , i did checked , and it shows that 1.114.5 is suggested , and its still throws same error as previous 1.114.x version , time sync , 1.113.x versions worked just fine.

back to 1.113.4 , that actually works

arrogantrabbit · October 8, 2024, 6:05pm

My scripts grab suggested version from that URL of course. 1.114.5 is being rolled out, and it’s now suggested versions.

All my nodes will die within 15 min.

"suggested":{"version":"1.114.5","url":"https://github.com/storj/storj/releases/download/v1.114.5/storagenode_{os}_{arch}.zip"}

Knowledge · October 8, 2024, 6:10pm

I’ve let the engineers know. Will follow up soon.

Vadim · October 8, 2024, 6:17pm

I have same on windows

alpharabbit · October 8, 2024, 6:17pm

Adding this to the config solved the problem for my windows nodes.

storage2.trust.sources: https://www.storj.io/dcs-satellites

I guess the new version defaults to a wrong url.

Vadim · October 8, 2024, 6:22pm

For me it solving the problem thank you

etienneb · October 8, 2024, 6:36pm

Thanks, enabling this settings solved it too on my TrueNAS Core 13.3 jail.
(My jail was just automatically updated to 1.114.5 when it went down, glad it works again)

arrogantrabbit · October 8, 2024, 6:42pm

Now Storj needs to do internal investigation to determine what gap in their QA processes allowed this escape, and address it.

I can’t image how the bug that reproduces on start, across OSes, can be missed. Unless they don’t test upgrade scenarios, and always install from scratch?

etienneb · October 8, 2024, 6:46pm

I do get a QUIC Misconfigured error now…

alpharabbit · October 8, 2024, 6:47pm

This seems to be related to the url change: Trusted Satellite Subdomain Updates

Vadim · October 8, 2024, 6:51pm

I would add enother problem, after update ingress suddenly near 0

arrogantrabbit · October 8, 2024, 6:53pm

Same.

On both nodes that managed to update before I nuked updater service I see pretty much this:

Not incoming traffic.

I’m now going to disable updaters and only update when node reports allHealhy=false. Who has time for this…

arrogantrabbit · October 8, 2024, 6:58pm

Then the better configuration would be

storage2.trust.sources: https://static.storj.io/dcs-satellites

Vadim · October 8, 2024, 7:07pm

looks like egress somehow working but ingress on 108 nodes 0

dancekid · October 8, 2024, 7:15pm

Hey Vadim just send your Ingress to my nodes. I could save some TB.
I am always here to help…

Michael

alpharabbit · October 8, 2024, 7:20pm

My 1.114.5 ubuntu node shows normal ingress/egress but windows node only egress. So I downgraded windows to 1.113.4.

Vadim · October 8, 2024, 8:01pm

One more minor issue with low priority.
2024-10-08T22:56:28+03:00 WARN collector collector/service.go:113 unable to delete piece {“Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Piece ID”: “DJU3RO7HDWXAAVAY2VKPWEKA3KWUUEIA2AOMAI7ZF5737ZGRGWXA”, “error”: “pieces error: filestore error: file does not exist”, “errorVerbose”: “pieces error: filestore error: file does not exist\n\tstorj.io/storj/storagenode/blobstore/filestore.(*blobStore).Stat:126\n\tstorj.io/storj/storagenode/blobstore/statcache.(*CachedStatBlobstore).Stat:68\n\tstorj.io/storj/storagenode/pieces.(*BlobsUsageCache).pieceSizes:350\n\tstorj.io/storj/storagenode/pieces.(*BlobsUsageCache).DeleteWithStorageFormat:329\n\tstorj.io/storj/storagenode/pieces.(*Store).DeleteSkipV0:375\n\tstorj.io/storj/storagenode/collector.(*Service).Collect:111\n\tstorj.io/storj/storagenode/collector.(*Service).Run.func1:68\n\tstorj.io/common/sync2.(*Cycle).Run:99\n\tstorj.io/storj/storagenode/collector.(*Service).Run:64\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:44\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78”}
storj.io/storj/storagenode/collector.(*Service).Collect
/go/src/storj.io/storj/storagenode/collector/service.go:113
storj.io/storj/storagenode/collector.(*Service).Run.func1
/go/src/storj.io/storj/storagenode/collector/service.go:68
storj.io/common/sync2.(*Cycle).Run
/go/pkg/mod/storj.io/common@v0.0.0-20240913104706-8fd6fa0286a3/sync2/cycle.go:99
storj.io/storj/storagenode/collector.(*Service).Run
/go/src/storj.io/storj/storagenode/collector/service.go:64
storj.io/storj/private/lifecycle.(*Group).Run.func2.1
/go/src/storj.io/storj/private/lifecycle/group.go:87
runtime/pprof.Do
/usr/local/go/src/runtime/pprof/runtime.go:44
storj.io/storj/private/lifecycle.(*Group).Run.func2
/go/src/storj.io/storj/private/lifecycle/group.go:86
golang.org/x/sync/errgroup.(*Group).Go.func1
/go/pkg/mod/golang.org/x/sync@v0.8.0/errgroup/errgroup.go:78

in Old time it was 1 row, now it like A.S. Pushkin poetry size, all understand that it is classic, but no one want to learn it in school.

arrogantrabbit · October 8, 2024, 8:03pm

Downgrade does not seem to be possible due to database versioning.

arrogantrabbit · October 8, 2024, 8:04pm

Yea, someone dropped multiple balls on this release.