🔥 V1.114.5 broken on FreeBSD 13.3-RELEASE

Vadim · October 8, 2024, 8:05pm

quote=“alpharabbit, post:17, topic:28176, full:true”]
My 1.114.5 ubuntu node shows normal ingress/egress but windows node only egress. So I downgraded windows to 1.113.4.
[/quote]

then interresting how he mannaged downgrade it?

arrogantrabbit · October 8, 2024, 8:06pm

Perhaps xier nuked databases. @alpharabbit ?

Vadim · October 8, 2024, 8:16pm

I think it will be wise to stop relese, as if all windows nodes stop getting ingress, it will be problematic.

Vadim · October 8, 2024, 8:29pm

Minor issue stopped to be minor, because or this row side, some of logs are up to 40 GB in 2h so c:\ start to go out of space and cant maintain databases.

thepaul · October 8, 2024, 8:31pm

The rollout has been stopped. Trying to get to the bottom of what the problem is here. While information is still coming in, can anyone verify whether this release brokenness manifests on all platforms, or just Windows and FreeBSD? And does anyone have a full debug log of a node from startup to preflight-check-failure?

So far I’ve seen a few reports that things are unbroken on Linux at least.

digitalfrank · October 8, 2024, 8:33pm

I stopped all automatic updates on all my 30 nodes for now. I have the 1.113.2 yet.

Ruskiem · October 8, 2024, 8:35pm

pssst. we are the TESTERS

unfortunately i turned all logs to “info” level, coz space problems.
Also did not notice any disruption on any of my 10 nodes, all windows, probbaly none had 1.114 yet.

Vadim · October 8, 2024, 8:41pm

My WARN level log gone up to 40 GB in 2h, so I cant read any log file. I will try to make debug

Vadim · October 8, 2024, 8:44pm

where can i upload it? here it will be to big
update, looks here works also

Vadim · October 8, 2024, 8:45pm

2024-10-08T23:42:40+03:00 INFO process/exec_conf.go:318 Configuration loaded {“Location”: “C:\Program Files\Storj\Storage Node\config.yaml”}
2024-10-08T23:42:40+03:00 INFO process/tracing.go:73 Anonymized tracing enabled
2024-10-08T23:42:40+03:00 DEBUG tracing collector monkit-jaeger@v0.0.0-20240221095020-52b0792fa6cd/thrift.go:149 started
2024-10-08T23:42:40+03:00 DEBUG process/debug.go:39 debug server listening on 127.0.0.1:50186
2024-10-08T23:42:40+03:00 INFO operator/operator.go:42 Operator email {“Address”: “vadim@nvp.ee”}
2024-10-08T23:42:40+03:00 INFO operator/operator.go:56 Operator wallet {“Address”: “0x4d81d604814253377f88ef5f641515ed0acd1529”}
2024-10-08T23:42:40+03:00 DEBUG db.filestatcache statcache/badger.go:105 All 148 tables opened in 11ms

2024-10-08T23:42:40+03:00 DEBUG db.filestatcache statcache/badger.go:105 Discard stats nextEmptySlot: 0

2024-10-08T23:42:40+03:00 DEBUG db.filestatcache statcache/badger.go:105 Set nextTxnTs to 30146251
2024-10-08T23:42:40+03:00 DEBUG db.filestatcache statcache/badger.go:105 Deleting empty file: D:\filestatcache\000012.vlog
2024-10-08T23:42:40+03:00 DEBUG storagenode/peer.go:344 Version info {“Version”: “1.114.5”, “Commit Hash”: “69af99d96d541eba191b6ded0edd87e7e0646608-dirty”, “Build Timestamp”: “2024-10-02 12:07:06 +0300 EEST”, “Release Build”: false}
2024-10-08T23:42:40+03:00 DEBUG db.filestatcache statcache/badger.go:109 [2] LOG Compact. Added 4999 keys. Skipped 0 keys. Iteration took: 286ms
2024-10-08T23:42:40+03:00 DEBUG process/metrics.go:104 Telemetry disabled
2024-10-08T23:42:40+03:00 INFO db.migration migrate/versions.go:221 Database Version {“version”: 62}
2024-10-08T23:42:40+03:00 DEBUG db migrate/versions.go:147 Database version is up to date {“version”: 62}
2024-10-08T23:42:40+03:00 INFO preflight:localtime preflight/localtime.go:54 start checking local system clock with trusted satellites’ system clock.
2024-10-08T23:42:40+03:00 ERROR storagenode/peer.go:1010 Failed preflight check. {“error”: “system clock is out of sync: system clock is out of sync with all trusted satellites”, “errorVerbose”: “system clock is out of sync: system clock is out of sync with all trusted satellites\n\tstorj.io/storj/storagenode/preflight.(*LocalTime).Check:96\n\tstorj.io/storj/storagenode.(*Peer).Run:1009\n\tmain.cmdRun:125\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.cleanup.func1.4:392\n\tstorj.io/common/process.cleanup.func1:410\n\tgithub.com/spf13/cobra.(*Command).execute:983\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1115\n\tgithub.com/spf13/cobra.(*Command).Execute:1039\n\tstorj.io/common/process.ExecWithCustomOptions:112\n\tstorj.io/common/process.ExecWithCustomConfigAndLogger:77\n\tstorj.io/common/process.ExecWithCustomConfig:72\n\tstorj.io/common/process.Exec:62\n\tmain.(*service).Execute.func1:107\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78”}
storj.io/storj/storagenode.(*Peer).Run
/go/src/storj.io/storj/storagenode/peer.go:1010
main.cmdRun
/go/src/storj.io/storj/cmd/storagenode/cmd_run.go:125
main.newRunCmd.func1
/go/src/storj.io/storj/cmd/storagenode/cmd_run.go:33
storj.io/common/process.cleanup.func1.4
/go/pkg/mod/storj.io/common@v0.0.0-20240913104706-8fd6fa0286a3/process/exec_conf.go:392
storj.io/common/process.cleanup.func1
/go/pkg/mod/storj.io/common@v0.0.0-20240913104706-8fd6fa0286a3/process/exec_conf.go:410
github.com/spf13/cobra.(*Command).execute
/go/pkg/mod/github.com/spf13/cobra@v1.8.0/command.go:983
github.com/spf13/cobra.(*Command).ExecuteC
/go/pkg/mod/github.com/spf13/cobra@v1.8.0/command.go:1115
github.com/spf13/cobra.(*Command).Execute
/go/pkg/mod/github.com/spf13/cobra@v1.8.0/command.go:1039
storj.io/common/process.ExecWithCustomOptions
/go/pkg/mod/storj.io/common@v0.0.0-20240913104706-8fd6fa0286a3/process/exec_conf.go:112
storj.io/common/process.ExecWithCustomConfigAndLogger
/go/pkg/mod/storj.io/common@v0.0.0-20240913104706-8fd6fa0286a3/process/exec_conf.go:77
storj.io/common/process.ExecWithCustomConfig
/go/pkg/mod/storj.io/common@v0.0.0-20240913104706-8fd6fa0286a3/process/exec_conf.go:72
storj.io/common/process.Exec
/go/pkg/mod/storj.io/common@v0.0.0-20240913104706-8fd6fa0286a3/process/exec_conf.go:62
main.(*service).Execute.func1
/go/src/storj.io/storj/cmd/storagenode/service_windows.go:107
golang.org/x/sync/errgroup.(*Group).Go.func1
/go/pkg/mod/golang.org/x/sync@v0.8.0/errgroup/errgroup.go:78
2024-10-08T23:42:40+03:00 DEBUG db.filestatcache statcache/badger.go:109 Closing database
2024-10-08T23:42:40+03:00 DEBUG db.filestatcache statcache/badger.go:105 Lifetime L0 stalled for: 0s

2024-10-08T23:42:41+03:00 DEBUG db.filestatcache statcache/badger.go:109 [2] LOG Compact. Added 73340 keys. Skipped 3550 keys. Iteration took: 889ms
2024-10-08T23:42:41+03:00 DEBUG db.filestatcache statcache/badger.go:109 [2] LOG Compact. Added 78664 keys. Skipped 4930 keys. Iteration took: 933ms
2024-10-08T23:42:42+03:00 DEBUG db.filestatcache statcache/badger.go:109 [2] LOG Compact. Added 75855 keys. Skipped 1208 keys. Iteration took: 655ms
2024-10-08T23:42:42+03:00 DEBUG db.filestatcache statcache/badger.go:109 [2] LOG Compact. Added 76645 keys. Skipped 4724 keys. Iteration took: 2.031s
2024-10-08T23:42:43+03:00 DEBUG db.filestatcache statcache/badger.go:109 [2] LOG Compact. Added 75691 keys. Skipped 1960 keys. Iteration took: 2.329s
2024-10-08T23:42:43+03:00 DEBUG db.filestatcache statcache/badger.go:109 [2] LOG Compact. Added 72400 keys. Skipped 1905 keys. Iteration took: 1.534s
2024-10-08T23:42:43+03:00 DEBUG db.filestatcache statcache/badger.go:109 [2] LOG Compact. Added 77491 keys. Skipped 4806 keys. Iteration took: 583ms
2024-10-08T23:42:43+03:00 DEBUG db.filestatcache statcache/badger.go:109 [2] LOG Compact. Added 77090 keys. Skipped 3809 keys. Iteration took: 1.038s
2024-10-08T23:42:44+03:00 DEBUG db.filestatcache statcache/badger.go:109 [2] LOG Compact. Added 75910 keys. Skipped 4191 keys. Iteration took: 597ms
2024-10-08T23:42:44+03:00 DEBUG db.filestatcache statcache/badger.go:109 [2] LOG Compact. Added 74650 keys. Skipped 174 keys. Iteration took: 2.464s
2024-10-08T23:42:45+03:00 DEBUG db.filestatcache statcache/badger.go:109 [2] LOG Compact. Added 72077 keys. Skipped 1172 keys. Iteration took: 1.198s
2024-10-08T23:42:45+03:00 DEBUG db.filestatcache statcache/badger.go:109 [2] LOG Compact. Added 77479 keys. Skipped 4729 keys. Iteration took: 2.281s
2024-10-08T23:42:46+03:00 DEBUG db.filestatcache statcache/badger.go:109 [2] LOG Compact. Added 72925 keys. Skipped 2518 keys. Iteration took: 3.105s
2024-10-08T23:42:47+03:00 DEBUG db.filestatcache statcache/badger.go:109 [2] LOG Compact. Added 77428 keys. Skipped 4821 keys. Iteration took: 1.205s
2024-10-08T23:42:47+03:00 DEBUG db.filestatcache statcache/badger.go:109 [2] LOG Compact. Added 72469 keys. Skipped 1914 keys. Iteration took: 1.94s
2024-10-08T23:42:47+03:00 DEBUG db.filestatcache statcache/badger.go:109 [2] LOG Compact. Added 77465 keys. Skipped 5279 keys. Iteration took: 6.947s
2024-10-08T23:42:47+03:00 DEBUG db.filestatcache statcache/badger.go:109 [2] LOG Compact. Added 32702 keys. Skipped 1607 keys. Iteration took: 468ms
2024-10-08T23:42:47+03:00 DEBUG db.filestatcache statcache/badger.go:109 Discard stats: map
2024-10-08T23:42:48+03:00 DEBUG db.filestatcache statcache/badger.go:109 [2] LOG Compact. Added 77654 keys. Skipped 5593 keys. Iteration took: 3.408s
2024-10-08T23:42:48+03:00 DEBUG db.filestatcache statcache/badger.go:109 [2] LOG Compact. Added 73185 keys. Skipped 1884 keys. Iteration took: 1.919s
2024-10-08T23:42:48+03:00 DEBUG db.filestatcache statcache/badger.go:109 [2] LOG Compact. Added 77449 keys. Skipped 4736 keys. Iteration took: 1.229s
2024-10-08T23:42:48+03:00 DEBUG db.filestatcache statcache/badger.go:109 [2] LOG Compact. Added 24174 keys. Skipped 0 keys. Iteration took: 556ms
2024-10-08T23:42:48+03:00 DEBUG db.filestatcache statcache/badger.go:109 Discard stats: map
2024-10-08T23:42:48+03:00 DEBUG db.filestatcache statcache/badger.go:109 [2] LOG Compact. Added 76231 keys. Skipped 2530 keys. Iteration took: 1.139s
2024-10-08T23:42:49+03:00 DEBUG db.filestatcache statcache/badger.go:109 [2] LOG Compact. Added 77392 keys. Skipped 4733 keys. Iteration took: 843ms
2024-10-08T23:42:49+03:00 DEBUG db.filestatcache statcache/badger.go:109 [2] LOG Compact. Added 76082 keys. Skipped 1846 keys. Iteration took: 1.012s
2024-10-08T23:42:49+03:00 DEBUG db.filestatcache statcache/badger.go:109 [2] LOG Compact. Added 77923 keys. Skipped 4989 keys. Iteration took: 660ms
2024-10-08T23:42:50+03:00 DEBUG db.filestatcache statcache/badger.go:109 [2] LOG Compact. Added 59381 keys. Skipped 3654 keys. Iteration took: 534ms
2024-10-08T23:42:50+03:00 DEBUG db.filestatcache statcache/badger.go:109 Discard stats: map
2024-10-08T23:42:50+03:00 DEBUG db.filestatcache statcache/badger.go:109 [2] LOG Compact. Added 76841 keys. Skipped 2451 keys. Iteration took: 998ms
2024-10-08T23:42:50+03:00 DEBUG db.filestatcache statcache/badger.go:109 [2] LOG Compact. Added 77648 keys. Skipped 4935 keys. Iteration took: 271ms
2024-10-08T23:42:51+03:00 DEBUG db.filestatcache statcache/badger.go:109 [2] LOG Compact. Added 78433 keys. Skipped 5768 keys. Iteration took: 1.325s
2024-10-08T23:42:51+03:00 DEBUG db.filestatcache statcache/badger.go:109 [2] LOG Compact. Added 77508 keys. Skipped 4731 keys. Iteration took: 638ms
2024-10-08T23:42:51+03:00 DEBUG db.filestatcache statcache/badger.go:109 [2] LOG Compact. Added 65653 keys. Skipped 4086 keys. Iteration took: 214ms
2024-10-08T23:42:51+03:00 DEBUG db.filestatcache statcache/badger.go:109 Discard stats: map
2024-10-08T23:42:51+03:00 DEBUG db.filestatcache statcache/badger.go:109 [2] LOG Compact. Added 75407 keys. Skipped 1005 keys. Iteration took: 866ms
2024-10-08T23:42:52+03:00 DEBUG db.filestatcache statcache/badger.go:109 [2] LOG Compact. Added 74503 keys. Skipped 120 keys. Iteration took: 167ms
2024-10-08T23:42:52+03:00 DEBUG db.filestatcache statcache/badger.go:109 [2] LOG Compact. Added 76361 keys. Skipped 3338 keys. Iteration took: 218ms
2024-10-08T23:42:52+03:00 DEBUG db.filestatcache statcache/badger.go:109 [2] LOG Compact. Added 58110 keys. Skipped 3534 keys. Iteration took: 203ms
2024-10-08T23:42:52+03:00 DEBUG db.filestatcache statcache/badger.go:109 Discard stats: map
2024-10-08T23:42:52+03:00 DEBUG db.filestatcache statcache/badger.go:105 [2] [E] LOG Compact 0->5 (5, 19 → 36 tables with 5 splits). [01691 01692 01693 01694 01695 . 01556 01566 01574 01575 01577 01579 01580 01581 01582 01565 01555 01571 01578 01590 01591 01592 01593 01594 01595 .] → [01696 01712 01718 01720 01723 01724 01726 01727 01701 01703 01708 01710 01716 01719 01722 01698 01699 01706 01714 01721 01725 01728 01729 01730 01731 01700 01704 01705 01707 01711 01713 01697 01702 01709 01715 01717 .], took 12.325s
, deleted 5651446 bytes
2024-10-08T23:42:52+03:00 DEBUG db.filestatcache statcache/badger.go:109 [Compactor: 2] Compaction for level: 0 DONE
2024-10-08T23:42:52+03:00 DEBUG db.filestatcache statcache/badger.go:109 Stopping garbage collection of values.
2024-10-08T23:42:52+03:00 DEBUG db.filestatcache statcache/badger.go:105
Level 0 : NumTables: 00. Size: 0 B of 0 B. Score: 0.00->0.00 StaleData: 0 B Target FileSize: 64 MiB
Level 1 : NumTables: 00. Size: 0 B of 10 MiB. Score: 0.00->0.00 StaleData: 0 B Target FileSize: 2.0 MiB
Level 2 : NumTables: 00. Size: 0 B of 10 MiB. Score: 0.00->0.00 StaleData: 0 B Target FileSize: 2.0 MiB
Level 3 : NumTables: 00. Size: 0 B of 10 MiB. Score: 0.00->0.00 StaleData: 0 B Target FileSize: 2.0 MiB
Level 4 [B]: NumTables: 00. Size: 0 B of 10 MiB. Score: 0.00->0.00 StaleData: 0 B Target FileSize: 2.0 MiB
Level 5 : NumTables: 36. Size: 128 MiB of 75 MiB. Score: 1.70->1.70 StaleData: 95 MiB Target FileSize: 4.0 MiB
Level 6 : NumTables: 124. Size: 752 MiB of 752 MiB. Score: 0.00->0.00 StaleData: 0 B Target FileSize: 8.0 MiB
Level Done

2024-10-08T23:42:52+03:00 DEBUG db.filestatcache statcache/badger.go:109 Waiting for closer
2024-10-08T23:42:52+03:00 ERROR storagenode/cmd_run.go:50 failure during run {“error”: “system clock is out of sync: system clock is out of sync with all trusted satellites”, “errorVerbose”: “system clock is out of sync: system clock is out of sync with all trusted satellites\n\tstorj.io/storj/storagenode/preflight.(*LocalTime).Check:96\n\tstorj.io/storj/storagenode.(*Peer).Run:1009\n\tmain.cmdRun:125\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.cleanup.func1.4:392\n\tstorj.io/common/process.cleanup.func1:410\n\tgithub.com/spf13/cobra.(*Command).execute:983\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1115\n\tgithub.com/spf13/cobra.(*Command).Execute:1039\n\tstorj.io/common/process.ExecWithCustomOptions:112\n\tstorj.io/common/process.ExecWithCustomConfigAndLogger:77\n\tstorj.io/common/process.ExecWithCustomConfig:72\n\tstorj.io/common/process.Exec:62\n\tmain.(*service).Execute.func1:107\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78”}
main.cmdRun.func1
/go/src/storj.io/storj/cmd/storagenode/cmd_run.go:50
main.cmdRun
/go/src/storj.io/storj/cmd/storagenode/cmd_run.go:128
main.newRunCmd.func1
/go/src/storj.io/storj/cmd/storagenode/cmd_run.go:33
storj.io/common/process.cleanup.func1.4
/go/pkg/mod/storj.io/common@v0.0.0-20240913104706-8fd6fa0286a3/process/exec_conf.go:392
storj.io/common/process.cleanup.func1
/go/pkg/mod/storj.io/common@v0.0.0-20240913104706-8fd6fa0286a3/process/exec_conf.go:410
github.com/spf13/cobra.(*Command).execute
/go/pkg/mod/github.com/spf13/cobra@v1.8.0/command.go:983
github.com/spf13/cobra.(*Command).ExecuteC
/go/pkg/mod/github.com/spf13/cobra@v1.8.0/command.go:1115
github.com/spf13/cobra.(*Command).Execute
/go/pkg/mod/github.com/spf13/cobra@v1.8.0/command.go:1039
storj.io/common/process.ExecWithCustomOptions
/go/pkg/mod/storj.io/common@v0.0.0-20240913104706-8fd6fa0286a3/process/exec_conf.go:112
storj.io/common/process.ExecWithCustomConfigAndLogger
/go/pkg/mod/storj.io/common@v0.0.0-20240913104706-8fd6fa0286a3/process/exec_conf.go:77
storj.io/common/process.ExecWithCustomConfig
/go/pkg/mod/storj.io/common@v0.0.0-20240913104706-8fd6fa0286a3/process/exec_conf.go:72
storj.io/common/process.Exec
/go/pkg/mod/storj.io/common@v0.0.0-20240913104706-8fd6fa0286a3/process/exec_conf.go:62
main.(*service).Execute.func1
/go/src/storj.io/storj/cmd/storagenode/service_windows.go:107
golang.org/x/sync/errgroup.(*Group).Go.func1
/go/pkg/mod/golang.org/x/sync@v0.8.0/errgroup/errgroup.go:78
2024-10-08T23:42:52+03:00 FATAL process/exec_conf.go:428 Unrecoverable error {“error”: “system clock is out of sync: system clock is out of sync with all trusted satellites”, “errorVerbose”: “system clock is out of sync: system clock is out of sync with all trusted satellites\n\tstorj.io/storj/storagenode/preflight.(*LocalTime).Check:96\n\tstorj.io/storj/storagenode.(*Peer).Run:1009\n\tmain.cmdRun:125\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.cleanup.func1.4:392\n\tstorj.io/common/process.cleanup.func1:410\n\tgithub.com/spf13/cobra.(*Command).execute:983\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1115\n\tgithub.com/spf13/cobra.(*Command).Execute:1039\n\tstorj.io/common/process.ExecWithCustomOptions:112\n\tstorj.io/common/process.ExecWithCustomConfigAndLogger:77\n\tstorj.io/common/process.ExecWithCustomConfig:72\n\tstorj.io/common/process.Exec:62\n\tmain.(*service).Execute.func1:107\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78”}
storj.io/common/process.cleanup.func1
/go/pkg/mod/storj.io/common@v0.0.0-20240913104706-8fd6fa0286a3/process/exec_conf.go:428
github.com/spf13/cobra.(*Command).execute
/go/pkg/mod/github.com/spf13/cobra@v1.8.0/command.go:983
github.com/spf13/cobra.(*Command).ExecuteC
/go/pkg/mod/github.com/spf13/cobra@v1.8.0/command.go:1115
github.com/spf13/cobra.(*Command).Execute
/go/pkg/mod/github.com/spf13/cobra@v1.8.0/command.go:1039
storj.io/common/process.ExecWithCustomOptions
/go/pkg/mod/storj.io/common@v0.0.0-20240913104706-8fd6fa0286a3/process/exec_conf.go:112
storj.io/common/process.ExecWithCustomConfigAndLogger
/go/pkg/mod/storj.io/common@v0.0.0-20240913104706-8fd6fa0286a3/process/exec_conf.go:77
storj.io/common/process.ExecWithCustomConfig
/go/pkg/mod/storj.io/common@v0.0.0-20240913104706-8fd6fa0286a3/process/exec_conf.go:72
storj.io/common/process.Exec
/go/pkg/mod/storj.io/common@v0.0.0-20240913104706-8fd6fa0286a3/process/exec_conf.go:62
main.(*service).Execute.func1
/go/src/storj.io/storj/cmd/storagenode/service_windows.go:107
golang.org/x/sync/errgroup.(*Group).Go.func1
/go/pkg/mod/golang.org/x/sync@v0.8.0/errgroup/errgroup.go:78

thepaul · October 8, 2024, 8:46pm

@Vadim Thank you, perfect, that should help.

Vadim · October 8, 2024, 8:48pm

what about ingress, all windows nodes dont have ingress about 2h after update already

arrogantrabbit · October 8, 2024, 9:01pm

I have the full log, if Vadims logs are not enough I will share mine in a bit.

alpharabbit · October 8, 2024, 9:06pm

Databases are not essential, I just deleted them.

thepaul · October 8, 2024, 9:33pm

Ok, the problem is that this build was somehow made without release defaults. We will still need to dig into why and how, to stop it from happening again, but the quick explanation is that there was no list of trusted satellites for the node to check, and that’s why it immediately errors saying its clock is out of sync with all satellites (technically true because there are no known satellites, but yes, it’s a misleading error message).

Setting storage2.trust.sources is probably not sufficient. Other things are going to be configured wrong. We need a new release build. We’ll get that out as soon as we can.

While we’re working on that we’ll also see if we can come up with a quick downgrade option, in case that gets done more quickly.

As to why this wasn’t caught in testing, apparently this is because the binaries that got built for these releases are not the same as what was tested (because we test before tagging and build after tagging, so if the build is not done in an identical way there are possibilities for new problems to creep in).

To clarify: the non-release defaults likely explain the ingress problem as well.

cois · October 8, 2024, 9:33pm

Same here. Using FreeBSD

mdmeyerpfa · October 8, 2024, 9:40pm

Fixed it for my Window node too. Once the corrected release is rolled out will I need to recomment the storage 2.trusted.sources line in the config?

JDA · October 8, 2024, 9:46pm

Windows Node broken here also…

arrogantrabbit · October 8, 2024, 10:02pm

Logs from FreeBSD, just in case, since the culprit has already been found.

2024-10-08T10:38:37-07:00	INFO	process/exec_conf.go:318	Configuration loaded	{"Process": "storagenode", "Location": "/mnt/storagenode-three/config/config.yaml"}
2024-10-08T10:38:37-07:00	INFO	process/tracing.go:73	Anonymized tracing enabled	{"Process": "storagenode"}
2024-10-08T10:38:37-07:00	INFO	operator/operator.go:42	Operator email	{"Process": "storagenode", "Address": "alex@saspus.com"}
2024-10-08T10:38:37-07:00	INFO	operator/operator.go:56	Operator wallet	{"Process": "storagenode", "Address": "0xa6A93eF3Cd9E5C9EcdA303ad6c56993Da0Eff8DA"}
2024-10-08T10:38:37-07:00	INFO	server	server/fastopen_freebsd.go:42	kernel support for server-side tcp fast open enabled.	{"Process": "storagenode"}
2024-10-08T10:38:38-07:00	INFO	db.migration.62	migrate/versions.go:196	Add total_content_size, piece_counts, resume_point columns to used_space_per_prefix table	{"Process": "storagenode"}
2024-10-08T10:38:38-07:00	INFO	db.migration	migrate/versions.go:221	Database Version	{"Process": "storagenode", "version": 62}
2024-10-08T10:38:38-07:00	INFO	preflight:localtime	preflight/localtime.go:54	start checking local system clock with trusted satellites' system clock.	{"Process": "storagenode"}
2024-10-08T10:38:38-07:00	ERROR	storagenode/peer.go:1010	Failed preflight check.	{"Process": "storagenode", "error": "system clock is out of sync: system clock is out of sync with all trusted satellites", "errorVerbose": "system clock is out of sync: system clock is out of sync with all trusted satellites\n\tstorj.io/storj/storagenode/preflight.(*LocalTime).Check:96\n\tstorj.io/storj/storagenode.(*Peer).Run:1009\n\tmain.cmdRun:125\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.cleanup.func1.4:392\n\tstorj.io/common/process.cleanup.func1:410\n\tgithub.com/spf13/cobra.(*Command).execute:983\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1115\n\tgithub.com/spf13/cobra.(*Command).Execute:1039\n\tstorj.io/common/process.ExecWithCustomOptions:112\n\tmain.main:34\n\truntime.main:271"}
storj.io/storj/storagenode.(*Peer).Run
	/go/src/storj.io/storj/storagenode/peer.go:1010
main.cmdRun
	/go/src/storj.io/storj/cmd/storagenode/cmd_run.go:125
main.newRunCmd.func1
	/go/src/storj.io/storj/cmd/storagenode/cmd_run.go:33
storj.io/common/process.cleanup.func1.4
	/go/pkg/mod/storj.io/common@v0.0.0-20240913104706-8fd6fa0286a3/process/exec_conf.go:392
storj.io/common/process.cleanup.func1
	/go/pkg/mod/storj.io/common@v0.0.0-20240913104706-8fd6fa0286a3/process/exec_conf.go:410
github.com/spf13/cobra.(*Command).execute
	/go/pkg/mod/github.com/spf13/cobra@v1.8.0/command.go:983
github.com/spf13/cobra.(*Command).ExecuteC
	/go/pkg/mod/github.com/spf13/cobra@v1.8.0/command.go:1115
github.com/spf13/cobra.(*Command).Execute
	/go/pkg/mod/github.com/spf13/cobra@v1.8.0/command.go:1039
storj.io/common/process.ExecWithCustomOptions
	/go/pkg/mod/storj.io/common@v0.0.0-20240913104706-8fd6fa0286a3/process/exec_conf.go:112
main.main
	/go/src/storj.io/storj/cmd/storagenode/main.go:34
runtime.main
	/usr/local/go/src/runtime/proc.go:271
2024-10-08T10:38:38-07:00	ERROR	storagenode/cmd_run.go:50	failure during run	{"Process": "storagenode", "error": "system clock is out of sync: system clock is out of sync with all trusted satellites", "errorVerbose": "system clock is out of sync: system clock is out of sync with all trusted satellites\n\tstorj.io/storj/storagenode/preflight.(*LocalTime).Check:96\n\tstorj.io/storj/storagenode.(*Peer).Run:1009\n\tmain.cmdRun:125\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.cleanup.func1.4:392\n\tstorj.io/common/process.cleanup.func1:410\n\tgithub.com/spf13/cobra.(*Command).execute:983\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1115\n\tgithub.com/spf13/cobra.(*Command).Execute:1039\n\tstorj.io/common/process.ExecWithCustomOptions:112\n\tmain.main:34\n\truntime.main:271"}
main.cmdRun.func1
	/go/src/storj.io/storj/cmd/storagenode/cmd_run.go:50
main.cmdRun
	/go/src/storj.io/storj/cmd/storagenode/cmd_run.go:128
main.newRunCmd.func1
	/go/src/storj.io/storj/cmd/storagenode/cmd_run.go:33
storj.io/common/process.cleanup.func1.4
	/go/pkg/mod/storj.io/common@v0.0.0-20240913104706-8fd6fa0286a3/process/exec_conf.go:392
storj.io/common/process.cleanup.func1
	/go/pkg/mod/storj.io/common@v0.0.0-20240913104706-8fd6fa0286a3/process/exec_conf.go:410
github.com/spf13/cobra.(*Command).execute
	/go/pkg/mod/github.com/spf13/cobra@v1.8.0/command.go:983
github.com/spf13/cobra.(*Command).ExecuteC
	/go/pkg/mod/github.com/spf13/cobra@v1.8.0/command.go:1115
github.com/spf13/cobra.(*Command).Execute
	/go/pkg/mod/github.com/spf13/cobra@v1.8.0/command.go:1039
storj.io/common/process.ExecWithCustomOptions
	/go/pkg/mod/storj.io/common@v0.0.0-20240913104706-8fd6fa0286a3/process/exec_conf.go:112
main.main
	/go/src/storj.io/storj/cmd/storagenode/main.go:34
runtime.main
	/usr/local/go/src/runtime/proc.go:271
Error: system clock is out of sync: system clock is out of sync with all trusted satellites

mdmeyerpfa · October 8, 2024, 10:26pm

But now the dashboard keeps saying QUIC is misconfigured. Restarting the node service does not fix it.