Hello, since the node has updated to version 1.92.1, I have this problem, it seems that the node searches for the new version and enters the loop. Resumes only by manually restarting
This is a docker version, not Windows GUI.
It doesn’t download binaries actually, because your versions already up to date.
The problem is with TLS negotiation, which points to the problems with your network - something is trying to interfere to the connection between the container and the version.storj.io site or blocks it. Do you have some kind of a smart firewall?
Hi Alexey, no, I don’t have smart firewalls set up. The node runs on qnap arm and the node on the other disk, stuck at version 1.91.2 works correctly. the problem arose with the update to 1.92
As mentioned, I spoke too soon. The second node also updated and is misconfigured and in a loop. Only a restart seems to unlock it
does it continues to behave like this after the restart?
They work for a few hours and then go into a loop
With the same TLS error?
I wonder what is blocking your connection?
My docker nodes doesn’t have this issue though.
This is the second node
2023-12-05T05:38:19Z ERROR Error retrieving version info. {“Process”: “storagenode-updater”, “error”: “version checker client: Get "https://version.storj.io": net/http: TLS handshake timeout”, “errorVerbose”: “version checker client: Get "https://version.storj.io": net/http: TLS handshake timeout\n\tstorj.io/storj/private/version/checker.(*Client).All:68\n\tmain.loopFunc:21\n\tstorj.io/common/sync2.(*Cycle).Run:160\n\tmain.cmdRun:136\n\tstorj.io/private/process.cleanup.func1.4:393\n\tstorj.io/private/process.cleanup.func1:411\n\tgithub.com/spf13/cobra.(*Command).execute:852\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:960\n\tgithub.com/spf13/cobra.(*Command).Execute:897\n\tstorj.io/private/process.ExecWithCustomOptions:112\n\tstorj.io/private/process.ExecWithCustomConfigAndLogger:77\n\tmain.main:20\n\truntime.main:267”}
I found this too, an error so long that I can’t post in full
{"log":"2023-12-04T10:42:50Z\u0009INFO\u0009services\u0009slow shutdown\u0009{\"process\": \"storagenode\", \"stack\": \"goroutine 970 [running]:\\nstorj.io/storj/private/lifecycle.(*Group).logStackTrace.func1()\\n\\t/go/src/storj.io/storj/private/lifecycle/group.go:107 +0x70\\nsync.(*Once).doSlow(0x4001024a80?, 0x400009b240?)\\n\\t/usr/local/go/src/sync/once.go:74 +0x100\\nsync.(*Once).Do(...)\\n\\t/usr/local/go/src/sync/once.go:65\\nstorj.io/storj/private/lifecycle.(*Group).logStackTrace(0x400006aeb0?)\\n\\t/go/src/storj.io/storj/private/lifecycle/group.go:104 +0x48\\nstorj.io/storj/private/lifecycle.(*Group).Run.func1({0x13850c0?, 0x40002b2090?})\\n\\t/go/src/storj.io/storj/private/lifecycle/group.go:77 +0x210\\nruntime/pprof.Do({0x1385378?, 0x4000776d20?}, {{0x4000337460?, 0x471a94?, 0x1?}}, 0x400076ad40)\\n\\t/usr/local/go/src/runtime/pprof/runtime.go:51 +0x80\\ncreated by storj.io/storj/private/lifecycle.(*Group).Run in goroutine 1\\n\\t/go/src/storj.io/storj/private/lifecycle/group.go:64 +0x3d4\\n\\ngoroutine 1 [semacquire, 1157 minutes]:\\nsync.runtime_Semacquire(0x0?)\\n\\t/usr/local/go/src/runtime/sema.go:62 +0x2c\\nsync.(*WaitGroup).Wait(0x400076ab48)\\n\\t/usr/local/go/src/sync/waitgroup.go:116 +0x74\\ngolang.org/x/sync/errgroup.(*Group).Wait(0x400076ab40)\\n\\t/go/pkg/mod/golang.org/x/sync@v0.3.0/errgroup/errgroup.go:53 +0x2c\\nstorj.io/storj/storagenode.(*Peer).Run(0x4000134780, {0x13850f8, 0x40006a4be0})\\n\\t/go/src/storj.io/storj/storagenode/peer.go:918 +0x304\\nmain.cmdRun(0x7f4df7daf0?, 0x4000089c00)\\n\\t/go/src/storj.io/storj/cmd/storagenode/cmd_run.go:120 +0xb2c\\nmain.newRunCmd.func1(0xeb0140?, {0x40001accc0?, 0x40001ab900?, 0x981280?})\\n\\t/go/src/storj.io/storj/cmd/storagenode/cmd_run.go:32 +0x20\\nstorj.io/private/process.cleanup.func1.4({0x1385378?, 0x400024a8c0})\\n\\t/go/pkg/mod/storj.io/private@v0.0.0-20230912093002-ca2d4ab44679/process/exec_conf.go:402 +0x1b4\\nstorj.io/private/process.cleanup.func1(0x40001ab900, {0x4000250500, 0x0, 0x10})\\n\\t/go/pkg/mod/storj.io/private@v0.0.0-20230912093002-ca2d4ab44679/process/exec_conf.go:420 +0x18b0\\ngithub.com/spf13/cobra.(*Command).execute(0x40001ab900, {0x4000250400, 0x10, 0x10})\\n\\t/go/pkg/mod/github.com/spf13/cobra@v1.1.3/command.go:852 +0x554\\ngithub.com/spf13/cobra.(*Command).ExecuteC(0x40001aac80)\\n\\t/go/pkg/mod/github.com/spf13/cobra@v1.1.3/command.go:960 +0x310\\ngithub.com/spf13/cobra.(*Command).Execute(...)\\n\\t/go/pkg/mod/github.com/spf13/cobra@v1.1.3/command.go:897\\nstorj.io/private/process.ExecWithCustomOptions(0x40001aac80, {0x1, 0x1, 0x1, 0x0, 0x1282590, 0x40001b77c0})\\n\\t/go/pkg/mod/storj.io/private@v0.0.0-20230912093002-ca2d4ab44679/process/exec_conf.go:113 +0x1bc\\nmain.main()\\n\\t/go/src/storj.io/storj/cmd/storagenode/main.go:30 +0x1a8\\n\\ngoroutine 5 [runnable]:\\ncontext.(*cancelCtx).Done(0x400009e3c0?)\\n\\t/usr/local/go/src/context/context.go:433 +0x22c\\nstorj.io/monkit-jaeger.(*UDPCollector).Run(0x4000100600, {0x13850f8, 0x400009e3c0})\\n\\t/go/pkg/mod/storj.io/monkit-jaeger@v0.0.0-20220915074555-d100d7589f41/udp.go:177 +0x618\\nstorj.io/private/process.cleanup.func1.2()\\n\\t/go/pkg/mod/storj.io/private@v0.0.0-20230912093002-ca2d4ab44679/process/exec_conf.go:352 +0x28\\ngolang.org/x/sync/errgroup.(*Group).Go.func1()\\n\\t/go/pkg/mod/golang.org/x/sync@v0.3.0/errgroup/errgroup.go:75 +0x58\\ncreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 1\\n\\t/go/pkg/mod/golang.org/x/sync@v0.3.0/errgroup/errgroup.go:72 +0x98\\n\\ngoroutine 579342 [runnable]:\\nstorj.io/drpc/drpcmanager.(*Manager).NewServerStream(0x4000a421c0, {0x13850c0, 0x4002e3fd10})\\n\\t/go/pkg/mod/storj.io/drpc@v0.0.33/drpcmanager/manager.go:445 +0x4e8\\nstorj.io/drpc/drpcserver.(*Server).ServeOne(0x4000370400, {0x1385730, 0x40002b3740}, {0x137f740?, 0x4001025680?})\\n\\t/go/pkg/mod/storj.io/drpc@v0.0.33/drpcserver/server.go:62 +0x164\\nstorj.io/drpc/drpcserver.(*Server).Serve.func2({0x1385730, 0x40002b3740})\\n\\t/go/pkg/mod/storj.io/drpc@v0.0.33/drpcserver/server.go:114 +0x5c\\nstorj.io/drpc/drpcctx.(*Tracker).track(0x40002b3740, 0x0?)\\n\\t/go/pkg/mod/storj.io/drpc@v0.0.33/drpcctx/tracker.go:35 +0x3c\\ncreated by storj.io/drpc/drpcctx.(*Tracker).Run in goroutine 1022\\n\\t/go/pkg/mod/storj.io/drpc@v0.0.33/drpcctx/tracker.go:30 +0x84\\n\\ngoroutine 6 [semacquire, 1158 minutes]:\\nsync.runtime_Semacquire(0x40003dfe38?)\\n\\t/usr/local/go/src/runtime/sema.go:62 +0x2c\\nsync.(*WaitGroup).Wait(0x400022e048)\\n\\t/usr/local/go/src/sync/waitgroup.go:116 +0x74\\ngolang.org/x/sync/errgroup.(*Group).Wait(0x400022e040)\\n\\t/go/pkg/mod/golang.org/x/sync@v0.3.0/errgroup/errgroup.go:53 +0x2c\\nstorj.io/private/debug.(*Server).Run(0x400023d4a0, {0x1385050?, 0x1b19c80?})\\n\\t/go/pkg/mod/storj.io/private@v0.0.0-20230912093002-ca2d4ab44679/debug/server.go:176 +0x114\\nstorj.io/private/process.initDebug.func1()\\n\\t/go/pkg/mod/storj.io/private@v0.0.0-20230912093002-ca2d4ab44679/process/debug.go:38 +0x100\\ncreated by storj.io/private/process.initDebug in goroutine 1\\n\\t/go/pkg/mod/storj.io/private@v0.0.0-20230912093002-ca2d4ab44679/process/debug.go:33 +0xe0\\n\\ngoroutine 8 [syscall, 1158 minutes]:\\nos/signal.signal_recv()\\n\\t/usr/local/go/src/runtime/sigqueue.go:152 +0x30\\nos/signal.loop()\\n\\t/usr/local/go/src/os/signal/signal_unix.go:23 +0x1c\\ncreated by os/signal.Notify.func1.1 in goroutine 1\\n\\t/usr/local/go/src/os/signal/signal.go:151 +0x28\\n\\ngoroutine 51 [chan receive, 1158 minutes]:\\nstorj.io/private/process.Ctx.func1()\\n\\t/go/pkg/mod/storj.io/private@v0.0.0-20230912093002-ca2d4ab44679/process/exec_conf.go:140 +0x38\\ncreated by storj.io/private/process.Ctx in goroutine 1\\n\\t/go/pkg/mod/storj.io/private@v0.0.0-20230912093002-ca2d4ab44679/process/exec_conf.go:139 +0x2d0\\n\\ngoroutine 22 [chan receive, 1158 minutes]:\\nstorj.io/private/debug.(*Server).Run.func1()\\n\\t/go/pkg/mod/storj.io/private@v0.0.0-20230912093002-ca2d4ab44679/debug/server.go:163 +0x40\\ngolang.org/x/sync/errgroup.(*Group).Go.func1()\\n\\t/go/pkg/mod/golang.org/x/sync@v0.3.0/errgroup/errgroup.go:75 +0x58\\ncreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 6\\n\\t/go/pkg/mod/golang.org/x/sync@v0.3.0/errgroup/errgroup.go:72 +0x98\\n\\ngoroutine 23 [IO wait, 1158 minutes]:\\ninternal/poll.runtime_pollWait(0x7f4df71eb0, 0x72)\\n\\t/usr/local/go/src/runtime/netpoll.go:343 +0xa0\\ninternal/poll.(*pollDesc).wait(0x400007c280?, 0xf570e0?, 0x0)\\n\\t/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x28\\ninternal/poll.(*pollDesc).waitRead(...)\\n\\t/usr/local/go/src/internal/poll/fd_poll_runtime.go:89\\ninternal/poll.(*FD).Accept(0x400007c280)\\n\\t/usr/local/go/src/internal/poll/fd_unix.go:611 +0x250\\nnet.(*netFD).accept(0x400007c280)\\n\\t/usr/local/go/src/net/fd_unix.go:172 +0x28\\nnet.(*TCPListener).accept(0x4000041580)\\n\\t/usr/local/go/src/net/tcpsock_posix.go:152 +0x28\\nnet.(*TCPListener).Accept(0x4000041580)\\n\\t/usr/local/go/src/net/tcpsock.go:315 +0x2c\\nnet/http.(*Server).Serve(0x400023d4b8, {0x1380340, 0x4000041580})\\n\\t/usr/local/go/src/net/http/server.go:3056 +0x2b8\\nstorj.io/private/debug.(*Server).Run.func2()\\n\\t/go/pkg/mod/storj.io/private@v0.0.0-20230912093002-ca2d4ab44679/debug/server.go:169 +0x48\\ngolang.org/x/sync/errgroup.(*Group).Go.func1()\\n\\t/go/pkg/mod/golang.org/x/sync@v0.3.0/errgroup/errgroup.go:75 +0x58\\ncreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 6\\n\\t/go/pkg/mod/golang.org/x/sync@v0.3.0/errgroup/errgroup.go:72 +0x98\\n\\ngoroutine 995 [runnable]:\\ncontext.WithCancel.func1()\\n\\t/usr/local/go/src/context/context.go:237 +0x50\\ndatabase/sql.(*Rows).close(0x400018e2d0, {0x0, 0x0})\\n\\t/usr/local/go/src/database/sql/sql.go:3390 +0x174\\ndatabase/sql.(*Rows).Close(0x137d2f0?)\\n\\t/usr/local/go/src/database/sql/sql.go:3367 +0x30\\ndatabase/sql.(*Rows).Next(0x400018e2d0)\\n\\t/usr/local/go/src/database/sql/sql.go:2997 +0x8c\\nstorj.io/private/tagsql.(*sqlRows).Next(0x400003e610?)\\n\\t/go/pkg/mod/storj.io/private@v0.0.0-20230912093002-ca2d4ab44679/tagsql/rows.go:86 +0x24\\nstorj.io/storj/storagenode/storagenodedb.(*satellitesDB).ListGracefulExits(0x40003fe0a0, {0x1385378, 0x4000267180})\\n\\t/go/src/storj.io/storj/storagenode/storagenodedb/satellites.go:198 +0x29c\\nstorj.io/storj/storagenode/gracefulexit.(*Service).ListPendingExits(0x4000205dd0, {0x1385378, 0x40008d20a0})\\n\\t/go/src/storj.io/storj/storagenode/gracefulexit/service.go:59 +0x110\\nstorj.io/storj/storagenode/gracefulexit.(*Chore).AddMissing(0x40000be6c0, {0x1385340, 0x40005742d0})\\n\\t/go/src/storj.io/storj/storagenode/gracefulexit/chore.go:58 +0x158\\nstorj.io/common/sync2.(*Cycle).Run(0x400022edc0, {0x1385378, 0x400065a820}, 0x4001613c28)\\n\\t/go/pkg/mod/storj.io/common@v0.0.0-20231005100446-96ee88859b9d/sync2/cycle.go:160 +0x4e4\\nstorj.io/storj/storagenode/gracefulexit.(*Chore).Run(0x40000be6c0, {0x1385378, 0x400065a820})\\n\\t/go/src/storj.io/storj/storagenode/gracefulexit/chore.go:51 +0x14c\\nstorj.io/storj/private/lifecycle.(*Group).Run.func2.1({0x13850c0?, 0x40003812c0?})\\n\\t/go/src/storj.io/storj/private/lifecycle/group.go:87 +0x34\\nruntime/pprof.Do({0x1385378?, 0x4000776d20?}, {{0x400054c3a0?, 0x2?, 0x5?}}, 0x4000462e60)\\n\\t/usr/local/go/src/runtime/pprof/runtime.go:51 +0x80\\nstorj.io/storj/private/lifecycle.(*Group).Run.func2()\\n\\t/go/src/storj.io/storj/private/lifecycle/group.go:86 +0x1f0\\ngolang.org/x/sync/errgroup.(*Group).Go.func1()\\n\\t/go/pkg/mod/golang.org/x/sync@v0.3.0/errgroup/errgroup.go:75 +0x58\\ncreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 1\\n\\t/go/pkg/mod/golang.org/x/sync@v0.3.0/errgroup/errgroup.go:72 +0x98\\n\\ngoroutine 966 [runnable]:\\ngo.uber.org/zap.(*Logger).check(0x4000374a20, 0x1, {0x1042a4b, 0x1e})\\n\\t/go/pkg/mod/go.uber.org/zap@v1.16.0/logger.go:321 +0x538\\ngo.uber.org/zap.(*Logger).Warn(0x400030eeb0?, {0x1042a4b?, 0xec4040?}, {0x400009b3c0, 0x1, 0x1})\\n\\t/go/pkg/mod/go.uber.org/zap@v1.16.0/logger.go:195 +0x40\\nstorj.io/storj/private/lifecycle.(*Group).Run.func1({0x13850c0?, 0x40002b2030?})\\n\\t/go/src/storj.io/storj/private/lifecycle/group.go:76 +0x208\\nruntime/pprof.Do({0x1385378?, 0x4000776c80?}, {{0x40003373e0?, 0x0?, 0x0?}}, 0x400076ab80)\\n\\t/usr/local/go/src/runtime/pprof/runtime.go:51 +0x80\\ncreated by storj.io/storj/private/lifecycle.(*Group).Run in goroutine 1\\n\\t/go/src/storj.io/storj/private/lifecycle/group.go:64 +0x3d4\\n\\ngoroutine 539085 [select, 13 minutes]:\\nstorj.io/drpc/drpcmanager.(*Manager).manageStreams(0x400038b500)\\n\\t/go/pkg/mod/storj.io/drpc@v0.0.33/drpcmanager/manager.go:313 +0xdc\\ncreated by storj.io/drpc/drpcmanager.NewWithOptions in goroutine 539083\\n\\t/go/pkg/mod/storj.io/drpc@v0.0.33/drpcmanager/manager.go:119 +0x40c\\n\\ngoroutine 579676 [select, 8 minutes]:\\nstorj.io/drpc/drpcmigrate.(*ListenMux).routeConn(0x40000f00e0, {0x1389108?, 0x4002fd4070})\\n\\t/go/pkg/mod/storj.io/drpc@v0.0.33/drpcmigrate/mux.go:164 +0x318\\ncreated by storj.io/drpc/drpcmigrate.(*ListenMux).monitorBase in goroutine 1043\\n\\t/go/pkg/mod/storj.io/drpc@v0.0.33/drpcmigrate/mux.go:124 +0x28\\n\\ngoroutine 561 [sync.Mutex.Lock]:\\nsync.runtime_SemacquireMutex(0x40001fa000?, 0xa0?, 0x40001fa058?)\\n\\t/usr/local/go/src/runtime/sema.go:77 +0x28\\nsync.(*Mutex).lockSlow(0x400009c360)\\n\\t/usr/local/go/src/sync/mutex.go:171 +0x174\\nsync.(*Mutex).Lock(...)\\n\\t/usr/local/go/src/sync/mutex.go:90\\ngithub.com/quic-go/quic-go.(*packetHandlerMap).Get(0x400009c360, {{0x35, 0x6b, 0x7c, 0xe, 0xaf, 0xc, 0x20, 0xa7, 0xb3, ...}, ...})\\n\\t/go/pkg/mod/github.com/quic-go/quic-go@v0.39.0/packet_handler_map.go:112 +0x80\\ngithub.com/quic-go/quic-go.(*Transport).handlePacket(0x400024c1e0, {0x4000786ea0, {0x137d4f8, 0x40008c75c0}, {0xc15388f2c00ab520, 0x3f085f098550, 0x1ae2c40}, {0x40003b4600, 0x4e4, 0x5ac}, ...})\\n\\t/go/pkg/mod/github.com/quic-go/quic-go@v0.39.0/transport.go:386 +0x208\\ngithub.com/quic-go/quic-go.(*Transport).listen(0x400024c1e0, {0x13884a8, 0x40002fc2d0})\\n\\t/go/pkg/mod/github.com/quic-go/quic-go@v0.39.0/transport.go:361 +0x30c\\ncreated by github.com/quic-go/quic-go.(*Transport).init.func1 in goroutine 969\\n\\t/go/pkg/mod/github.com/quic-go/quic-go@v0.39.0/transport.go:242 +0x418\\n\\ngoroutine 580066 [runnable]:\\nstorj.io/drpc/drpcwire.(*Reader).ReadPacketUsing(0x400280c1e0, {0x0?, 0x0?, 0x0?})\\n\\t/go/pkg/mod/storj.io/drpc@v0.0.33/drpcwire/reader.go:91 +0x31c\\nstorj.io/drpc/drpcmanager.(*Manager).manageReader(0x400280a1c0)\\n\\t/go/pkg/mod/storj.io/drpc@v0.0.33/drpcmanager/manager.go:226 +0xd0\\ncreated by storj.io/drpc/drpcmanager.NewWithOptions in goroutine 579956\\n\\t/go/pkg/mod/storj.io/drpc@v0.0.33/drpcmanager/manager.go:118 +0x3cc\\n\\ngoroutine 580150 [select, 8 minutes]:\\nstorj.io/drpc/drpcmigrate.(*ListenMux).routeConn(0x40000f00e0, {0x1389108?, 0x40003fcf00})\\n\\t/go/pkg/mod/storj.io/drpc@v0.0.33/drpcmigrate/mux.go:164 +0x318\\ncreated by storj.io/drpc/drpcmigrate.(*ListenMux).monitorBase in goroutine 1043\\n\\t/go/pkg/mod/storj.io/drpc@v0.0.33/drpcmigrate/mux.go:124 +0x28\\n\\ngoroutine 579979 [chan receive]:\\nstorj.io/drpc/drpcsignal.(*Signal).Wait(...)\\n\\t/go/pkg/mod/storj.io/drpc@v0.0.33/drpcsignal/signal.go:29\\nstorj.io/drpc/drpcmanager.(*Manager).Close(0x400114ee00)\\n\\t/go/pkg/mod/storj.io/drpc@v0.0.33/drpcmanager/manager.go:401 +0xbc\\nstorj.io/drpc/drpcserver.(*Server).ServeOne.func1()\\n\\t/go/pkg/mod/storj.io/drpc@v0.0.33/drpcserver/server.go:54 +0x28\\nstorj.io/drpc/drpcserver.(*Server).ServeOne(0x4000370480, {0x1385730, 0x40002b3680}, {0x7f4dbdcb30?, 0x40024bb880?})\\n\\t/go/pkg/mod/storj.io/drpc@v0.0.33/drpcserver/server.go:64 +0x214\\nstorj.io/drpc/drpcserver.(*Server).Serve.func2({0x1385730, 0x40002b3680})\\n\\t/go/pkg/mod/storj.io/drpc@v0.0.33/drpcserver/server.go:114 +0x5c\\nstorj.io/drpc/drpcctx.(*Tracker).track(0x40002b3680, 0x0?)\\n\\t/go/pkg/mod/storj.io/drpc@v0.0.33/drpcctx/tracker.go:35 +0x3c\\ncreated by storj.io/drpc/drpcctx.(*Tracker).Run in goroutine 1020\\n\\t/go/pkg/mod/storj.io/drpc@v0.0.33/drpcctx/tracker.go:30 +0x84\\n\\ngoroutine 1022 [runnable]:\\ngithub.com/flynn/noise.NewHandshakeState({{0x1389d40, 0x400037a5a0}, {0x0, 0x0}, {{0x1024bc3, 0x2}, {0x0, 0x0, 0x0}, {0x1b1a1a0, ...}, ...}, ...})\\n\\t/go/pkg/mod/github.com/flynn/noise@v1.0.0/state.go:324 +0x568\\ngithub.com/jtolio/noiseconn.NewConnWithOptions({_, _}, {{0x1389d40, 0x400037a5a0}, {0x0, 0x0}, {{0x1024bc3, 0x2}, {0x0, 0x0, ...}, ...}, ...}, ...)\\n\\t/go/pkg/mod/github.com/jtolio/noiseconn@v0.0.0-20230301220541-88105e6c8ac6/conn.go:64 +0x58\\ngithub.com/jtolio/noiseconn.(*Listener).Accept(0x40001e29a0)\\n\\t/go/pkg/mod/github.com/jtolio/noiseconn@v0.0.0-20230301220541-88105e6c8ac6/listener.go:26 +0xa4\\nstorj.io/drpc/drpcserver.(*Server).Serve(0x4000370400, {0x13850f8, 0x40002840a0}, {0x1383eb0?, 0x40001e29a0})\\n\\t/go/pkg/mod/storj.io/drpc@v0.0.33/drpcserver/server.go:87 +0x158\\nstorj.io/storj/private/server.(*Server).Run.(*Server).Run.func4.func10()\\n\\t/go/src/storj.io/storj/private/server/server.go:383 +0x50\\ngolang.org/x/sync/errgroup.(*Group).Go.func1()\\n\\t/go/pkg/mod/golang.org/x/sync@v0.3.0/errgroup/errgroup.go:75 +0x58\\ncreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 969\\n\\t/go/pkg/mod/golang.org/x/sync@v0.3.0/errgroup/errgroup.go:72 +0x98\\n\\ngoroutine 335 [select, 1158 minutes]:\\ndatabase/sql.(*DB).connectionOpener(0x400050b860, {0x13850f8, 0x4000239db0})\\n\\t/usr/local/go/src/database/sql/sql.go:1218 +0x80\\ncreated by database/sql.OpenDB in goroutine 1\\n\\t/usr/local/go/src/database/sql/sql.go:791 +0x158\\n\\ngoroutine 579747 [select, 8 minutes]:\\nstorj.io/drpc/drpcmigrate.(*ListenMux).routeConn(0x40000f00e0, {0x1389108?, 0x400033b670})\\n\\t/go/pkg/mod/storj.io/drpc@v0.0.33/drpcmigrate/mux.go:164 +0x318\\ncreated by storj.io/drpc/drpcmigrate.(*ListenMux).monitorBase in goroutine 1043\\n\\t/go/pkg/mod/storj.io/drpc@v0.0.33/drpcmigrate/mux.go:124 +0x28\\n\\ngoroutine 579357 [chan receive]:\\nstorj.io/drpc/drpcsignal.(*Signal).Wait(...)\\n\\t/go/pkg/mod/storj.io/drpc@v0.0.33/drpcsignal/signal.go:29\\nstorj.io/drpc/drpcmanager.(*Manager).Close(0x4000cb4700)\\n\\t/go/pkg/mod/storj.io/drpc@v0.0.33/drpcmanager/manager.go:401 +0xbc\\nstorj.io/drpc/drpcserver.(*Server).ServeOne.func1()\\n\\t/go/pkg/mod/storj.io/drpc@v0.0.33/drpcserver/server.go:54 +0x28\\nstorj.io/drpc/drpcserver.(*Server).ServeOne(0x4000370400, {0x1385730, 0x40002b3740}, {0x137f740?, 0x40004a0e40?})\\n\\t/go/pkg/mod/storj.io/drpc@v0.0.33/drpcserver/server.go:64 +0x214\\nstorj.io/drpc/drpcserver.(*Server).Serve.func2({0x1385730, 0x40002b3740})\\n\\t/go/pkg/mod/storj.io/drpc@v0.0.33/drpcserver/server.go:114 +0x5c\\nstorj.io/drpc/drpcctx.(*Tracker).track(0x40002b3740, 0x0?)\\n\\t/go/pkg/mod/storj.io/drpc@v0.0.33/drpcctx/tracker.go:35 +0x3c\\ncreated by storj.io/drpc/","stream":"stdout","time":"2023-12-04T10:54:17.633115593Z"}
This is one of 88 lines
this is a result of the node to be shutdown because of some FATAL error.
Do you have a FATAL errors by the way?
[~] #
[~] # docker logs storagenode2 2>&1 | grep -i fatal
[~] # docker logs storagenode 2>&1 | grep -i fatal
2023-12-05T01:28:53Z INFO piecestore upload started {"process": "storagenode", "Piece ID": "VZK7HKFATALBHKHNPJ44BZNT4AED74MVCLAXZSFFIGFMBK3CW76A", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT", "Available Space": 270291055625, "Remote Address": "79.127.203.194:55376"}
2023-12-05T02:34:12Z ERROR piecestore upload failed {"process": "storagenode", "Piece ID": "VZK7HKFATALBHKHNPJ44BZNT4AED74MVCLAXZSFFIGFMBK3CW76A", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT", "error": "order: grace period passed for order limit", "errorVerbose": "order: grace period passed for order limit\n\tstorj.io/storj/storagenode/orders.(*FileStore).BeginEnqueue:86\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).beginSaveOrder:883\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:407\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:243\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:61\n\tstorj.io/common/experiment.(*Handler).HandleRPC:42\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:124\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:114\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35", "Size": 0, "Remote Address": "79.127.203.194:55376"}
[~] #
The nas gave ram errors, closed applications. Happens since 1.92.1
I can’t increase the ram and the active services are the basic ones
Now they’re fine…
Looks like you have a time sync issues. Can you sync your local time on your NAS?
You may limit the RAM usage for the container with the option --memory, the value you should figure out yourself. How much free RAM do you have?
Perhaps you need to click the "Aggiorna" button to make it synced right away.
Done. 20 characters 20 characters
Did it help to do not have this error anymore?
Now they are working, but the error reappears after a few hours