Two nodes crashing with goroutine logs

agente · April 29, 2024, 8:17am

I have two nodes that are crashing with a lot of goroutine message. IO wait? Db to be fixed?

goroutine 181969 [IO wait]:
runtime.gopark(0x276756108319b9e9?, 0xb?, 0x0?, 0x0?, 0x76f?)
        /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc08788f8e0 sp=0xc08788f8c0 pc=0x43f10e
runtime.netpollblock(0x487758?, 0x4080e6?, 0x0?)
        /usr/local/go/src/runtime/netpoll.go:564 +0xf7 fp=0xc08788f918 sp=0xc08788f8e0 pc=0x437897
internal/poll.runtime_pollWait(0x7f8b5cd81b20, 0x72)
        /usr/local/go/src/runtime/netpoll.go:343 +0x85 fp=0xc08788f938 sp=0xc08788f918 pc=0x46ca05
internal/poll.(*pollDesc).wait(0xc0b6fae300?, 0xc12da908b4?, 0x0)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x27 fp=0xc08788f960 sp=0xc08788f938 pc=0x4add87
internal/poll.(*pollDesc).waitRead(...)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc0b6fae300, {0xc12da908b4, 0x4e3a, 0x574c})
        /usr/local/go/src/internal/poll/fd_unix.go:164 +0x27a fp=0xc08788f9f8 sp=0xc08788f960 pc=0x4af07a
net.(*netFD).Read(0xc0b6fae300, {0xc12da908b4?, 0x41187d?, 0x9?})
        /usr/local/go/src/net/fd_posix.go:55 +0x25 fp=0xc08788fa40 sp=0xc08788f9f8 pc=0x55fde5
net.(*conn).Read(0xc0046ff728, {0xc12da908b4?, 0xc0a034ec00?, 0xc12da86000?})
        /usr/local/go/src/net/net.go:179 +0x45 fp=0xc08788fa88 sp=0xc08788fa40 pc=0x56f2e5
net.(*TCPConn).Read(0x40e6fa?, {0xc12da908b4?, 0x0?, 0x7f8bdf865c00?})
        <autogenerated>:1 +0x25 fp=0xc08788fab8 sp=0xc08788fa88 pc=0x581365
storj.io/common/netutil.(*closeTrackingConn).Read(0x10000?, {0xc12da908b4?, 0xc08788fb20?, 0x40f5ac?})
        <autogenerated>:1 +0x29 fp=0xc08788fae8 sp=0xc08788fab8 pc=0xac09c9
io.ReadAtLeast({0x7f8b981e8fb0, 0xc03a63cd20}, {0xc12da86000, 0xf6ee, 0x10000}, 0xf6ee)
        /usr/local/go/src/io/io.go:335 +0x90 fp=0xc08788fb30 sp=0xc08788fae8 pc=0x4a66d0
io.ReadFull(...)
        /usr/local/go/src/io/io.go:354
github.com/jtolio/noiseconn.(*Conn).readMsg(0xc02de45980, {0x0, 0x0, 0x0})
        /go/pkg/mod/github.com/jtolio/noiseconn@v0.0.0-20230301220541-88105e6c8ac6/conn.go:220 +0x157 fp=0xc08788fbb8 sp=0xc08788fb30 pc=0xaddc57
github.com/jtolio/noiseconn.(*Conn).hsRead(0xc02de45980)
        /go/pkg/mod/github.com/jtolio/noiseconn@v0.0.0-20230301220541-88105e6c8ac6/conn.go:96 +0x26 fp=0xc08788fc00 sp=0xc08788fbb8 pc=0xadcda6
github.com/jtolio/noiseconn.(*Conn).Read(0xc02de45980, {0xc0ba63f000, 0x1000, 0x1000})
        /go/pkg/mod/github.com/jtolio/noiseconn@v0.0.0-20230301220541-88105e6c8ac6/conn.go:160 +0x3d9 fp=0xc08788fcb8 sp=0xc08788fc00 pc=0xadd319
storj.io/drpc/drpcwire.(*Reader).read(0xc0d92f8e00, {0xc0ba63f000, 0x1000, 0x1000})
        /go/pkg/mod/storj.io/drpc@v0.0.34/drpcwire/reader.go:68 +0x74 fp=0xc08788fcf0 sp=0xc08788fcb8 pc=0xa86cd4
storj.io/drpc/drpcwire.(*Reader).ReadPacketUsing(0xc0d92f8e00, {0x0?, 0x2?, 0x0?})
        /go/pkg/mod/storj.io/drpc@v0.0.34/drpcwire/reader.go:113 +0x4e5 fp=0xc08788fe70 sp=0xc08788fcf0 pc=0xa87345
storj.io/drpc/drpcmanager.(*Manager).manageReader(0xc0eeef3c20)
        /go/pkg/mod/storj.io/drpc@v0.0.34/drpcmanager/manager.go:229 +0xfb fp=0xc08788ffc8 sp=0xc08788fe70 pc=0xaed11b
storj.io/drpc/drpcmanager.NewWithOptions.func1()
        /go/pkg/mod/storj.io/drpc@v0.0.34/drpcmanager/manager.go:121 +0x25 fp=0xc08788ffe0 sp=0xc08788ffc8 pc=0xaec9e5
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc08788ffe8 sp=0xc08788ffe0 pc=0x4724e1
created by storj.io/drpc/drpcmanager.NewWithOptions in goroutine 181968
        /go/pkg/mod/storj.io/drpc@v0.0.34/drpcmanager/manager.go:121 +0x416

goroutine 182018 [select]:
runtime.gopark(0xc04f585f90?, 0x2?, 0x0?, 0x0?, 0xc04f585f5c?)
        /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc04f585e00 sp=0xc04f585de0 pc=0x43f10e
runtime.selectgo(0xc04f585f90, 0xc04f585f58, 0x0?, 0x0, 0x0?, 0x1)
        /usr/local/go/src/runtime/select.go:327 +0x725 fp=0xc04f585f20 sp=0xc04f585e00 pc=0x44f625
storj.io/drpc/drpcmanager.(*Manager).manageStreams(0xc0eeef3c20)
        /go/pkg/mod/storj.io/drpc@v0.0.34/drpcmanager/manager.go:319 +0x11d fp=0xc04f585fc8 sp=0xc04f585f20 pc=0xaeda3d
storj.io/drpc/drpcmanager.NewWithOptions.func2()
        /go/pkg/mod/storj.io/drpc@v0.0.34/drpcmanager/manager.go:122 +0x25 fp=0xc04f585fe0 sp=0xc04f585fc8 pc=0xaec985
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc04f585fe8 sp=0xc04f585fe0 pc=0x4724e1
created by storj.io/drpc/drpcmanager.NewWithOptions in goroutine 181968
        /go/pkg/mod/storj.io/drpc@v0.0.34/drpcmanager/manager.go:122 +0x456

goroutine 182020 [select]:
runtime.gopark(0xc0051ebd28?, 0x4?, 0x0?, 0x0?, 0xc0051ebc20?)
        /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc0051ebaa0 sp=0xc0051eba80 pc=0x43f10e
runtime.selectgo(0xc0051ebd28, 0xc0051ebc18, 0xc070de4600?, 0x0, 0xc0051ebc38?, 0x1)
        /usr/local/go/src/runtime/select.go:327 +0x725 fp=0xc0051ebbc0 sp=0xc0051ebaa0 pc=0x44f625
storj.io/drpc/drpcmanager.(*Manager).NewServerStream(0xc0eeef3e00, {0x1560100, 0xc070de4600})
        /go/pkg/mod/storj.io/drpc@v0.0.34/drpcmanager/manager.go:446 +0x28e fp=0xc0051ebde8 sp=0xc0051ebbc0 pc=0xaee4ee
storj.io/drpc/drpcserver.(*Server).ServeOne(0xc00043c0a0, {0x1560818, 0xc000e3b3b0}, {0x155a120?, 0xc02de45c80?})
        /go/pkg/mod/storj.io/drpc@v0.0.34/drpcserver/server.go:105 +0x1c5 fp=0xc0051ebf50 sp=0xc0051ebde8 pc=0xba4725
storj.io/drpc/drpcserver.(*Server).Serve.func2({0x1560818, 0xc000e3b3b0})
        /go/pkg/mod/storj.io/drpc@v0.0.34/drpcserver/server.go:157 +0x59 fp=0xc0051ebf98 sp=0xc0051ebf50 pc=0xba4e99
storj.io/drpc/drpcctx.(*Tracker).track(0xc000e3b3b0, 0xc06d3cc000?)
        /go/pkg/mod/storj.io/drpc@v0.0.34/drpcctx/tracker.go:35 +0x2e fp=0xc0051ebfc0 sp=0xc0051ebf98 pc=0xa73ece
storj.io/drpc/drpcctx.(*Tracker).Run.func1()
        /go/pkg/mod/storj.io/drpc@v0.0.34/drpcctx/tracker.go:30 +0x25 fp=0xc0051ebfe0 sp=0xc0051ebfc0 pc=0xa73e65
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0051ebfe8 sp=0xc0051ebfe0 pc=0x4724e1
created by storj.io/drpc/drpcctx.(*Tracker).Run in goroutine 1137
        /go/pkg/mod/storj.io/drpc@v0.0.34/drpcctx/tracker.go:30 +0x79

goroutine 182021 [IO wait]:
runtime.gopark(0x20?, 0xb?, 0x0?, 0x0?, 0x776?)
        /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc003f6c8e0 sp=0xc003f6c8c0 pc=0x43f10e
runtime.netpollblock(0x487758?, 0x4080e6?, 0x0?)
        /usr/local/go/src/runtime/netpoll.go:564 +0xf7 fp=0xc003f6c918 sp=0xc003f6c8e0 pc=0x437897
internal/poll.runtime_pollWait(0x7f8b977e5280, 0x72)
        /usr/local/go/src/runtime/netpoll.go:343 +0x85 fp=0xc003f6c938 sp=0xc003f6c918 pc=0x46ca05
internal/poll.(*pollDesc).wait(0xc0b6fae380?, 0xc12daa9db8?, 0x0)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x27 fp=0xc003f6c960 sp=0xc003f6c938 pc=0x4add87
internal/poll.(*pollDesc).waitRead(...)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc0b6fae380, {0xc12daa9db8, 0xb935, 0xc248})
        /usr/local/go/src/internal/poll/fd_unix.go:164 +0x27a fp=0xc003f6c9f8 sp=0xc003f6c960 pc=0x4af07a
net.(*netFD).Read(0xc0b6fae380, {0xc12daa9db8?, 0x41187d?, 0xc0a82af800?})
        /usr/local/go/src/net/fd_posix.go:55 +0x25 fp=0xc003f6ca40 sp=0xc003f6c9f8 pc=0x55fde5
net.(*conn).Read(0xc0046ff738, {0xc12daa9db8?, 0xc0b727a400?, 0xc12daa6000?})
        /usr/local/go/src/net/net.go:179 +0x45 fp=0xc003f6ca88 sp=0xc003f6ca40 pc=0x56f2e5
net.(*TCPConn).Read(0x40e6fa?, {0xc12daa9db8?, 0x0?, 0x7f8bdf865300?})
        <autogenerated>:1 +0x25 fp=0xc003f6cab8 sp=0xc003f6ca88 pc=0x581365
storj.io/common/netutil.(*closeTrackingConn).Read(0x10000?, {0xc12daa9db8?, 0xc003f6cb20?, 0x40f5ac?})
        <autogenerated>:1 +0x29 fp=0xc003f6cae8 sp=0xc003f6cab8 pc=0xac09c9
io.ReadAtLeast({0x7f8b981e8fb0, 0xc03a63cd70}, {0xc12daa6000, 0xf6ed, 0x10000}, 0xf6ed)
        /usr/local/go/src/io/io.go:335 +0x90 fp=0xc003f6cb30 sp=0xc003f6cae8 pc=0x4a66d0
io.ReadFull(...)
        /usr/local/go/src/io/io.go:354
github.com/jtolio/noiseconn.(*Conn).readMsg(0xc02de45c80, {0x0, 0x0, 0x0})
        /go/pkg/mod/github.com/jtolio/noiseconn@v0.0.0-20230301220541-88105e6c8ac6/conn.go:220 +0x157 fp=0xc003f6cbb8 sp=0xc003f6cb30 pc=0xaddc57
github.com/jtolio/noiseconn.(*Conn).hsRead(0xc02de45c80)
        /go/pkg/mod/github.com/jtolio/noiseconn@v0.0.0-20230301220541-88105e6c8ac6/conn.go:96 +0x26 fp=0xc003f6cc00 sp=0xc003f6cbb8 pc=0xadcda6
github.com/jtolio/noiseconn.(*Conn).Read(0xc02de45c80, {0xc0ba407000, 0x1000, 0x1000})
        /go/pkg/mod/github.com/jtolio/noiseconn@v0.0.0-20230301220541-88105e6c8ac6/conn.go:160 +0x3d9 fp=0xc003f6ccb8 sp=0xc003f6cc00 pc=0xadd319
storj.io/drpc/drpcwire.(*Reader).read(0xc0d92f8e70, {0xc0ba407000, 0x1000, 0x1000})
        /go/pkg/mod/storj.io/drpc@v0.0.34/drpcwire/reader.go:68 +0x74 fp=0xc003f6ccf0 sp=0xc003f6ccb8 pc=0xa86cd4
storj.io/drpc/drpcwire.(*Reader).ReadPacketUsing(0xc0d92f8e70, {0x0?, 0xc000be0c00?, 0x4a66d0?})
        /go/pkg/mod/storj.io/drpc@v0.0.34/drpcwire/reader.go:113 +0x4e5 fp=0xc003f6ce70 sp=0xc003f6ccf0 pc=0xa87345
storj.io/drpc/drpcmanager.(*Manager).manageReader(0xc0eeef3e00)
        /go/pkg/mod/storj.io/drpc@v0.0.34/drpcmanager/manager.go:229 +0xfb fp=0xc003f6cfc8 sp=0xc003f6ce70 pc=0xaed11b
storj.io/drpc/drpcmanager.NewWithOptions.func1()
        /go/pkg/mod/storj.io/drpc@v0.0.34/drpcmanager/manager.go:121 +0x25 fp=0xc003f6cfe0 sp=0xc003f6cfc8 pc=0xaec9e5
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc003f6cfe8 sp=0xc003f6cfe0 pc=0x4724e1
created by storj.io/drpc/drpcmanager.NewWithOptions in goroutine 182020
        /go/pkg/mod/storj.io/drpc@v0.0.34/drpcmanager/manager.go:121 +0x416

goroutine 182022 [select]:
runtime.gopark(0xc0019af790?, 0x2?, 0x0?, 0x0?, 0xc0019af75c?)
        /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc0019af600 sp=0xc0019af5e0 pc=0x43f10e
runtime.selectgo(0xc0019af790, 0xc0019af758, 0x0?, 0x0, 0x0?, 0x1)
        /usr/local/go/src/runtime/select.go:327 +0x725 fp=0xc0019af720 sp=0xc0019af600 pc=0x44f625
storj.io/drpc/drpcmanager.(*Manager).manageStreams(0xc0eeef3e00)
        /go/pkg/mod/storj.io/drpc@v0.0.34/drpcmanager/manager.go:319 +0x11d fp=0xc0019af7c8 sp=0xc0019af720 pc=0xaeda3d
storj.io/drpc/drpcmanager.NewWithOptions.func2()
        /go/pkg/mod/storj.io/drpc@v0.0.34/drpcmanager/manager.go:122 +0x25 fp=0xc0019af7e0 sp=0xc0019af7c8 pc=0xaec985
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0019af7e8 sp=0xc0019af7e0 pc=0x4724e1
created by storj.io/drpc/drpcmanager.NewWithOptions in goroutine 182020
        /go/pkg/mod/storj.io/drpc@v0.0.34/drpcmanager/manager.go:122 +0x456
2024-04-29T08:00:01Z    INFO    Downloading versions.   {"Process": "storagenode-updater", "Server Address": "https://version.storj.io"}
2024-04-29T08:00:01Z    INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode", "Version": "v1.101.3"}
2024-04-29T08:00:01Z    INFO    New version is being rolled out but hasn't made it to this node yet     {"Process": "storagenode-updater", "Service": "storagenode"}
2024-04-29T08:00:01Z    INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode-updater", "Version": "v1.101.3"}
2024-04-29T08:00:01Z    INFO    New version is being rolled out but hasn't made it to this node yet     {"Process": "storagenode-updater", "Service": "storagenode-updater"}

Alexey · April 29, 2024, 9:01am

I would recommend to stop and remove the container, delete the image and run the node back.

docker stop -t 300 storagenode
docker rm storagenode
docker rmi storjlabs/storagenode:latest

then run the container back with all your parameters.

kocoten1992 · April 29, 2024, 9:20am

Is this the first time it happened or it happen frequently?
Was two nodes running on the same machine?
What was your operating system (specific to which distro and kernel version - if you compile your kernel - which gcc version) - see Debugging an evil Go runtime bug - marcan.st?
DB usually don’t need to be fix (if this is not a hardware failure).

agente · April 29, 2024, 10:06am

First time. Same server. More or less same tb stored. Ubuntu 22 root@rm02:~# uname -a
Linux rm02 5.15.0-97-generic #107-Ubuntu SMP Wed Feb 7 13:26:48 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

I just restarted nodes… if happen again I will rm and rmi