Node Crash v1.3.3

[qinhang3]

I found my node was crashed.It run on v1.3.3 since May.
I have tried “docker restart” but didn’t work.
Then I try “docker pull storjlabs/storagenode:beta”. (It’s success.and pull the v1.5.2 image)
Node goes normal after update the image.

Here is my log…

Summary

runtime: pointer 0xc0009b9790 to unallocated span span.base()=0xc0009b2000 span.limit=0xc0009ba000 span.state=3
fatal error: found bad pointer in Go heap (incorrect use of unsafe or cgo?)

runtime stack:
runtime.throw(0x1166866, 0x3e)
/usr/local/go/src/runtime/panic.go:774 +0x72 fp=0x7f0f26c34d78 sp=0x7f0f26c34d48 pc=0x430a82
runtime.findObject(0xc0009b9790, 0x0, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/runtime/mbitmap.go:397 +0x3b4 fp=0x7f0f26c34dc8 sp=0x7f0f26c34d78 pc=0x415fd4
runtime.wbBufFlush1(0xc000042500)
/usr/local/go/src/runtime/mwbbuf.go:288 +0xa5 fp=0x7f0f26c34e20 sp=0x7f0f26c34dc8 pc=0x42c5c5
runtime.wbBufFlush.func1()
/usr/local/go/src/runtime/mwbbuf.go:218 +0x3a fp=0x7f0f26c34e38 sp=0x7f0f26c34e20 pc=0x45ce0a
runtime.systemstack(0x1c12a68)
/usr/local/go/src/runtime/asm_amd64.s:370 +0x66 fp=0x7f0f26c34e40 sp=0x7f0f26c34e38 pc=0x45ec16
runtime.mstart()
/usr/local/go/src/runtime/proc.go:1146 fp=0x7f0f26c34e48 sp=0x7f0f26c34e40 pc=0x435070

goroutine 3126440 [running]:
runtime.systemstack_switch()
/usr/local/go/src/runtime/asm_amd64.s:330 fp=0xc0008bf300 sp=0xc0008bf2f8 pc=0x45eba0
runtime.wbBufFlush(0x0, 0x0)
/usr/local/go/src/runtime/mwbbuf.go:206 +0x66 fp=0xc0008bf330 sp=0xc0008bf300 pc=0x42c496
runtime.bulkBarrierPreWrite(0xc000186d00, 0xc0008bf990, 0x40040)
/usr/local/go/src/runtime/mbitmap.go:640 +0x2eb fp=0xc0008bf3b8 sp=0xc0008bf330 pc=0x41650b
runtime.typedmemmove(0x1098ee0, 0xc000186d00, 0xc0008bf990)
/usr/local/go/src/runtime/mbarrier.go:161 +0xa1 fp=0xc0008bf3f0 sp=0xc0008bf3b8 pc=0x415321
storj.io/storj/storagenode/piecestore.(*Endpoint).doUpload(0xc0000d2160, 0x7f0f2655b1b0, 0xc0002b1df0, 0x0, 0xc0002b1df0, 0x414201)
/go/src/storj.io/storj/storagenode/piecestore/endpoint.go:324 +0x21f8 fp=0xc0008bfae0 sp=0xc0008bf3f0 pc=0xcc3608
storj.io/storj/storagenode/piecestore.(*drpcEndpoint).Upload(0xc000152210, 0x12e3400, 0xc0002b1df0, 0x12dd280, 0xc00089e380)
/go/src/storj.io/storj/storagenode/piecestore/endpoint.go:216 +0x85 fp=0xc0008bfb28 sp=0xc0008bfae0 pc=0xcc13e5
storj.io/common/pb.DRPCPiecestoreDescription.Method.func1(0x10da820, 0xc000152210, 0x12d9cc0, 0xc00058ef40, 0x101c7a0, 0xc00089e380, 0x101c7a0, 0xc00089e380, 0x40d726, 0xc00089e380, …)
/go/pkg/mod/storj.io/common@v0.0.0-20200419143955-e54806ee8c56/pb/piecestore2.pb.go:987 +0xe8 fp=0xc0008bfb80 sp=0xc0008bfb28 pc=0x903288
storj.io/drpc/drpcmux.(*Mux).HandleRPC(0xc000152128, 0x12dd280, 0xc00089e380, 0xc000619ba0, 0x1d, 0xc000597cf0, 0x12bb5a0)
/go/pkg/mod/storj.io/drpc@v0.0.12-0.20200413163255-debb458a7474/drpcmux/mux.go:107 +0x2c3 fp=0xc0008bfc98 sp=0xc0008bfb80 pc=0x956623
storj.io/common/rpc/rpctracing.(*Handler).HandleRPC(0xc000337220, 0x12dd2c0, 0xc0005c6420, 0xc000619ba0, 0x1d, 0x0, 0x0)
/go/pkg/mod/storj.io/common@v0.0.0-20200419143955-e54806ee8c56/rpc/rpctracing/handler.go:66 +0x14a fp=0xc0008bfd68 sp=0xc0008bfc98 pc=0x956b1a
storj.io/drpc/drpcserver.(*Server).handleRPC(0xc00044e560, 0xc0005c6420, 0xc000619ba0, 0x1d, 0xc000619ba0, 0x1d)
/go/pkg/mod/storj.io/drpc@v0.0.12-0.20200413163255-debb458a7474/drpcserver/server.go:111 +0x25d fp=0xc0008bfe38 sp=0xc0008bfd68 pc=0xc7b72d
storj.io/drpc/drpcserver.(*Server).ServeOne(0xc00044e560, 0x12da400, 0xc0001c48a0, 0x7f0f26814048, 0xc000013c00, 0x438ba6, 0x1181df8)
/go/pkg/mod/storj.io/drpc@v0.0.12-0.20200413163255-debb458a7474/drpcserver/server.go:62 +0x2ec fp=0xc0008bff60 sp=0xc0008bfe38 pc=0xc7ad7c
storj.io/drpc/drpcserver.(*Server).Serve.func2(0x12da400, 0xc0001c48a0)
/go/pkg/mod/storj.io/drpc@v0.0.12-0.20200413163255-debb458a7474/drpcserver/server.go:99 +0x6a fp=0xc0008bffb0 sp=0xc0008bff60 pc=0xc7bb1a
storj.io/drpc/drpcctx.(*Tracker).track(0xc0001c48a0, 0xc000b44040)
/go/pkg/mod/storj.io/drpc@v0.0.12-0.20200413163255-debb458a7474/drpcctx/transport.go:51 +0x3c fp=0xc0008bffd0 sp=0xc0008bffb0 pc=0x91df7c
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc0008bffd8 sp=0xc0008bffd0 pc=0x460c71
created by storj.io/drpc/drpcctx.(*Tracker).Run
/go/pkg/mod/storj.io/drpc@v0.0.12-0.20200413163255-debb458a7474/drpcctx/transport.go:46 +0x66

goroutine 1 [semacquire, 366 minutes]:
runtime.gopark(0x1181f08, 0x1bff280, 0xc000131912, 0x4)
/usr/local/go/src/runtime/proc.go:304 +0xe0 fp=0xc0007264a0 sp=0xc000726480 pc=0x432820
runtime.goparkunlock(…)
/usr/local/go/src/runtime/proc.go:310
runtime.semacquire1(0xc0002ec5e0, 0x0, 0x1, 0x0)
/usr/local/go/src/runtime/sema.go:144 +0x1c0 fp=0xc000726508 sp=0xc0007264a0 pc=0x442e90
sync.runtime_Semacquire(0xc0002ec5e0)
/usr/local/go/src/runtime/sema.go:56 +0x42 fp=0xc000726538 sp=0xc000726508 pc=0x442ae2
sync.(*WaitGroup).Wait(0xc0002ec5d8)
/usr/local/go/src/sync/waitgroup.go:130 +0x64 fp=0xc000726560 sp=0xc000726538 pc=0x471374
golang.org/x/sync/errgroup.(*Group).Wait(0xc0002ec5d0, 0x12d9cc0, 0xc0001742c0)
/go/pkg/mod/golang.org/x/sync@v0.0.0-20190911185100-cd5d95a43a6e/errgroup/errgroup.go:40 +0x31 fp=0xc000726578 sp=0xc000726560 pc=0x9c4191
storj.io/storj/storagenode.(*Peer).Run(0xc00017cd80, 0x12d9cc0, 0xc0000998c0, 0x0, 0x0)
/go/src/storj.io/storj/storagenode/peer.go:678 +0x57a fp=0xc000726700 sp=0xc000726578 pc=0xd3d2da
main.cmdRun(0x1a2da80, 0xc0003558c0, 0x0, 0xb, 0x0, 0x0)
/go/src/storj.io/storj/cmd/storagenode/main.go:209 +0xf77 fp=0xc000726f40 sp=0xc000726700 pc=0xdd4f37
storj.io/private/process.cleanup.func1.4(0x12d9fc0, 0xc000034dc0)
/go/pkg/mod/storj.io/private@v0.0.0-20200415180736-fe6c6bb80e76/process/exec_conf.go:344 +0x13b fp=0xc000726fd8 sp=0xc000726f40 pc=0xbe7bab
storj.io/private/process.cleanup.func1(0x1a2da80, 0xc0003558c0, 0x0, 0xb, 0x0, 0x0)
/go/pkg/mod/storj.io/private@v0.0.0-20200415180736-fe6c6bb80e76/process/exec_conf.go:362 +0x1df3 fp=0xc000727d38 sp=0xc000726fd8 pc=0xbe9a53
github.com/spf13/cobra.(*Command).execute(0x1a2da80, 0xc000355810, 0xb, 0xb, 0x1a2da80, 0xc000355810)
/go/pkg/mod/github.com/spf13/cobra@v0.0.6/command.go:840 +0x460 fp=0xc000727e10 sp=0xc000727d38 pc=0x6367b0
github.com/spf13/cobra.(*Command).ExecuteC(0x1a2d7e0, 0xc00009a100, 0x1182360, 0x1)
/go/pkg/mod/github.com/spf13/cobra@v0.0.6/command.go:945 +0x317 fp=0xc000727ee8 sp=0xc000727e10 pc=0x6372a7
github.com/spf13/cobra.(*Command).Execute(…)
/go/pkg/mod/github.com/spf13/cobra@v0.0.6/command.go:885
storj.io/private/process.ExecWithCustomConfig(0x1a2d7e0, 0x0, 0x1182360)
/go/pkg/mod/storj.io/private@v0.0.0-20200415180736-fe6c6bb80e76/process/exec_conf.go:86 +0x192 fp=0xc000727f38 sp=0xc000727ee8 pc=0xbe34d2
storj.io/private/process.ExecCustomDebug(...)
/go/pkg/mod/storj.io/private@v0.0.0-20200415180736-fe6c6bb80e76/process/exec_conf.go:68
main.main()
/go/src/storj.io/storj/cmd/storagenode/main.go:329 +0x3e fp=0xc000727f60 sp=0xc000727f38 pc=0xdd63ce
runtime.main()
/usr/local/go/src/runtime/proc.go:203 +0x21e fp=0xc000727fe0 sp=0xc000727f60 pc=0x43241e
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc000727fe8 sp=0xc000727fe0 pc=0x460c71

goroutine 2 [force gc (idle), 366 minutes]:
runtime.gopark(0x1181f08, 0x1bf13b0, 0x1411, 0x1)
/usr/local/go/src/runtime/proc.go:304 +0xe0 fp=0xc00006efb0 sp=0xc00006ef90 pc=0x432820
runtime.goparkunlock(…)
/usr/local/go/src/runtime/proc.go:310
runtime.forcegchelper()
/usr/local/go/src/runtime/proc.go:253 +0xb7 fp=0xc00006efe0 sp=0xc00006efb0 pc=0x4326a7
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc00006efe8 sp=0xc00006efe0 pc=0x460c71
created by runtime.init.5
/usr/local/go/src/runtime/proc.go:242 +0x35

goroutine 3 [GC sweep wait]:
runtime.gopark(0x1181f08, 0x1bf2100, 0x140c, 0x1)
/usr/local/go/src/runtime/proc.go:304 +0xe0 fp=0xc00006f7a8 sp=0xc00006f788 pc=0x432820
runtime.goparkunlock(…)
/usr/local/go/src/runtime/proc.go:310
runtime.bgsweep(0xc000096000)
/usr/local/go/src/runtime/mgcsweep.go:89 +0x131 fp=0xc00006f7d8 sp=0xc00006f7a8 pc=0x423ee1
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc00006f7e0 sp=0xc00006f7d8 pc=0x460c71
created by runtime.gcenable
/usr/local/go/src/runtime/mgc.go:210 +0x5c

goroutine 4 [runnable]:
runtime.gopark(0x1181f08, 0x1bf3f20, 0x1313, 0x2)
/usr/local/go/src/runtime/proc.go:304 +0xe0 fp=0xc00006fef0 sp=0xc00006fed0 pc=0x432820
runtime.goparkunlock(…)
/usr/local/go/src/runtime/proc.go:310
runtime.scavengeSleep(0x1cc25d, 0x1bf3f01)
/usr/local/go/src/runtime/mgcscavenge.go:276 +0xee fp=0xc00006ff40 sp=0xc00006fef0 pc=0x4233be
runtime.bgscavenge(0xc000096000)
/usr/local/go/src/runtime/mgcscavenge.go:400 +0x2e0 fp=0xc00006ffd8 sp=0xc00006ff40 pc=0x4236d0
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc00006ffe0 sp=0xc00006ffd8 pc=0x460c71
created by runtime.gcenable
/usr/local/go/src/runtime/mgc.go:211 +0x7e

goroutine 5 [finalizer wait, 366 minutes]:
runtime.gopark(0x1181f08, 0x1c13538, 0x1410, 0x1)
/usr/local/go/src/runtime/proc.go:304 +0xe0 fp=0xc00006e758 sp=0xc00006e738 pc=0x432820
runtime.goparkunlock(…)
/usr/local/go/src/runtime/proc.go:310
runtime.runfinq()
/usr/local/go/src/runtime/mfinal.go:175 +0xa3 fp=0xc00006e7e0 sp=0xc00006e758 pc=0x419d43
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc00006e7e8 sp=0xc00006e7e0 pc=0x460c71
created by runtime.createfing
/usr/local/go/src/runtime/mfinal.go:156 +0x61

goroutine 6 [syscall]:
runtime.notetsleepg(0x1bf9820, 0x2540abd9c, 0x0)
/usr/local/go/src/runtime/lock_futex.go:227 +0x34 fp=0xc000070760 sp=0xc000070730 pc=0x40c504
runtime.timerproc(0x1bf9800)
/usr/local/go/src/runtime/time.go:311 +0x2f1 fp=0xc0000707d8 sp=0xc000070760 pc=0x44f971
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc0000707e0 sp=0xc0000707d8 pc=0x460c71
created by runtime.(*timersBucket).addtimerLocked
/usr/local/go/src/runtime/time.go:169 +0x10e

goroutine 34 [syscall, 366 minutes]:
runtime.notetsleepg(0x1c13cc0, 0xffffffffffffffff, 0xc000070fc8)
/usr/local/go/src/runtime/lock_futex.go:227 +0x34 fp=0xc000070f98 sp=0xc000070f68 pc=0x40c504
os/signal.signal_recv(0x0)
/usr/local/go/src/runtime/sigqueue.go:147 +0x9c fp=0xc000070fc0 sp=0xc000070f98 pc=0x44703c
os/signal.loop()
/usr/local/go/src/os/signal/signal_unix.go:23 +0x22 fp=0xc000070fe0 sp=0xc000070fc0 pc=0xa06682
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc000070fe8 sp=0xc000070fe0 pc=0x460c71
created by os/signal.init.0
/usr/local/go/src/os/signal/signal_unix.go:29 +0x41

goroutine 13 [sleep, 6 minutes]:
runtime.gopark(0x1181f08, 0x1bf9800, 0x1313, 0x2)
/usr/local/go/src/runtime/proc.go:304 +0xe0 fp=0xc00006a748 sp=0xc00006a728 pc=0x432820
runtime.goparkunlock(…)
/usr/local/go/src/runtime/proc.go:310
time.Sleep(0x8bb2c97000)
/usr/local/go/src/runtime/time.go:105 +0x157 fp=0xc00006a798 sp=0xc00006a748 pc=0x44ee07
github.com/spacemonkeygo/monkit/v3.(*ticker).run(0x1bf3ae0)
/go/pkg/mod/github.com/spacemonkeygo/monkit/v3@v3.0.6-0.20200327233048-76480079049a/meter.go:203 +0x6a fp=0xc00006a7d8 sp=0xc00006a798 pc=0x910a9a
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc00006a7e0 sp=0xc00006a7d8 pc=0x460c71
created by github.com/spacemonkeygo/monkit/v3.(*ticker).register
/go/pkg/mod/github.com/spacemonkeygo/monkit/v3@v3.0.6-0.20200327233048-76480079049a/meter.go:195 +0x131

goroutine 14 [GC worker (idle)]:
runtime.gopark(0x1181da0, 0xc0001460d0, 0x401418, 0x0)
/usr/local/go/src/runtime/proc.go:304 +0xe0 fp=0xc000071760 sp=0xc000071740 pc=0x432820
runtime.gcBgMarkWorker(0xc000040000)
/usr/local/go/src/runtime/mgc.go:1846 +0xff fp=0xc0000717d8 sp=0xc000071760 pc=0x41d74f
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc0000717e0 sp=0xc0000717d8 pc=0x460c71
created by runtime.gcBgMarkStartWorkers
/usr/local/go/src/runtime/mgc.go:1794 +0x77

goroutine 12 [select]:
runtime.gopark(0x1181f48, 0x0, 0x1809, 0x1)
/usr/local/go/src/runtime/proc.go:304 +0xe0 fp=0xc0001ce5e0 sp=0xc0001ce5c0 pc=0x432820
runtime.selectgo(0xc0001ce750, 0xc0001ce734, 0x3, 0x1bf43a0, 0x112d301)
/usr/local/go/src/runtime/select.go:313 +0xc9b fp=0xc0001ce708 sp=0xc0001ce5e0 pc=0x44232b
go.opencensus.io/stats/view.(*worker).start(0xc0000cd540)
/go/pkg/mod/go.opencensus.io@v0.22.2/stats/view/worker.go:154 +0x100 fp=0xc0001ce7d8 sp=0xc0001ce708 pc=0xb5f770
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc0001ce7e0 sp=0xc0001ce7d8 pc=0x460c71
created by go.opencensus.io/stats/view.init.0
/go/pkg/mod/go.opencensus.io@v0.22.2/stats/view/worker.go:32 +0x57

goroutine 36 [syscall]:
runtime.notetsleepg(0x1bf98a0, 0x1a0a71c5e1c, 0x0)
/usr/local/go/src/runtime/lock_futex.go:227 +0x34 fp=0xc0001cef60 sp=0xc0001cef30 pc=0x40c504
runtime.timerproc(0x1bf9880)
/usr/local/go/src/runtime/time.go:311 +0x2f1 fp=0xc0001cefd8 sp=0xc0001cef60 pc=0x44f971
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc0001cefe0 sp=0xc0001cefd8 pc=0x460c71
created by runtime.(*timersBucket).addtimerLocked
/usr/local/go/src/runtime/time.go:169 +0x10e

goroutine 21 [GC worker (idle)]:
runtime.gopark(0x1181da0, 0xc000154270, 0x1418, 0x0)
/usr/local/go/src/runtime/proc.go:304 +0xe0 fp=0xc00006af60 sp=0xc00006af40 pc=0x432820
runtime.gcBgMarkWorker(0xc000042500)
/usr/local/go/src/runtime/mgc.go:1846 +0xff fp=0xc00006afd8 sp=0xc00006af60 pc=0x41d74f
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc00006afe0 sp=0xc00006afd8 pc=0x460c71
created by runtime.gcBgMarkStartWorkers
/usr/local/go/src/runtime/mgc.go:1794 +0x77

goroutine 15 [GC worker (idle)]:
runtime.gopark(0x1181da0, 0xc000154280, 0x1418, 0x0)
/usr/local/go/src/runtime/proc.go:304 +0xe0 fp=0xc000071f60 sp=0xc000071f40 pc=0x432820
runtime.gcBgMarkWorker(0xc000044a00)
/usr/local/go/src/runtime/mgc.go:1846 +0xff fp=0xc000071fd8 sp=0xc000071f60 pc=0x41d74f
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc000071fe0 sp=0xc000071fd8 pc=0x460c71
created by runtime.gcBgMarkStartWorkers
/usr/local/go/src/runtime/mgc.go:1794 +0x77

goroutine 37 [GC worker (idle)]:
runtime.systemstack_switch()
/usr/local/go/src/runtime/asm_amd64.s:330 fp=0xc0001cf760 sp=0xc0001cf758 pc=0x45eba0
runtime.gcBgMarkWorker(0xc000046f00)
/usr/local/go/src/runtime/mgc.go:1900 +0x1bd fp=0xc0001cf7d8 sp=0xc0001cf760 pc=0x41d80d
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc0001cf7e0 sp=0xc0001cf7d8 pc=0x460c71
created by runtime.gcBgMarkStartWorkers
/usr/local/go/src/runtime/mgc.go:1794 +0x77

goroutine 16 [GC worker (idle)]:
runtime.gopark(0x1181da0, 0xc0001460f0, 0x1418, 0x0)
/usr/local/go/src/runtime/proc.go:304 +0xe0 fp=0xc0001ca760 sp=0xc0001ca740 pc=0x432820
runtime.gcBgMarkWorker(0xc000049400)
/usr/local/go/src/runtime/mgc.go:1846 +0xff fp=0xc0001ca7d8 sp=0xc0001ca760 pc=0x41d74f
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc0001ca7e0 sp=0xc0001ca7d8 pc=0x460c71
created by runtime.gcBgMarkStartWorkers
/usr/local/go/src/runtime/mgc.go:1794 +0x77

goroutine 38 [GC worker (idle)]:
runtime.gopark(0x1181da0, 0xc000146100, 0x1418, 0x0)
/usr/local/go/src/runtime/proc.go:304 +0xe0 fp=0xc0001cff60 sp=0xc0001cff40 pc=0x432820
runtime.gcBgMarkWorker(0xc00004b900)
/usr/local/go/src/runtime/mgc.go:1846 +0xff fp=0xc0001cffd8 sp=0xc0001cff60 pc=0x41d74f
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc0001cffe0 sp=0xc0001cffd8 pc=0x460c71
created by runtime.gcBgMarkStartWorkers
/usr/local/go/src/runtime/mgc.go:1794 +0x77

goroutine 50 [GC worker (idle)]:
runtime.gopark(0x1181da0, 0xc000146110, 0x1418, 0x0)
/usr/local/go/src/runtime/proc.go:304 +0xe0 fp=0xc0001caf60 sp=0xc0001caf40 pc=0x432820
runtime.gcBgMarkWorker(0xc00004e000)
/usr/local/go/src/runtime/mgc.go:1846 +0xff fp=0xc0001cafd8 sp=0xc0001caf60 pc=0x41d74f
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc0001cafe0 sp=0xc0001cafd8 pc=0x460c71
created by runtime.gcBgMarkStartWorkers
/usr/local/go/src/runtime/mgc.go:1794 +0x77

goroutine 39 [GC worker (idle)]:
runtime.gopark(0x1181da0, 0xc000146120, 0x1418, 0x0)
/usr/local/go/src/runtime/proc.go:304 +0xe0 fp=0xc0001d0760 sp=0xc0001d0740 pc=0x432820
runtime.gcBgMarkWorker(0xc000050500)
/usr/local/go/src/runtime/mgc.go:1846 +0xff fp=0xc0001d07d8 sp=0xc0001d0760 pc=0x41d74f
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc0001d07e0 sp=0xc0001d07d8 pc=0x460c71
created by runtime.gcBgMarkStartWorkers
/usr/local/go/src/runtime/mgc.go:1794 +0x77

goroutine 51 [GC worker (idle)]:
runtime.gopark(0x1181da0, 0xc000146130, 0x1418, 0x0)
/usr/local/go/src/runtime/proc.go:304 +0xe0 fp=0xc0001cb760 sp=0xc0001cb740 pc=0x432820
runtime.gcBgMarkWorker(0xc000052a00)
/usr/local/go/src/runtime/mgc.go:1846 +0xff fp=0xc0001cb7d8 sp=0xc0001cb760 pc=0x41d74f
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc0001cb7e0 sp=0xc0001cb7d8 pc=0x460c71
created by runtime.gcBgMarkStartWorkers
/usr/local/go/src/runtime/mgc.go:1794 +0x77

goroutine 40 [GC worker (idle)]:
runtime.gopark(0x1181da0, 0xc000146140, 0x1418, 0x0)
/usr/local/go/src/runtime/proc.go:304 +0xe0 fp=0xc0001d0f60 sp=0xc0001d0f40 pc=0x432820
runtime.gcBgMarkWorker(0xc000054f00)
/usr/local/go/src/runtime/mgc.go:1846 +0xff fp=0xc0001d0fd8 sp=0xc0001d0f60 pc=0x41d74f
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc0001d0fe0 sp=0xc0001d0fd8 pc=0x460c71
created by runtime.gcBgMarkStartWorkers
/usr/local/go/src/runtime/mgc.go:1794 +0x77

goroutine 22 [GC worker (idle)]:
runtime.gopark(0x1181da0, 0xc0004cf110, 0x1418, 0x0)
/usr/local/go/src/runtime/proc.go:304 +0xe0 fp=0xc00006b760 sp=0xc00006b740 pc=0x432820
runtime.gcBgMarkWorker(0xc000057400)
/usr/local/go/src/runtime/mgc.go:1846 +0xff fp=0xc00006b7d8 sp=0xc00006b760 pc=0x41d74f
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc00006b7e0 sp=0xc00006b7d8 pc=0x460c71
created by runtime.gcBgMarkStartWorkers
/usr/local/go/src/runtime/mgc.go:1794 +0x77

goroutine 41 [GC worker (idle)]:
runtime.gopark(0x1181da0, 0xc0004cf120, 0x1418, 0x0)
/usr/local/go/src/runtime/proc.go:304 +0xe0 fp=0xc0001d1760 sp=0xc0001d1740 pc=0x432820
runtime.gcBgMarkWorker(0xc000059900)
/usr/local/go/src/runtime/mgc.go:1846 +0xff fp=0xc0001d17d8 sp=0xc0001d1760 pc=0x41d74f
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc0001d17e0 sp=0xc0001d17d8 pc=0x460c71
created by runtime.gcBgMarkStartWorkers
/usr/local/go/src/runtime/mgc.go:1794 +0x77

goroutine 23 [GC worker (idle), 5 minutes]:
runtime.gopark(0x1181da0, 0xc0004cf130, 0x1418, 0x0)
/usr/local/go/src/runtime/proc.go:304 +0xe0 fp=0xc00006bf60 sp=0xc00006bf40 pc=0x432820
runtime.gcBgMarkWorker(0xc00005c000)
/usr/local/go/src/runtime/mgc.go:1846 +0xff fp=0xc00006bfd8 sp=0xc00006bf60 pc=0x41d74f
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc00006bfe0 sp=0xc00006bfd8 pc=0x460c71
created by runtime.gcBgMarkStartWorkers
/usr/local/go/src/runtime/mgc.go:1794 +0x77

goroutine 42 [GC worker (idle), 8 minutes]:
runtime.gopark(0x1181da0, 0xc0004cf140, 0x1418, 0x0)
/usr/local/go/src/runtime/proc.go:304 +0xe0 fp=0xc0001d1f60 sp=0xc0001d1f40 pc=0x432820
runtime.gcBgMarkWorker(0xc00005e500)
/usr/local/go/src/runtime/mgc.go:1846 +0xff fp=0xc0001d1fd8 sp=0xc0001d1f60 pc=0x41d74f
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc0001d1fe0 sp=0xc0001d1fd8 pc=0x460c71
created by runtime.gcBgMarkStartWorkers
/usr/local/go/src/runtime/mgc.go:1794 +0x77

goroutine 24 [GC worker (idle), 12 minutes]:
runtime.gopark(0x1181da0, 0xc0004cf150, 0x1418, 0x0)
/usr/local/go/src/runtime/proc.go:304 +0xe0 fp=0xc00006c760 sp=0xc00006c740 pc=0x432820
runtime.gcBgMarkWorker(0xc000060a00)
/usr/local/go/src/runtime/mgc.go:1846 +0xff fp=0xc00006c7d8 sp=0xc00006c760 pc=0x41d74f
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc00006c7e0 sp=0xc00006c7d8 pc=0x460c71
created by runtime.gcBgMarkStartWorkers
/usr/local/go/src/runtime/mgc.go:1794 +0x77

goroutine 43 [GC worker (idle), 15 minutes]:
runtime.gopark(0x1181da0, 0xc0004cf160, 0x1418, 0x0)
/usr/local/go/src/runtime/proc.go:304 +0xe0 fp=0xc000500760 sp=0xc000500740 pc=0x432820
runtime.gcBgMarkWorker(0xc000062f00)
/usr/local/go/src/runtime/mgc.go:1846 +0xff fp=0xc0005007d8 sp=0xc000500760 pc=0x41d74f
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc0005007e0 sp=0xc0005007d8 pc=0x460c71
created by runtime.gcBgMarkStartWorkers
/usr/local/go/src/runtime/mgc.go:1794 +0x77

goroutine 25 [syscall]:
runtime.notetsleepg(0x1bf9920, 0x2f1d050e1da, 0x0)
/usr/local/go/src/runtime/lock_futex.go:227 +0x34 fp=0xc0001cbf60 sp=0xc0001cbf30 pc=0x40c504
runtime.timerproc(0x1bf9900)
/usr/local/go/src/runtime/time.go:311 +0x2f1 fp=0xc0001cbfd8 sp=0xc0001cbf60 pc=0x44f971
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc0001cbfe0 sp=0xc0001cbfd8 pc=0x460c71
created by runtime.(*timersBucket).addtimerLocked
/usr/local/go/src/runtime/time.go:169 +0x10e

goroutine 44 [select, 366 minutes, locked to thread]:
runtime.gopark(0x1181f48, 0x0, 0x1809, 0x1)
/usr/local/go/src/runtime/proc.go:304 +0xe0 fp=0xc00007fde0 sp=0xc00007fdc0 pc=0x432820
runtime.selectgo(0xc00007ff80, 0xc00007ff40, 0x2, 0x8, 0x1)
/usr/local/go/src/runtime/select.go:313 +0xc9b fp=0xc00007ff08 sp=0xc00007fde0 pc=0x44232b
runtime.ensureSigM.func1()
/usr/local/go/src/runtime/signal_unix.go:549 +0x1e8 fp=0xc00007ffe0 sp=0xc00007ff08 pc=0x45e2f8
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc00007ffe8 sp=0xc00007ffe0 pc=0x460c71
created by runtime.ensureSigM
/usr/local/go/src/runtime/signal_unix.go:532 +0xd5

goroutine 26 [chan receive, 366 minutes]:
runtime.gopark(0x1181f08, 0xc000442418, 0x170e, 0x3)
/usr/local/go/src/runtime/proc.go:304 +0xe0 fp=0xc0001cc6b8 sp=0xc0001cc698 pc=0x432820
runtime.goparkunlock(…)
/usr/local/go/src/runtime/proc.go:310
runtime.chanrecv(0xc0004423c0, 0xc0001cc7b0, 0x1, 0x0)
/usr/local/go/src/runtime/chan.go:524 +0x2e8 fp=0xc0001cc748 sp=0xc0001cc6b8 pc=0x4070c8
runtime.chanrecv1(0xc0004423c0, 0xc0001cc7b0)
/usr/local/go/src/runtime/chan.go:406 +0x2b fp=0xc0001cc778 sp=0xc0001cc748 pc=0x406d8b
storj.io/private/process.Ctx.func1(0xc0004423c0, 0xc0003252f0)
/go/pkg/mod/storj.io/private@v0.0.0-20200415180736-fe6c6bb80e76/process/exec_conf.go:112 +0x41 fp=0xc0001cc7d0 sp=0xc0001cc778 pc=0xbe73b1
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc0001cc7d8 sp=0xc0001cc7d0 pc=0x460c71
created by storj.io/private/process.Ctx
/go/pkg/mod/storj.io/private@v0.0.0-20200415180736-fe6c6bb80e76/process/exec_conf.go:111 +0x2d5

goroutine 66 [syscall]:
runtime.notetsleepg(0x1bf99a0, 0x1a2fe4dda96, 0x0)
/usr/local/go/src/runtime/lock_futex.go:227 +0x34 fp=0xc0004fff60 sp=0xc0004fff30 pc=0x40c504
runtime.timerproc(0x1bf9980)
/usr/local/go/src/runtime/time.go:311 +0x2f1 fp=0xc0004fffd8 sp=0xc0004fff60 pc=0x44f971
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc0004fffe0 sp=0xc0004fffd8 pc=0x460c71
created by runtime.(*timersBucket).addtimerLocked
/usr/local/go/src/runtime/time.go:169 +0x10e

goroutine 29 [select, 366 minutes]:
runtime.gopark(0x1181f48, 0x0, 0x1809, 0x1)
/usr/local/go/src/runtime/proc.go:304 +0xe0 fp=0xc0001cd610 sp=0xc0001cd5f0 pc=0x432820
runtime.selectgo(0xc0001cd768, 0xc0001cd760, 0x2, 0x0, 0x0)
/usr/local/go/src/runtime/select.go:313 +0xc9b fp=0xc0001cd738 sp=0xc0001cd610 pc=0x44232b
database/sql.(*DB).connectionOpener(0xc000434000, 0x12d9cc0, 0xc0001e2340)
/usr/local/go/src/database/sql/sql.go:1052 +0xe8 fp=0xc0001cd7c8 sp=0xc0001cd738 pc=0xbf5038
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc0001cd7d0 sp=0xc0001cd7c8 pc=0x460c71
created by database/sql.OpenDB
/usr/local/go/src/database/sql/sql.go:722 +0x15d

goroutine 30 [select, 366 minutes]:
runtime.gopark(0x1181f48, 0x0, 0x1809, 0x1)
/usr/local/go/src/runtime/proc.go:304 +0xe0 fp=0xc0001cddf8 sp=0xc0001cddd8 pc=0x432820
runtime.selectgo(0xc0001cdf68, 0xc0001cdf48, 0x2, 0x0, 0x0)
/usr/local/go/src/runtime/select.go:313 +0xc9b fp=0xc0001cdf20 sp=0xc0001cddf8 pc=0x44232b
database/sql.(*DB).connectionResetter(0xc000434000, 0x12d9cc0, 0xc0001e2340)
/usr/local/go/src/database/sql/sql.go:1065 +0xfb fp=0xc0001cdfc8 sp=0xc0001cdf20 pc=0xbf516b
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc0001cdfd0 sp=0xc0001cdfc8 pc=0x460c71
created by database/sql.OpenDB
/usr/local/go/src/database/sql/sql.go:723 +0x193

goroutine 31 [select, 366 minutes]:
runtime.gopark(0x1181f48, 0x0, 0x1809, 0x1)
/usr/local/go/src/runtime/proc.go:304 +0xe0 fp=0xc00006ce10 sp=0xc00006cdf0 pc=0x432820
runtime.selectgo(0xc00006cf68, 0xc00006cf60, 0x2, 0x0, 0x0)
/usr/local/go/src/runtime/select.go:313 +0xc9b fp=0xc00006cf38 sp=0xc00006ce10 pc=0x44232b
database/sql.(*DB).connectionOpener(0xc0004340c0, 0x12d9cc0, 0xc0001e2380)
/usr/local/go/src/database/sql/sql.go:1052 +0xe8 fp=0xc00006cfc8 sp=0xc00006cf38 pc=0xbf5038
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc00006cfd0 sp=0xc00006cfc8 pc=0x460c71
created by database/sql.OpenDB
/usr/local/go/src/database/sql/sql.go:722 +0x15d

goroutine 32 [select, 366 minutes]:
runtime.gopark(0x1181f48, 0x0, 0x1809, 0x1)
/usr/local/go/src/runtime/proc.go:304 +0xe0 fp=0xc00006d5f8 sp=0xc00006d5d8 pc=0x432820
runtime.selectgo(0xc00006d768, 0xc00006d748, 0x2, 0x0, 0x0)
/usr/local/go/src/runtime/select.go:313 +0xc9b fp=0xc00006d720 sp=0xc00006d5f8 pc=0x44232b
database/sql.(*DB).connectionResetter(0xc0004340c0, 0x12d9cc0, 0xc0001e2380)
/usr/local/go/src/database/sql/sql.go:1065 +0xfb fp=0xc00006d7c8 sp=0xc00006d720 pc=0xbf516b
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc00006d7d0 sp=0xc00006d7c8 pc=0x460c71
created by database/sql.OpenDB
/usr/local/go/src/database/sql/sql.go:723 +0x193

goroutine 33 [select, 366 minutes]:
runtime.gopark(0x1181f48, 0x0, 0x1809, 0x1)
/usr/local/go/src/runtime/proc.go:304 +0xe0 fp=0xc00006de10 sp=0xc00006ddf0 pc=0x432820
runtime.selectgo(0xc00006df68, 0xc00006df60, 0x2, 0x0, 0x0)
/usr/local/go/src/runtime/select.go:313 +0xc9b fp=0xc00006df38 sp=0xc00006de10 pc=0x44232b
database/sql.(*DB).connectionOpener(0xc000434180, 0x12d9cc0, 0xc0001e23c0)
/usr/local/go/src/database/sql/sql.go:1052 +0xe8 fp=0xc00006dfc8 sp=0xc00006df38 pc=0xbf5038
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc00006dfd0 sp=0xc00006dfc8 pc=0x460c71
created by database/sql.OpenDB
/usr/local/go/src/database/sql/sql.go:722 +0x15d

goroutine 82 [select, 366 minutes]:
runtime.gopark(0x1181f48, 0x0, 0x1809, 0x1)
/usr/local/go/src/runtime/proc.go:304 +0xe0 fp=0xc000459df8 sp=0xc000459dd8 pc=0x432820
runtime.selectgo(0xc000459f68, 0xc000459f48, 0x2, 0x0, 0x0)
/usr/local/go/src/runtime/select.go:313 +0xc9b fp=0xc000459f20 sp=0xc000459df8 pc=0x44232b
database/sql.(*DB).connectionResetter(0xc000434180, 0x12d9cc0, 0xc0001e23c0)
/usr/local/go/src/database/sql/sql.go:1065 +0xfb fp=0xc000459fc8 sp=0xc000459f20 pc=0xbf516b
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc000459fd0 sp=0xc000459fc8 pc=0x460c71
created by database/sql.OpenDB
/usr/local/go/src/database/sql/sql.go:723 +0x193

goroutine 83 [select, 366 minutes]:
runtime.gopark(0x1181f48, 0x0, 0x1809, 0x1)
/usr/local/go/src/runtime/proc.go:304 +0xe0 fp=0xc0004fc610 sp=0xc0004fc5f0 pc=0x432820
runtime.selectgo(0xc0004fc768, 0xc0004fc760, 0x2, 0x0, 0x0)
/usr/local/go/src/runtime/select.go:313 +0xc9b fp=0xc0004fc738 sp=0xc0004fc610 pc=0x44232b
database/sql.(*DB).connectionOpener(0xc000434240, 0x12d9cc0, 0xc0001e2440)
/usr/local/go/src/database/sql/sql.go:1052 +0xe8 fp=0xc0004fc7c8 sp=0xc0004fc738 pc=0xbf5038
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc0004fc7d0 sp=0xc0004fc7c8 pc=0x460c71
created by database/sql.OpenDB
/usr/local/go/src/database/sql/sql.go:722 +0x15d

goroutine 84 [select, 366 minutes]:
runtime.gopark(0x1181f48, 0x0, 0x1809, 0x1)
/usr/local/go/src/runtime/proc.go:304 +0xe0 fp=0xc0004fcdf8 sp=0xc0004fcdd8 pc=0x432820
runtime.selectgo(0xc0004fcf68, 0xc0004fcf48, 0x2, 0x0, 0x0)
/usr/local/go/src/runtime/select.go:313 +0xc9b fp=0xc0004fcf20 sp=0xc0004fcdf8 pc=0x44232b
database/sql.(*DB).connectionResetter(0xc000434240, 0x12d9cc0, 0xc0001e2440)
/usr/local/go/src/database/sql/sql.go:1065 +0xfb fp=0xc0004fcfc8 sp=0xc0004fcf20 pc=0xbf516b
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc0004fcfd0 sp=0xc0004fcfc8 pc=0x460c71
created by database/sql.OpenDB
/usr/local/go/src/database/sql/sql.go:723 +0x193

goroutine 85 [select, 366 minutes]:
runtime.gopark(0x1181f48, 0x0, 0x1809, 0x1)
/usr/local/go/src/runtime/proc.go:304 +0xe0 fp=0xc0004fd610 sp=0xc0004fd5f0 pc=0x432820
runtime.selectgo(0xc0004fd768, 0xc0004fd760, 0x2, 0x0, 0x0)
/usr/local/go/src/runtime/select.go:313 +0xc9b fp=0xc0004fd738 sp=0xc0004fd610 pc=0x44232b
database/sql.(*DB).connectionOpener(0xc000434300, 0x12d9cc0, 0xc0001e24c0)
/usr/local/go/src/database/sql/sql.go:1052 +0xe8 fp=0xc0004fd7c8 sp=0xc0004fd738 pc=0xbf5038
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc0004fd7d0 sp=0xc0004fd7c8 pc=0x460c71
created by database/sql.OpenDB
/usr/local/go/src/database/sql/sql.go:722 +0x15d

goroutine 86 [select, 366 minutes]:
runtime.gopark(0x1181f48, 0x0, 0x1809, 0x1)
/usr/local/go/src/runtime/proc.go:304 +0xe0 fp=0xc0004fddf8 sp=0xc0004fddd8 pc=0x432820
runtime.selectgo(0xc0004fdf68, 0xc0004fdf48, 0x2, 0x0, 0x0)
/usr/local/go/src/runtime/select.go:313 +0xc9b fp=0xc0004fdf20 sp=0xc0004fddf8 pc=0x44232b
database/sql.(*DB).connectionResetter(0xc000434300, 0x12d9cc0, 0xc0001e24c0)
/usr/local/go/src/database/sql/sql.go:1065 +0xfb fp=0xc0004fdfc8 sp=0xc0004fdf20 pc=0xbf516b
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc0004fdfd0 sp=0xc0004fdfc8 pc=0x460c71
created by database/sql.OpenDB
/usr/local/go/src/database/sql/sql.go:723 +0x193

… And So On .But got limited

By the way. I haven’t get any tips abort update image since v1.3.3 …

[qinhang3]

Docker has tried to restart this container …But can’t solve this problem .

[nerdatwork]

Could you show log by using command

docker logs --tail 20 storagenode

[qinhang3]

Thanks for your reply …
Here comes my log …

my log

2020-06-09T00:40:03.903Z INFO Configuration loaded {“Location”: “/app/config/config.yaml”}
2020-06-09T00:40:03.903Z INFO tracing disabled
2020-06-09T00:40:03.910Z INFO Operator email {“Address”: “xx@xxx.com”}
2020-06-09T00:40:03.910Z INFO Operator wallet {“Address”: “0x60C2924537xxxxxxxxxxxxxxxx87279d30a”}
2020-06-09T00:40:04.894Z INFO db.migration Database Version {“version”: 36}
2020-06-09T00:40:07.196Z INFO preflight:localtime start checking local system clock with trusted satellites’ system clock.
2020-06-09T00:40:09.143Z INFO preflight:localtime local system clock is in sync with trusted satellites’ system clock.
2020-06-09T00:40:09.143Z INFO bandwidth Performing bandwidth usage rollups
runtime: pointer 0xc0002b7ff0 to unused region of span span.base()=0xc0002b6000 span.limit=0xc0002b7f80 span.state=1
fatal error: found bad pointer in Go heap (incorrect use of unsafe or cgo?)

runtime stack:
runtime.throw(0x1166866, 0x3e)
/usr/local/go/src/runtime/panic.go:774 +0x72 fp=0x7f20ada1ad78 sp=0x7f20ada1ad48 pc=0x430a82
runtime.findObject(0xc0002b7ff0, 0x0, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/runtime/mbitmap.go:397 +0x3b4 fp=0x7f20ada1adc8 sp=0x7f20ada1ad78 pc=0x415fd4
runtime.wbBufFlush1(0xc00004b900)
/usr/local/go/src/runtime/mwbbuf.go:288 +0xa5 fp=0x7f20ada1ae20 sp=0x7f20ada1adc8 pc=0x42c5c5
runtime.wbBufFlush.func1()
/usr/local/go/src/runtime/mwbbuf.go:218 +0x3a fp=0x7f20ada1ae38 sp=0x7f20ada1ae20 pc=0x45ce0a
runtime.systemstack(0x1c12a68)
/usr/local/go/src/runtime/asm_amd64.s:370 +0x66 fp=0x7f20ada1ae40 sp=0x7f20ada1ae38 pc=0x45ec16
runtime.mstart()
/usr/local/go/src/runtime/proc.go:1146 fp=0x7f20ada1ae48 sp=0x7f20ada1ae40 pc=0x435070

goroutine 892 [running]:
runtime.systemstack_switch()
/usr/local/go/src/runtime/asm_amd64.s:330 fp=0xc000298c88 sp=0xc000298c80 pc=0x45eba0
runtime.wbBufFlush(0x0, 0x0)
/usr/local/go/src/runtime/mwbbuf.go:206 +0x66 fp=0xc000298cb8 sp=0xc000298c88 pc=0x42c496
runtime.bulkBarrierPreWrite(0xc00031c080, 0xc000298dd0, 0x40040)
/usr/local/go/src/runtime/mbitmap.go:640 +0x2eb fp=0xc000298d40 sp=0xc000298cb8 pc=0x41650b
runtime.typedmemmove(0x1098ee0, 0xc00031c080, 0xc000298dd0)
/usr/local/go/src/runtime/mbarrier.go:161 +0xa1 fp=0xc000298d78 sp=0xc000298d40 pc=0x415321
storj.io/storj/storagenode/trust.(*Pool).Run(0xc0001c6be0, 0x12d9fc0, 0xc000578be0, 0xc00041d4d0, 0x20)
/go/src/storj.io/storj/storagenode/trust/service.go:110 +0x420 fp=0xc000298ee0 sp=0xc000298d78 pc=0xc9d400
storj.io/storj/storagenode/trust.(*Pool).Run-fm(0x12d9fc0, 0xc000578be0, 0x90ac90, 0x42f82a)
/go/src/storj.io/storj/storagenode/trust/service.go:107 +0x3e fp=0xc000298f18 sp=0xc000298ee0 pc=0xd3dc0e
storj.io/storj/private/lifecycle.(*Group).Run.func1(0xc000298f68, 0x952038)
/go/src/storj.io/storj/private/lifecycle/group.go:56 +0x3a fp=0xc000298f58 sp=0xc000298f18 pc=0xc822aa
golang.org/x/sync/errgroup.(*Group).Go.func1(0xc000674480, 0xc0006746c0)
/go/pkg/mod/golang.org/x/sync@v0.0.0-20190911185100-cd5d95a43a6e/errgroup/errgroup.go:57 +0x64 fp=0xc000298fd0 sp=0xc000298f58 pc=0x9c4324
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc000298fd8 sp=0xc000298fd0 pc=0x460c71
created by golang.org/x/sync/errgroup.(*Group).Go
/go/pkg/mod/golang.org/x/sync@v0.0.0-20190911185100-cd5d95a43a6e/errgroup/errgroup.go:54 +0x66

goroutine 1 [semacquire]:
runtime.gopark(0x1181f08, 0x1bfe900, 0xc0003f1912, 0x4)
/usr/local/go/src/runtime/proc.go:304 +0xe0 fp=0xc0006f84a0 sp=0xc0006f8480 pc=0x432820
runtime.goparkunlock(…)
/usr/local/go/src/runtime/proc.go:310
runtime.semacquire1(0xc000674490, 0x0, 0x1, 0x0)
/usr/local/go/src/runtime/sema.go:144 +0x1c0 fp=0xc0006f8508 sp=0xc0006f84a0 pc=0x442e90
sync.runtime_Semacquire(0xc000674490)
/usr/local/go/src/runtime/sema.go:56 +0x42 fp=0xc0006f8538 sp=0xc0006f8508 pc=0x442ae2
sync.(*WaitGroup).Wait(0xc000674488)
/usr/local/go/src/sync/waitgroup.go:130 +0x64 fp=0xc0006f8560 sp=0xc0006f8538 pc=0x471374
golang.org/x/sync/errgroup.(*Group).Wait(0xc000674480, 0x12d9cc0, 0xc000336200)
/go/pkg/mod/golang.org/x/sync@v0.0.0-20190911185100-cd5d95a43a6e/errgroup/errgroup.go:40 +0x31 fp=0xc0006f8578 sp=0xc0006f8560 pc=0x9c4191
storj.io/storj/storagenode.(*Peer).Run(0xc000558d80, 0x12d9cc0, 0xc0000f9940, 0x0, 0x0)
/go/src/storj.io/storj/storagenode/peer.go:678 +0x57a fp=0xc0006f8700 sp=0xc0006f8578 pc=0xd3d2da
main.cmdRun(0x1a2da80, 0xc0001cbd90, 0x0, 0xb, 0x0, 0x0)
/go/src/storj.io/storj/cmd/storagenode/main.go:209 +0xf77 fp=0xc0006f8f40 sp=0xc0006f8700 pc=0xdd4f37
storj.io/private/process.cleanup.func1.4(0x12d9fc0, 0xc0001803c0)
/go/pkg/mod/storj.io/private@v0.0.0-20200415180736-fe6c6bb80e76/process/exec_conf.go:344 +0x13b fp=0xc0006f8fd8 sp=0xc0006f8f40 pc=0xbe7bab
storj.io/private/process.cleanup.func1(0x1a2da80, 0xc0001cbd90, 0x0, 0xb, 0x0, 0x0)
/go/pkg/mod/storj.io/private@v0.0.0-20200415180736-fe6c6bb80e76/process/exec_conf.go:362 +0x1df3 fp=0xc0006f9d38 sp=0xc0006f8fd8 pc=0xbe9a53
github.com/spf13/cobra.(*Command).execute(0x1a2da80, 0xc0001cbce0, 0xb, 0xb, 0x1a2da80, 0xc0001cbce0)
/go/pkg/mod/github.com/spf13/cobra@v0.0.6/command.go:840 +0x460 fp=0xc0006f9e10 sp=0xc0006f9d38 pc=0x6367b0
github.com/spf13/cobra.(*Command).ExecuteC(0x1a2d7e0, 0xc0000d8100, 0x1182360, 0x1)
/go/pkg/mod/github.com/spf13/cobra@v0.0.6/command.go:945 +0x317 fp=0xc0006f9ee8 sp=0xc0006f9e10 pc=0x6372a7
github.com/spf13/cobra.(*Command).Execute(…)
/go/pkg/mod/github.com/spf13/cobra@v0.0.6/command.go:885
storj.io/private/process.ExecWithCustomConfig(0x1a2d7e0, 0x0, 0x1182360)
/go/pkg/mod/storj.io/private@v0.0.0-20200415180736-fe6c6bb80e76/process/exec_conf.go:86 +0x192 fp=0xc0006f9f38 sp=0xc0006f9ee8 pc=0xbe34d2
storj.io/private/process.ExecCustomDebug(...)
/go/pkg/mod/storj.io/private@v0.0.0-20200415180736-fe6c6bb80e76/process/exec_conf.go:68
main.main()
/go/src/storj.io/storj/cmd/storagenode/main.go:329 +0x3e fp=0xc0006f9f60 sp=0xc0006f9f38 pc=0xdd63ce
runtime.main()
/usr/local/go/src/runtime/proc.go:203 +0x21e fp=0xc0006f9fe0 sp=0xc0006f9f60 pc=0x43241e
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc0006f9fe8 sp=0xc0006f9fe0 pc=0x460c71

goroutine 2 [force gc (idle)]:
runtime.gopark(0x1181f08, 0x1bf13b0, 0x1411, 0x1)
/usr/local/go/src/runtime/proc.go:304 +0xe0 fp=0xc00006efb0 sp=0xc00006ef90 pc=0x432820
runtime.goparkunlock(…)
/usr/local/go/src/runtime/proc.go:310
runtime.forcegchelper()
/usr/local/go/src/runtime/proc.go:253 +0xb7 fp=0xc00006efe0 sp=0xc00006efb0 pc=0x4326a7
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc00006efe8 sp=0xc00006efe0 pc=0x460c71
created by runtime.init.5
/usr/local/go/src/runtime/proc.go:242 +0x35

goroutine 3 [GC sweep wait]:
runtime.gopark(0x1181f08, 0x1bf2100, 0x140c, 0x1)
/usr/local/go/src/runtime/proc.go:304 +0xe0 fp=0xc00006f7a8 sp=0xc00006f788 pc=0x432820
runtime.goparkunlock(…)
/usr/local/go/src/runtime/proc.go:310
runtime.bgsweep(0xc000096000)
/usr/local/go/src/runtime/mgcsweep.go:89 +0x131 fp=0xc00006f7d8 sp=0xc00006f7a8 pc=0x423ee1
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc00006f7e0 sp=0xc00006f7d8 pc=0x460c71
created by runtime.gcenable
/usr/local/go/src/runtime/mgc.go:210 +0x5c

goroutine 4 [sleep]:
runtime.gopark(0x1181f08, 0x1bf3f20, 0x401313, 0x2)
/usr/local/go/src/runtime/proc.go:304 +0xe0 fp=0xc00006fef0 sp=0xc00006fed0 pc=0x432820
runtime.goparkunlock(…)
/usr/local/go/src/runtime/proc.go:310
runtime.scavengeSleep(0x61a80000, 0x1bf3f01)
/usr/local/go/src/runtime/mgcscavenge.go:276 +0xee fp=0xc00006ff40 sp=0xc00006fef0 pc=0x4233be
runtime.bgscavenge(0xc000096000)
/usr/local/go/src/runtime/mgcscavenge.go:388 +0x2f8 fp=0xc00006ffd8 sp=0xc00006ff40 pc=0x4236e8
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc00006ffe0 sp=0xc00006ffd8 pc=0x460c71
created by runtime.gcenable
/usr/local/go/src/runtime/mgc.go:211 +0x7e

goroutine 18 [finalizer wait]:
runtime.gopark(0x1181f08, 0x1c13538, 0x1410, 0x1)
/usr/local/go/src/runtime/proc.go:304 +0xe0 fp=0xc00006e758 sp=0xc00006e738 pc=0x432820
runtime.goparkunlock(…)
/usr/local/go/src/runtime/proc.go:310
runtime.runfinq()
/usr/local/go/src/runtime/mfinal.go:175 +0xa3 fp=0xc00006e7e0 sp=0xc00006e758 pc=0x419d43
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc00006e7e8 sp=0xc00006e7e0 pc=0x460c71
created by runtime.createfing
/usr/local/go/src/runtime/mfinal.go:156 +0x61

goroutine 19 [syscall]:
runtime.notetsleepg(0x1bf98a0, 0x49a02a3c0, 0x0)
/usr/local/go/src/runtime/lock_futex.go:227 +0x34 fp=0xc00006a760 sp=0xc00006a730 pc=0x40c504
runtime.timerproc(0x1bf9880)
/usr/local/go/src/runtime/time.go:311 +0x2f1 fp=0xc00006a7d8 sp=0xc00006a760 pc=0x44f971
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc00006a7e0 sp=0xc00006a7d8 pc=0x460c71
created by runtime.(*timersBucket).addtimerLocked
/usr/local/go/src/runtime/time.go:169 +0x10e

goroutine 6 [GC worker (idle)]:
runtime.gopark(0x1181da0, 0xc0000c7320, 0x401418, 0x0)
/usr/local/go/src/runtime/proc.go:304 +0xe0 fp=0xc00006af60 sp=0xc00006af40 pc=0x432820
runtime.gcBgMarkWorker(0xc000046f00)
/usr/local/go/src/runtime/mgc.go:1846 +0xff fp=0xc00006afd8 sp=0xc00006af60 pc=0x41d74f
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc00006afe0 sp=0xc00006afd8 pc=0x460c71
created by runtime.gcBgMarkStartWorkers
/usr/local/go/src/runtime/mgc.go:1794 +0x77

goroutine 39 [GC worker (idle)]:
runtime.gopark(0x1181da0, 0xc0000c7310, 0x401418, 0x0)
/usr/local/go/src/runtime/proc.go:304 +0xe0 fp=0xc00019c760 sp=0xc00019c740 pc=0x432820
runtime.gcBgMarkWorker(0xc000040000)
/usr/local/go/src/runtime/mgc.go:1846 +0xff fp=0xc00019c7d8 sp=0xc00019c760 pc=0x41d74f
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc00019c7e0 sp=0xc00019c7d8 pc=0x460c71
created by runtime.gcBgMarkStartWorkers
/usr/local/go/src/runtime/mgc.go:1794 +0x77

goroutine 5 [syscall]:
runtime.notetsleepg(0x1bf9820, 0x4a816118f, 0x1)
/usr/local/go/src/runtime/lock_futex.go:227 +0x34 fp=0xc00019cf60 sp=0xc00019cf30 pc=0x40c504
runtime.timerproc(0x1bf9800)
/usr/local/go/src/runtime/time.go:311 +0x2f1 fp=0xc00019cfd8 sp=0xc00019cf60 pc=0x44f971
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc00019cfe0 sp=0xc00019cfd8 pc=0x460c71
created by runtime.(*timersBucket).addtimerLocked
/usr/local/go/src/runtime/time.go:169 +0x10e

goroutine 38 [sleep]:
runtime.gopark(0x1181f08, 0x1bf9800, 0x1313, 0x2)
/usr/local/go/src/runtime/proc.go:304 +0xe0 fp=0xc00006b748 sp=0xc00006b728 pc=0x432820
runtime.goparkunlock(…)
/usr/local/go/src/runtime/proc.go:310
time.Sleep(0x8bb2c97000)
/usr/local/go/src/runtime/time.go:105 +0x157 fp=0xc00006b798 sp=0xc00006b748 pc=0x44ee07
github.com/spacemonkeygo/monkit/v3.(*ticker).run(0x1bf3ae0)
/go/pkg/mod/github.com/spacemonkeygo/monkit/v3@v3.0.6-0.20200327233048-76480079049a/meter.go:203 +0x6a fp=0xc00006b7d8 sp=0xc00006b798 pc=0x910a9a
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc00006b7e0 sp=0xc00006b7d8 pc=0x460c71
created by github.com/spacemonkeygo/monkit/v3.(*ticker).register
/go/pkg/mod/github.com/spacemonkeygo/monkit/v3@v3.0.6-0.20200327233048-76480079049a/meter.go:195 +0x131

goroutine 23 [syscall]:
runtime.notetsleepg(0x1c13cc0, 0xffffffffffffffff, 0x0)
/usr/local/go/src/runtime/lock_futex.go:227 +0x34 fp=0xc00006bf98 sp=0xc00006bf68 pc=0x40c504
os/signal.signal_recv(0x0)
/usr/local/go/src/runtime/sigqueue.go:147 +0x9c fp=0xc00006bfc0 sp=0xc00006bf98 pc=0x44703c
os/signal.loop()
/usr/local/go/src/os/signal/signal_unix.go:23 +0x22 fp=0xc00006bfe0 sp=0xc00006bfc0 pc=0xa06682
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc00006bfe8 sp=0xc00006bfe0 pc=0x460c71
created by os/signal.init.0
/usr/local/go/src/os/signal/signal_unix.go:29 +0x41

[cdhowie]

@qinhang3

runtime: pointer 0xc0002b7ff0 to unused region of span span.base()=0xc0002b6000 span.limit=0xc0002b7f80 span.state=1
fatal error: found bad pointer in Go heap (incorrect use of unsafe or cgo?)

This is a strong indicator of memory corruption. I would highly recommend making a memtest CD or USB stick on a different system and rebooting this system using that CD/stick. Let memtest run for at least several hours, and even a full 24 hours if you are able. If it finds anything (there will be very clear red lines on the screen) then at least one test failed and your RAM returned something other than what was stored there.

Here’s an example of what it will look like while it’s running, if there are no errors:

If it finds any bad memory cells, then it will display those in red:

Any error indicates that you should immediately stop using this system until you have replaced the faulty memory; it is highly likely that data/filesystem damage has already occurred if the memory is faulty, and everything on the system would be suspect.

[qinhang3]

Thanks for your recommend.
I am going to run memtest immediately…
Is there any bad will cause if I stop my node for 24 hours?

[kevink]

@qinhang3

Is there any bad will cause if I stop my node for 24 hours?

no, you’ll be fine…

[SGC]

pretty sure you should be able to run a pretty decent mem test without shutting down the OS

also maybe set your machine to do periodic reboots… non ecc memory can be a bit temperamental over extended up times, doesn’t have to mean the RAM is bad… it might indicate that other systems are not running correctly… basically anything below… which could be PSU ( pretty common component to get … by else your RAM slots might not connect well enough… some of my RAM slots or RAM are kinda like that… haven’t really figured out why… most likely because of dirty slots… if i touch them the wrong way they go all wonky…

i really should get that fixed, but ain’t nobody got time for that lol

your mobo and ram might also be incompatible… not because this is usually a problem… but in rare cases that can be so… we are often cheap when finding components… and often it will bite us in the bitehind.

usually it’s either unstable psu power output or bad dimm contact… atleast 8 out of like 10 times… RAM are usually pretty reliable.

also turn off quick reboot… it does take a little longer to reboot… but those 10 minutes of checking the ram is well spent to be sure they worked correctly before you boot.

could also be that 1.3.3 is no longer opeartional… 1.6.2 or whatever is out soon… so it’s like 3 generations or older… which in the past has been pretty close to the line of when it goes into the invalid version range. duno if that actually shuts down the node in v3 tho

[cdhowie]

@SGC

pretty sure you should be able to run a pretty decent mem test without shutting down the OS

We don’t want “pretty decent,” we want exhaustive and thorough.

@SGC

could also be that 1.3.3 is no longer opeartional

“Found bad pointer in Go heap” has nothing whatsoever to do with the version of anything. It means one of three things:

1. Unsafe Go code or external (C/C++) code corrupted the Go heap.
2. There is a bug in the Go compiler that caused the emitted code to have a bug that corrupted the Go heap.
3. System memory was corrupted.

The most likely explanation for this to spontaneously start happening on an old version of storagenode in the absence of similar reports from others is number 3.

[SGC]

i usually just pull the ram out and remount them, most often it’s the connection in my experience.

[Alexey]

@SGC

i usually just pull the ram out and remount them

Also worth to try to use a eraser. Just polish the contacts, but be careful against static electricity, and then insert the RAM module back

[qinhang3]

Thank you guys.
I’m just finished the ramtest for about 12 hours, everything is fine now.

In fact. I’m pretty the memory maybe cause the crash.Because before Storage Node crashed, I had updated the memory from 32G（16G * 2 3200Mhz） to 64G（32G * 2 3000Mhz）.
Also I realized that I haven’t change the bios XMP config after update the hardware, so the new memory running at 3200Mhz yesterday, that maybe cause the crash.

[kevink]

@qinhang3

so the new memory running at 3200Mhz yesterday

That might be possible. Memory can be tricky sometimes. Let’s hope that’s what it was.

Some nice rig with i9-9900k and 64GB RAM

[SGC]

might not want to run xmp for stability reasons… but if it’s stable with it on… then sure… give 'er full steam mr murdock…

and remember to wave goodbye to your warranty, ofc its doubtful they can catch it… so if you end up with a warranty issue just remember to turn it off and if they ask then act ignorant tell them they are ddr4 ram or something like that.
then its usually not a problem