Node keeps crashing after a few hours

Node Operators troubleshooting
,
1

Hey :smiley:, I recently added a few new nodes on my Windows System and now they keep mysteriously crashing after a few hours. For the first week I had no problems at all but now after a few hours they run into an error I can only fix by removing the container and adding it again. When the error occurs, the container keeps running and the node port is still reachable. The Dashboard on 127.0.0.1:14002 is no longer working and the ram fills up to 100%. The docker log is full of errors I cannot understand.
Does anyone has experienced this behaviour before? What could be the cause of it?
Thanks in advance!

Here a few lines of the errors:

[syscall, 2 minutes]:\nsyscall.Syscall(0x107, 0xffffffffffffff9c, 0xc045796b40, 0x0)\n\t/usr/local/go/src/syscall/asm_linux_amd64.s:20 +0x5\nsyscall.unlinkat(0xf22840, {0xc03ff40e10, 0x0}, 0x0)\n\t/usr/local/go/src/syscall/zsyscall_linux_amd64.go:127 +0xd4\nsyscall.Unlink(ā€¦)\n\t/usr/local/go/src/syscall/syscall_linux.go:185\nos.Remove.func1(ā€¦)\n\t/usr/local/go/src/os/file_unix.go:302\nos.ignoringEINTR(ā€¦)\n\t/usr/local/go/src/os/file_posix.go:246\nos.Remove({0xc03ff40e10, 0x0})\n\t/usr/local/go/src/os/file_unix.go:301 +0x45\nstorj.io/storj/storage/filestore.(*blobWriter).Cancel(0xc0343dd7a0, {0x13cd368, 0xc03ca508c0})\n\t/go/src/storj.io/storj/storage/filestore/blob.go:102 +0x1fb\nstorj.io/storj/storagenode/pieces.(*Writer).Cancel(0xc01bd9ff80, {0x13cd368, 0xc01d12d7c0})\n\t/go/src/storj.io/storj/storagenode/pieces/readwrite.go:209

2

Hi @alpha4701,

Please show more of the log surrounding the [syscall, 2 minutes] error. That will give us a better idea of what the cause may be. You can post the log between 3 backticks:

```
Log
```
3

Hi @jeremy here are the surroundings. The lines repeat over and over again in a similar way.

go:47 +0x92\n\ngoroutine 621847 [select]:\nstorj.io/drpc/drpcmigrate.(*ListenMux).routeConn(0xc00029ef50, {0x13df390, 0xc0ce284280})\n\t/go/pkg/mod/storj.io/drpc@v0.0.32/drpcmigrate/mux.go:161 +0x356\ncreated by storj.io/drpc/drpcmigrate.(*ListenMux).monitorBase\n\t/go/pkg/mod/storj.io/drpc@v0.0.32/drpcmigrate/mux.go:121 +0x29\n\ngoroutine 404938 [syscall, 2 minutes]:\nsyscall.Syscall(0x107, 0xffffffffffffff9c, 0xc00063b980, 0x0)\n\t/usr/local/go/src/syscall/asm_linux_amd64.s:20 +0x5\nsyscall.unlinkat(0xf22840, {0xc016c92ae0, 0x0}, 0x0)\n\t/usr/local/go/src/syscall/zsyscall_linux_amd64.go:127 +0xd4\nsyscall.Unlink(...)\n\t/usr/local/go/src/syscall/syscall_linux.go:185\nos.Remove.func1(...)\n\t/usr/local/go/src/os/file_unix.go:302\nos.ignoringEINTR(...)\n\t/usr/local/go/src/os/file_posix.go:246\nos.Remove({0xc016c92ae0, 0x0})\n\t/usr/local/go/src/os/file_unix.go:301 +0x45\nstorj.io/storj/storage/filestore.(*blobWriter).Cancel(0xc01bb67b00, {0x13cd368, 0xc0ce3adc20})\n\t/go/src/storj.io/storj/storage/filestore/blob.go:102 +0x1fb\nstorj.io/storj/storagenode/pieces.(*Writer).Cancel(0xc013dc4930, {0x13cd368, 0xc0523b03c0})\n\t/go/src/storj.io/storj/storagenode/pieces/readwrite.go:209 +0x1a8\nstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload.func4()\n\t/go/src/storj.io/storj/storagenode/piecestore/endpoint.go:310 +0x46\nstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload(0xc00018c000, {0x13dabf0, 0xc050d889b0})\n\t/go/src/storj.io/storj/storagenode/piecestore/endpoint.go:323 +0x1110\nstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1({0x102bca0, 0xc00018c000}, {0xc0521da5a0, 0x1d}, {0xfa1a60, 0xc04345c160}, {0x203010, 0x203010})\n\t/go/pkg/mod/storj.io/common@v0.0.0-20221024150824-a2a5c611dacf/pb/piecestore2_drpc.pb.go:220 +0xb0\nstorj.io/drpc/drpcmux.(*Mux).HandleRPC(0xc0006c8a80, {0x13d6220, 0xc04345c160}, {0xc0521da5a0, 0x1d})\n\t/go/pkg/mod/storj.io/drpc@v0.0.32/drpcmux/handle_rpc.go:33 +0x22d\nstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC(0xc00011e3c0, {0x13d61a0, 0xc04345c100}, {0xc0521da5a0, 0x1d})\n\t/go/pkg/mod/storj.io/common@v0.0.0-20221024150824-a2a5c611dacf/rpc/rpctracing/handler.go:61 +0x30b\nstorj.io/common/experiment.(*Handler).HandleRPC(0xc00011c650, {0x13d62a0, 0xc05ef93200}, {0xc0521da5a0, 0x1d})\n\t/go/pkg/mod/storj.io/common@v0.0.0-20221024150824-a2a5c611dacf/experiment/import.go:42 +0x170\nstorj.io/drpc/drpcserver.(*Server).handleRPC(0xc0617fcfc0, 0x13ccf40, {0xc0521da5a0, 0x1b872b8})\n\t/go/pkg/mod/storj.io/drpc@v0.0.32/drpcserver/server.go:122 +0x42\nstorj.io/drpc/drpcserver.(*Server).ServeOne(0xc00018a0e0, {0x13cd640, 0xc00053a2a0}, {0x7f4dd171ff20, 0xc034769880})\n\t/go/pkg/mod/storj.io/drpc@v0.0.32/drpcserver/server.go:66 +0x1f3\nstorj.io/drpc/drpcserver.(*Server).Serve.func2({0x13cd640, 0xc00053a2a0})\n\t/go/pkg/mod/storj.io/drpc@v0.0.32/drpcserver/server.go:112 +0x58\nstorj.io/drpc/drpcctx.(*Tracker).track(0xc00053a2a0, 0xc050d88200)\n\t/go/pkg/mod/storj.io/drpc@v0.0.32/drpcctx/transport.go:52 +0x2b\ncreated by storj.io/drpc/drpcctx.(*Tracker).Run\n\t/go/pkg/mod/storj.io/drpc@v0.0.32/drpcctx/transport.go:47 +0x92\n\ngoroutine 404941 [syscall, 2 minutes]:\nsyscall.Syscall(0x107, 0xffffffffffffff9c, 0xc059ef7800, 0x0)\n\t/usr/local/go/src/syscall/asm_linux_amd64.s:20 +0x5\nsyscall.unlinkat(0xf22840, {0xc01516b410, 0x0}, 0x0)\n\t/usr/local/go/src/syscall/zsyscall_linux_amd64.go:127 +0xd4\nsyscall.Unlink(...)\n\t/usr/local/go/src/syscall/syscall_linux.go:185\nos.Remove.func1(...)\n\t/usr/local/go/src/os/file_unix.go:302\nos.ignoringEINTR(...)\n\t/usr/local/go/src/os/file_posix.go:246\nos.Remove({0xc01516b410, 0x0})\n\t/usr/local/go/src/os/file_unix.go:301 +0x45\nstorj.io/storj/storage/filestore.(*blobWriter).Cancel(0xc01b64af00, {0x13cd368, 0xc035d5c3c0})\n\t/go/src/storj.io/storj/storage/filestore/blob.go:102 +0x1fb\nstorj.io/storj/storagenode/pieces.(*Writer).Cancel(0xc05f69f030, {0x13cd368, 0xc00a727680})\n\t/go/src/storj.io/storj/storagenode/pieces/readwrite.go:209 +0x1a8\nstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload.func4()\n\t/go/src/storj.io/storj/storagenode/piecestore/endpoint.go:310 +0x46\nstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload(0xc00018c000, {0x13dabf0, 0xc051e7f1a0})\n\t/go/src/storj.io/storj/storagenode/piecestore/endpoint.go:323 +0x1110\nstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1({0x102bca0, 0xc00018c000}, {0xc00b1b0b60, 0x1d}, {0xfa1a60, 0xc05b0dfde0}, {0x203016, 0x203016})\n\t/go/pkg/mod/storj.io/common@v0.0.0-20221024150824-a2a5c611dacf/pb/piecestore2_drpc.pb.go:220 +0xb0\nstorj.io/drpc/drpcmux.(*Mux).HandleRPC(0xc0006c8a80, {0x13d6220, 0xc05b0dfde0}, {0xc00b1b0b60, 0x1d})\n\t/go/pkg/mod/storj.io/drpc@v0.0.32/drpcmux/handle_rpc.go:33 +0x22d\nstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC(0xc00011e3c0, {0x13d61a0, 0xc05b0dfda0}, {0xc00b1b0b60, 0x1d})\n\t/go/pkg/mod/storj.io/common@v0.0.0-20221024150824-a2a5c611dacf/rpc/rpctracing/handler.go:61 +0x30b\nstorj.io/common/experiment.(*Handler).HandleRPC(0xc00011c650, {0x13d62a0, 0xc05a634fc0}, {0xc00b1b0b60, 0x1d})\n\t/go/pkg/mod/storj.io/common@v0.0.0-20221024150824-a2a5c611dacf/experiment/import.go:42 +0x170\nstorj.io/drpc/drpcserver.(*Server).handleRPC(0xc0617fd180, 0x13ccf40, {0xc00b1b0b60, 0x1b872b8})\n\t/go/pkg/mod/storj.io/drpc@v0.0.32/drpcserver/server.go:122 +0x42\nstorj.io/drpc/drpcserver.(*Server).ServeOne(0xc00018a0e0, {0x13cd640, 0xc00053a2a0}, {0x7f4dd171ff20, 0xc034769c00})\n\t/go/pkg/mod/storj.io/drpc@v0.0.32/drpcserver/server.go:66 +0x1f3\nstorj.io/drpc/drpcserver.(*Server).Serve.func2({0x13cd640, 0xc00053a2a0})\n\t/go/pkg/mod/storj.io/drpc@v0.0.32/drpcserver/server.go:112 +0x58\nstorj.io/drpc/drpcctx.(*Tracker).track(0xc00053a2a0, 0xc050d89570)\n\t/go/pkg/mod/storj.io/drpc@v0.0.32/drpcctx/transport.go:52 +0x2b\ncreated by storj.io/drpc/drpcctx.(*Tracker).Run\n\t/go/pkg/mod/storj.io/drpc@v0.0.32/drpcctx/transport.go:47 +0x92\n\ngoroutine 404350 [syscall, 2 minutes]:\nsyscall.Syscall(0x107, 0xffffffffffffff9c, 0xc0ce2f03f0, 0x0)\n\t/usr/local/go/src/syscall/asm_linux_amd64.s:20 +0x5\nsyscall.unlinkat(0xf22840, {0xc0340b7920, 0x0}, 0x0)\n\t/usr/local/go/src/syscall/zsyscall_linux_amd64.go:127 +0xd4\nsyscall.Unlink(...)\n\t/usr/local/go/src/syscall/syscall_linux.go:185\nos.Remove.func1(...)\n\t/usr/local/go/src/os/file_unix.go:302\nos.ignoringEINTR(...)\n\t/usr/local/go/src/os/file_posix.go:246\nos.Remove({0xc0340b7920, 0x0})\n\t/usr/local/go/src/os/file_unix.go:301 +0x45\nstorj.io/storj/storage/filestore.(*blobWriter).Cancel(0xc009c7d4a0, {0x13cd368, 0xc0137d7680})\n\t/go/src/storj.io/storj/storage/filestore/blob.go:102 +0x1fb\nstorj.io/storj/storagenode/pieces.(*Writer).Cancel(0xc004b68c40, {0x13cd368, 0xc03688f9a0})\n\t/go/src/storj.io/storj/storagenode/pieces/readwrite.go:209 +0x1a8\nstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload.func4()\n\t/go/src/storj.io/storj/storagenode/piecestore/endpoint.go:310 +0x46\nstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload(0xc00018c000, {0x13dabf0, 0xc051e7f670})\n\t/go/src/storj.io/storj/storagenode/piecestore/endpoint.go:323 +0x1110\nstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1({0x102bca0, 0xc00018c000}, {0xc0064e8260, 0x1d}, {0xfa1a60, 0xc02a99cca0}, {0x20300a, 0x20300a})\n\t/go/pkg/mod/storj.io/common@v0.0.0-20221024150824-a2a5c611dacf/pb/piecestore2_drpc.pb
.go:220 +0xb0\nstorj.io/drpc/drpcmux.(*Mux).HandleRPC(0xc0006c8a80, {0x13d6220, 0xc02a99cca0}, {0xc0064e8260, 0x1d})\n\t/go/pkg/mod/storj.io/drpc@v0.0.32/drpcmux/handle_rpc.go:33 +0x22d\nstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC(0xc00011e3c0, {0x13d61a0, 0xc02a99c9e0}, {0xc0064e8260, 0x1d})\n\t/go/pkg/mod/storj.io/common@v0.0.0-20221024150824-a2a5c611dacf/rpc/rpctracing/handler.go:61 +0x30b\nstorj.io/common/experiment.(*Handler).HandleRPC(0xc00011c650, {0x13d62a0, 0xc05a635200}, {0xc0064e8260, 0x1d})\n\t/go/pkg/mod/storj.io/common@v0.0.0-20221024150824-a2a5c611dacf/experiment/import.go:42 +0x170\nstorj.io/drpc/drpcserver.(*Server).handleRPC(0xc04c42a000, 0x13ccf40, {0xc0064e8260, 0x1b872b8})\n\t/go/pkg/mod/storj.io/drpc@v0.0.32/drpcserver/server.go:122 +0x42\nstorj.io/drpc/drpcserver.(*Server).ServeOne(0xc00018a0e0, {0x13cd640, 0xc00053a2a0}, {0x7f4dd171ff20, 0xc061255c00})\n\t/go/pkg/mod/storj.io/drpc@v0.0.32/drpcserver/server.go:66 +0x1f3\nstorj.io/drpc/drpcserver.(*Server).Serve.func2({0x13cd640, 0xc00053a2a0})\n\t/go/pkg/mod/storj.io/drpc@v0.0.32/drpcserver/server.go:112 +0x58\nstorj.io/drpc/drpcctx.(*Tracker).track(0xc00053a2a0, 0xc050d895a0)\n\t/go/pkg/mod/storj.io/drpc@v0.0.32/drpcctx/transport.go:52 +0x2b\ncreated by storj.io/drpc/drpcctx.(*Tracker).Run\n\t/go/pkg/mod/storj.io/drpc@v0.0.32/drpcctx/transport.go:47 +0x92\n\ngoroutine 405071 [syscall, 2 minutes]:\nsyscall.Syscall(0x107, 0xffffffffffffff9c, 0xc045796c60, 0x0)\n\t/usr/local/go/src/syscall/asm_linux_amd64.s:20 +0x5\nsyscall.unlinkat(0xf22840, {0xc016c92630, 0x0}, 0x0)\n\t/usr/local/go/src/syscall/zsyscall_linux_amd64.go:127 +0xd4\nsyscall.Unlink(...)\n\t/usr/local/go/src/syscall/syscall_linux.go:185\nos.Remove.func1(...)\n\t/usr/local/go/src/os/file_unix.go:302\nos.ignoringEINTR(...)\n\t/usr/local/go/src/os/file_posix.go:246\nos.Remove({0xc016c92630, 0x0})\n\t/usr/local/go/src/os/file_unix.go:301 +0x45\nstorj.io/storj/storage/filestore.(*blobWriter).Cancel(0xc01fdc8300, {0x13cd368, 0xc03ca51040})\n\t/go/src/storj.io/storj/storage/filestore/blob.go:102 +0x1fb\nstorj.io/storj/storagenode/pieces.(*Writer).Cancel(0xc00bb63420, {0x13cd368, 0xc0523b1900})\n\t/go/src/storj.io/storj/storagenode/pieces/readwrite.go:209 +0x1a8\nstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload.func4()\n\t/go/src/storj.io/storj/storagenode/piecestore/endpoint.go:310 +0x46\nstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload(0xc00018c000, {0x13dabf0, 0xc050d89f60})\n\t/go/src/storj.io/storj/storagenode/piecestore/endpoint.go:323 +0x1110\nstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1({0x102bca0, 0xc00018c000}, {0xc01bdf53a0, 0x1d}, {0xfa1a60, 0xc04345dc80}, {0x203010, 0x203010})\n\t/go/pkg/mod/storj.io/common@v0.0.0-20221024150824-a2a5c611dacf/pb/piecestore2_drpc.pb.go:220 +0xb0\nstorj.io/drpc/drpcmux.(*Mux).HandleRPC(0xc0006c8a80, {0x13d6220, 0xc04345dc80}, {0xc01bdf53a0, 0x1d})\n\t/go/pkg/mod/storj.io/drpc@v0.0.32/drpcmux/handle_rpc.go:33 +0x22d\nstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC(0xc00011e3c0, {0x13d61a0, 0xc04345dc40}, {0xc01bdf53a0, 0x1d})\n\t/go/pkg/mod/storj.io/common@v0.0.0-20221024150824-a2a5c611dacf/rpc/rpctracing/handler.go:61 +0x30b\nstorj.io/common/experiment.(*Handler).HandleRPC(0xc00011c650, {0x13d62a0, 0xc05ef93440}, {0xc01bdf53a0, 0x1d})\n\t/go/pkg/mod/storj.io/common@v0.0.0-20221024150824-a2a5c611dacf/experiment/import.go:42 +0x170\nstorj.io/drpc/drpcserver.(*Server).handleRPC(0xc061872fc0, 0x13ccf40, {0xc01bdf53a0, 0x1b872b8})\n\t/go/pkg/mod/storj.io/drpc@v0.0.32/drpcserver/server.go:122 +0x42\nstorj.io/drpc/drpcserver.(*Server).ServeOne(0xc00018a0e0, {0x13cd640, 0xc00053a2a0}, {0x7f4dd171ff20, 0xc056dec000})\n\t/go/pkg/mod/storj.io/drpc@v0.0.32/drpcserver/server.go:66 +0x1f3\nstorj.io/drpc/drpcserver.(*Server).Serve.func2({0x13cd640, 0xc00053a2a0})\n\t/go/pkg/mod/storj.io/drpc@v0.0.32/drpcserver/server.go:112 +0x58\nstorj.io/drpc/drpcctx.(*Tracker).track(0xc00053a2a0, 0xc0107efb70)\n\t/go/pkg/mod/storj.io/drpc@v0.0.32/drpcctx/transport.go:52 +0x2b\ncreated by storj.io/drpc/drpcctx.(*Tracker).Run\n\t/go/pkg/mod/storj.io/drpc@v0.0.32/drpcctx/transport.
4

Thank you. What does the log look like before the first occurrence of this type of error? Does ā€œpanicā€ appear anywhere in your logs?

5

ram will fill up if the storagenodes cpu time is to limited (overloaded cpu) or if the storage cannot keep upā€¦

6

ā€œpanicā€ does not appear anywhere.
This is the first Error. After the it occurs there appear more and more error messages until there are only errors.

2022-11-21T14:02:21.687Z	INFO	piecedeleter	delete piece sent to trash	{"Process": "storagenode", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Piece ID": "E2HNU6NCIDJ2W2FHKEDCPLBGUWUIQMFLLJRBBUHXROYD23AD2HXQ"}
2022-11-21T14:02:22.090Z	INFO	piecedeleter	delete piece sent to trash	{"Process": "storagenode", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Piece ID": "LETI3C4RDHX6CPZTCK7ABVBL24CS4TQXU6AXJJA7ZMHS6UUXVJMQ"}
2022-11-21T14:02:22.190Z	INFO	piecestore	upload started	{"Process": "storagenode", "Piece ID": "THH443NUHGJIZ6Q4COYHCJ4DH3WFV7JWU7TGZF4EB4Q3FARSFNSA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Available Space": 1866150438912}
2022-11-21T14:02:22.604Z	INFO	piecestore	upload started	{"Process": "storagenode", "Piece ID": "AVHOF55H42NWE2Y7FSLG6MLSNKK2I7V7YROSURDGZYM4I2E4J2JA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Available Space": 1866150438912}
2022-11-21T14:02:22.603Z	INFO	piecestore	upload started	{"Process": "storagenode", "Piece ID": "DZQCU4VAE7MXUX3GX4SNISKHIWSTAQZAR5474YNETAE7Q5HF3JEQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT", "Available Space": 1866150438912}
2022-11-21T14:02:22.610Z	INFO	piecestore	upload started	{"Process": "storagenode", "Piece ID": "BHHVPUFZVKNA6LCWEROQC5NZLHMNFN5AKVOG27GRBOHYHOHENNJA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Available Space": 1866150438912}
2022-11-21T14:02:22.603Z	INFO	piecestore	upload started	{"Process": "storagenode", "Piece ID": "LHNYBHV63FO5I7CBPA53FMT3XNIWOVYTVHZ77BAX2X4IGTM2ACGA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Available Space": 1866150438912}
2022-11-21T14:02:22.892Z	INFO	piecestore	upload started	{"Process": "storagenode", "Piece ID": "EAMTGGK6ZACCTJ4MLDJPGKT4T5N4KSYRI5NRZWBOKUQVAOJDEPDA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT_REPAIR", "Available Space": 1866150438912}
2022-11-21T14:02:24.097Z	ERROR	piecedeleter	could not send delete piece to trash	{"Process": "storagenode", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Piece ID": "MT5L6WPIZEPFV7TUUOKKLEC5F7OCAOOVD5RWWOR7ARQGE52RW5QA", "error": "pieces error: filestore error: rename config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/mt/5l6wpizepfv7tuuokklec5f7ocaoovd5rwwor7arqge52rw5qa.sj1 config/storage/trash/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/mt/5l6wpizepfv7tuuokklec5f7ocaoovd5rwwor7arqge52rw5qa.sj1: input/output error", "errorVerbose": "pieces error: filestore error: rename config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/mt/5l6wpizepfv7tuuokklec5f7ocaoovd5rwwor7arqge52rw5qa.sj1 config/storage/trash/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/mt/5l6wpizepfv7tuuokklec5f7ocaoovd5rwwor7arqge52rw5qa.sj1: input/output error\n\tstorj.io/storj/storage/filestore.(*blobStore).Trash:140\n\tstorj.io/storj/storagenode/pieces.(*BlobsUsageCache).Trash:295\n\tstorj.io/storj/storagenode/pieces.(*Store).Trash:365\n\tstorj.io/storj/storagenode/pieces.(*Deleter).deleteOrTrash:185\n\tstorj.io/storj/storagenode/pieces.(*Deleter).work:135\n\tstorj.io/storj/storagenode/pieces.(*Deleter).Run.func1:72\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75"}
...
2022-11-21T14:03:54.869Z	ERROR	piecestore	upload failed	{"Process": "storagenode", "Piece ID": "DXGSGR4GO25HMYW3XNUTUAUFVV52K26TCN2JP3MPFADICFJZF6SQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "error": "order: grace period passed for order limit", "errorVerbose": "order: grace period passed for order limit\n\tstorj.io/storj/storagenode/orders.(*FileStore).BeginEnqueue:86\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).beginSaveOrder:705\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:321\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:220\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:61\n\tstorj.io/common/experiment.(*Handler).HandleRPC:42\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:122\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:112\n\tstorj.io/drpc/drpcctx.(*Tracker).track:52", "Size": 0}
2022-11-21T14:03:54.872Z	ERROR	piecestore	upload failed	{"Process": "storagenode", "Piece ID": "XSBTWFWIZ5NJIURUG4IRXL3IZHLHCNIECBK6HGXFMKKELT3J6J6A", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "error": "order: grace period passed for order limit", "errorVerbose": "order: grace period passed for order limit\n\tstorj.io/storj/storagenode/orders.(*FileStore).BeginEnqueue:86\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).beginSaveOrder:705\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:321\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:220\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:61\n\tstorj.io/common/experiment.(*Handler).HandleRPC:42\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:122\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:112\n\tstorj.io/drpc/drpcctx.(*Tracker).track:52", "Size": 0}
2022-11-21T14:03:54.919Z	ERROR	piecestore	upload failed	{"Process": "storagenode", "Piece ID": "NO7FHV4QES3LFRWPRJ6EEFSBHTJMTUO33ZO5ILEFPF4WV2TCADWA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "error": "order: grace period passed for order limit", "errorVerbose": "order: grace period passed for order limit\n\tstorj.io/storj/storagenode/orders.(*FileStore).BeginEnqueue:86\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).beginSaveOrder:705\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:321\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:220\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:61\n\tstorj.io/common/experiment.(*Handler).HandleRPC:42\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:122\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:112\n\tstorj.io/drpc/drpcctx.(*Tracker).track:52", "Size": 0}
2022-11-21T14:03:54.924Z	ERROR	piecestore	upload failed	{"Process": "storagenode", "Piece ID": "KSVIVQFYJV44YJ6QGEPKWKSFAJ5OIGC4BNKHREGX4M3RKYE3X73Q", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "error": "order: grace period passed for order limit", "errorVerbose": "order: grace period passed for order limit\n\tstorj.io/storj/storagenode/orders.(*FileStore).BeginEnqueue:86\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).beginSaveOrder:705\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:321\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:220\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:61\n\tstorj.io/common/experiment.(*Handler).HandleRPC:42\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:122\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:112\n\tstorj.io/drpc/drpcctx.(*Tracker).track:52", "Size": 0}
2022-11-21T14:03:54.926Z	ERROR	piecestore	upload failed	{"Process": "storagenode", "Piece ID": "SX55RO5FVD7KJQXYOYCBMZLGZG5SNQA5Z4AGIQLVNBUDP633SZLQ", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Action": "PUT", "error": "order: grace period passed for order limit", "errorVerbose": "order: grace period passed for order limit\n\tstorj.io/storj/storagenode/orders.(*FileStore).BeginEnqueue:86\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).beginSaveOrder:705\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:321\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:220\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:61\n\tstorj.io/common/experiment.(*Handler).HandleRPC:42\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:122\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:112\n\tstorj.io/drpc/drpcctx.(*Tracker).track:52", "Size": 0}
2022-11-21T14:03:54.942Z	ERROR	piecestore	upload failed	{"Process": "storagenode", "Piece ID": "K6FA2QRIOVL6VFUZDUQOSC6I4BKVUBHSK2TLJTWBF7UB26QVAOZA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "error": "order: grace period passed for order limit", "errorVerbose": "order: grace period passed for order limit\n\tstorj.io/storj/storagenode/orders.(*FileStore).BeginEnqueue:86\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).beginSaveOrder:705\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:321\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:220\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:61\n\tstorj.io/common/experiment.(*Handler).HandleRPC:42\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:122\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:112\n\tstorj.io/drpc/drpcctx.(*Tracker).track:52", "Size": 0}

After a while this results in this (as the error is way too long this is only the beginning):

2022-11-21T14:03:39.033Z	INFO	services	slow shutdown	{"Process": "storagenode", "stack": "goroutine 1211 [running]:\nstorj.io/storj/private/lifecycle.(*Group).logStackTrace.func1()\n\t/go/src/storj.io/storj/private/lifecycle/group.go:107 +0x85\nsync.(*Once).doSlow(0xc098afc000, 0xc0cdbe2600)\n\t/usr/local/go/src/sync/once.go:68 +0xd2\nsync.(*Once).Do(...)\n\t/usr/local/go/src/sync/once.go:59\nstorj.io/storj/private/lifecycle.(*Group).logStackTrace(0xc0007c46b8)\n\t/go/src/storj.io/storj/private/lifecycle/group.go:104 +0x45\nstorj.io/storj/private/lifecycle.(*Group).Run.func1({0x13ccf40, 0xc00053b770})\n\t/go/src/storj.io/storj/private/lifecycle/group.go:77 +0x2e5\nruntime/pprof.Do({0x13cd368, 0xc0004be5a0}, {{0xc0003c2280, 0x0, 0x0}}, 0xc0004e0cc0)\n\t/usr/local/go/src/runtime/pprof/runtime.go:40 +0xa3\ncreated by storj.io/storj/private/lifecycle.(*Group).Run\n\t/go/src/storj.io/storj/private/lifecycle/group.go:64 +0x630\n\ngoroutine 1 [semacquire, 249 minutes]:\nsync.runtime_Semacquire(0x0)\n\t/usr/local/go/src/runtime/sema.go:56 +0x25\nsync.(*WaitGroup).Wait(0xc0003b8620)\n\t/usr/local/go/src/sync/waitgroup.go:130 +0x71\ngolang.org/x/sync/errgroup.(*Group).Wait(0xc0004e0140)\n\t/go/pkg/mod/golang.org/x/sync@v0.0.0-20220819030929-7fc1605a5dde/errgroup/errgroup.go:53 +0x27\nstorj.io/storj/storagenode.(*Peer).Run(0xc0001c6000, {0x13cce98, 0xc0002929c0})\n\t/go/src/storj.io/storj/storagenode/peer.go:911 +0x49d\nmain.cmdRun(0xf314e0, {0xc000283080, 0x1ad6e80, 0x441cee})\n\t/go/src/storj.io/storj/cmd/storagenode/main.go:251 +0xe85\nstorj.io/private/process.cleanup.func1.4({0x13cd368, 0xc0004be1e0})\n\t/go/pkg/mod/storj.io/private@v0.0.0-20221011183246-586e5f48357a/process/exec_conf.go:377 +0x14f\nstorj.io/private/process.cleanup.func1(0x1ad6e80, {0xc0002be300, 0x0, 0xc})\n\t/go/pkg/mod/storj.io/private@v0.0.0-20221011183246-586e5f48357a/process/exec_conf.go:395 +0x1ca8\ngithub.com/spf13/cobra.(*Command).execute(0x1ad6e80, {0xc0002be240, 0xc, 0xc})\n\t/go/pkg/mod/github.com/spf13/cobra@v1.1.3/command.go:852 +0x60e\ngithub.com/spf13/cobra.(*Command).ExecuteC(0x1ad6c00)\n\t/go/pkg/mod/github.com/spf13/cobra@v1.1.3/command.go:960 +0x3ad\ngithub.com/spf13/cobra.(*Command).Execute(...)\n\t/go/pkg/mod/github.com/spf13/cobra@v1.1.3/command.go:897\nstorj.io/private/process.ExecWithCustomConfigAndLogger(0x1ad6c00, 0xd0, 0xedf360, 0xc0000001a0)\n\t/go/pkg/mod/storj.io/private@v0.0.0-20221011183246-586e5f48357a/process/exec_conf.go:92 +0x190\nmain.main()\n\t/go/src/storj.io/storj/cmd/storagenode/main.go:478 +0x30\n\ngoroutine 13 [select]:\nstorj.io/monkit-jaeger.(*UDPCollector).Run(0xc0002be3c0, {0x13cce98, 0xc000292940})\n\t/go/pkg/mod/storj.io/monkit-jaeger@v0.0.0-20220915074555-d100d7589f41/udp.go:160 +0x7ff\nstorj.io/private/process.cleanup.func1.2()\n\t/go/pkg/mod/storj.io/private@v0.0.0-20221011183246-586e5f48357a/process/exec_conf.go:328 +0x25\ngolang.org/x/sync/errgroup.(*Group).Go.func1()\n\t/go/pkg/mod/golang.org/x/sync@v0.0.0-20220819030929-7fc1605a5dde/errgroup/errgroup.go:75 +0x64\ncreated by golang.org/x/sync/errgroup.(*Group).Go\n\t/go/pkg/mod/golang.org/x/sync@v0.0.0-20220819030929-7fc1605a5dde/errgroup/errgroup.go:72 +0xa9\n\ngoroutine 15 [syscall, 249 minutes]:\nos/signal.signal_recv()\n\t/usr/local/go/src/runtime/sigqueue.go:169 +0x98\nos/signal.loop()\n\t/usr/local/go/src/os/signal/signal_unix.go:24 +0x19\ncreated by os/signal.Notify.func1.1\n\t/usr/local/go/src/os/signal/signal.go:151 +0x2c\n\ngoroutine 53 [chan receive, 249 minutes]:\nstorj.io/private/process.Ctx.func1()\n\t/go/pkg/mod/storj.io/private@v0.0.0-20221011183246-586e5f48357a/process/exec_conf.go:119 +0x3a\ncreated by storj.io/private/process.Ctx\n\t/go/pkg/mod/storj.io/private@v0.0.0-20221011183246-586e5f48357a/process/exec_conf.go:118 +0x2de\n\ngoroutine 71 [select, 249 minutes]:\ndatabase/sql.(*DB).connectionOpener(0xc00027e000, {0x13cce98, 0xc000292000})\n\t/usr/local/go/src/database/sql/sql.go:1196 +0x93\ncreated by database/sql.OpenDB\n\t/usr/local/go/src/database/sql/sql.go:794 +0x188\n\ngoroutine 312985 [semacquire, 32 minutes]:\nsync.runtime_SemacquireMutex(0xc0276c8e68, 0xaa, 0x18)\n\t/usr/local/go/src/runtime/sema.go:71 +0x25\nsync.(*Mutex).lockSlow(0xc000789a88)\n\t/usr/local/go/src/sync/mutex.go:138 +0x165\nsync.(*Mutex).Lock(...)\n\t/usr/local/go/src/sync/mutex.go:81\nstorj.io/storj/storagenode/orders.(*FileStore).BeginEnqueue.func1(0xc0a60d7070)\n\t/go/src/storj.io/storj/storagenode/orders/store.go:94 +0x94\nstorj.io/storj/storagenode/piecestore.(*Endpoint).beginSaveOrder.func1({0x13cd368, 0xc00543d2c0}, 0xc0116b9bd0)\n\t/go/src/storj.io/storj/storagenode/piecestore/endpoint.go:726 +0x106\nstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload(0xc00018c000, {0x13dabf0, 0xc0061f9780})\n\t/go/src/storj.io/storj/storagenode/piecestore/endpoint.go:443 +0x1fe0\nstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1({0x102bca0, 0xc00018c000}, {0xc021ea2f40, 0x1d}, {0xfa1a60, 0xc00a432c20}, {0x203002, 0x203002})\n\t/go/pkg/mod/storj.io/common@v0.0.0-20221024150824-a2a5c611dacf/pb/piecestore2_drpc.pb.go:220 +0xb0\nstorj.io/drpc/drpcmux.(*Mux).HandleRPC(0xc0006c8a80, {0x13d6220, 0xc00a432c20}, {0xc021ea2f40, 0x1d})\n\t/go/pkg/mod/storj.io/drpc@v0.0.32/drpcmux/handle_rpc.go:33 +0x22d\nstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC(0xc00011e3c0, {0x13d61a0, 0xc00a432ac0}, {0xc021ea2f40, 0x1d})\n\t/go/pkg/mod/storj.io/common@v0.0.0-20221024150824-a2a5c611dacf/rpc/rpctracing/handler.go:61 +0x30b\nstorj.io/common/experiment.(*Handler).HandleRPC(0xc00011c650, {0x13d62a0, 0xc00f1f4b40}, {0xc021ea2f40, 0x1d})\n\t/go/pkg/mod/storj.io/common@v0.0.0-20221024150824-a2a5c611dacf/experiment/import.go:42 +0x170\nstorj.io/drpc/drpcserver.(*Server).handleRPC(0xc033fc3180, 0x13ccf40, {0xc021ea2f40, 0x1b872b8})\n\t/go/pkg/mod/storj.io/drpc@v0.0.32/drpcserver/server.go:122 +0x42\nstorj.io/drpc/drpcserver.(*Server).ServeOne(0xc00018a0e0, {0x13cd640, 0xc00053a2a0}, {0x7f4dd171ff20, 0xc014647880})\n\t/go/pkg/mod/storj.io/drpc@v0.0.32/drpcserver/server.go:66 +0x1f3\nstorj.io/drpc/drpcserver.(*Server).Serve.func2({0x13cd640, 0xc00053a2a0})\n\t/go/pkg/mod/storj.io/drpc@v0.0.32/drpcserver/server.go:112 +0x58\nstorj.io/drpc/drpcctx.(*Tracker).track(0xc00053a2a0, 0xc01fd91f70)\n\t/go/pkg/mod/storj.io/drpc@v0.0.32/drpcctx/transport.go:52 +0x2b\ncreated by storj.io/drpc/drpcctx.(*Tracker).Run\n\t/go/pkg/mod/storj.io/drpc@v0.0.32/drpcctx/transport.go:47 +0x92\n\ngoroutine 248452 [semacquire, 52 minutes]:\nsync.runtime_SemacquireMutex(0xc006b80e68, 0xaa, 0x18)\n\t/usr/local/go/src/runtime/sema.go:71 +0x25\nsync.(*Mutex).lockSlow(0xc000789a88)\n\t/usr/local/go/src/sync/mutex.go:138 +0x165\nsync.(*Mutex).Lock(...)\n\t/usr/local/go/src/sync/mutex.go:81\nstorj.io/storj/storagenode/orders.(*FileStore).BeginEnqueue.func1(0xc01ab02d70)\n\t/go/src/storj.io/storj/storagenode/orders/store.go:94 +0x94\nstorj.io/storj/storagenode/piecestore.(*Endpoint).beginSaveOrder.func1({0x13cd368, 0xc0057e21e0}, 0xc02df36ff0)\n\t/go/src/storj.io/storj/storagenode/piecestore/endpoint.go:726 +0x106\nstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload(0xc00018c000, {0x13dabf0, 0xc014f709b0})\n\t/go/src/storj.io/storj/storagenode/piecestore/endpoint.go:443 +0x1fe0\nstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1({0x102bca0, 0xc00018c000}, {0xc01ba094e0, 0x1d}, {0xfa1a60, 0xc03d12d440}, {0x20300f, 0x20300f})\n\t/go/pkg/mod/storj.io/common@v0.0.0-20221024150824-a2a5c611dacf/pb/piecestore2_drpc.pb.go:220 +0xb0\nstorj.io/drpc/drpcmux.(*Mux).HandleRPC(0xc0006c8a80, {0x13d6220, 0xc03d12d440}, {0xc01ba094e0, 0x1d})\n\t/go/pkg/mod/storj.io/drpc@v0.0.32/drpcmux/handle_rpc.go:33 +0x22d\nstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC(0xc00011e3c0, {0x13d61a0, 0xc03d12d400}, {0xc01ba094e0, 0x1d})\n\t/go/pkg/mod/storj.io/common@v0.0.0-20221024150824-a2a5c611dacf/rpc/rpctracing/handler.go:61 +0x30b\nstorj.io/common/experiment.(*Handler).HandleRPC(0xc00011c650, {0x13d62a0, 0xc01682f680}, {0xc01ba094e0, 0x1d})\n\t/go/pkg/mod/storj.io/common@v0.0.0-20221024150824-a2a5c611dacf/experiment/import.go:42 +0x170\nstorj.io/drpc/drpcserver.(*Server).handleRPC(0xc01946b6c0, 0x13ccf40, {0xc01ba094e0, 0x1b872b8})\n\t/go/pkg/mod/storj.io/drpc@v0.0.32/drpcserver/server.go:122 +0x42\nstorj.io/drpc/drpcserver.(*Server).ServeOne(0xc00018a0e0, {0x13cd640, 0xc00053a2a0}, {0x7f4dd171ff20, 0xc039c3d880})\n\t/go/pkg/mod/storj.io/drpc@v0.0.32/drpcserver/server.go:66 +0x1f3\nstorj.io/drpc/drpcserver.(*Server).Serve.func2({0x13cd640, 0xc00053a2a0})\n\t/go/pkg/mod/storj.io/drpc@v0.0.32/drpcserver/server.go:112 +0x58\nstorj.io/drpc/drpcctx.(*Tracker).track(0xc00053a2a0, 0xc00d9f9210)\n\t/go/pkg/mod/storj.io/drpc@v0.0.32/drpcctx/transport.go:52 +0x2b\ncreated by storj.io/drpc/drpcctx.(*Tracker).Run\n\t/go/pkg/mod/storj.io/drpc@v0.0.32/drpcctx/transport.go:47 +0x92\n\ngoroutine 228 [select, 249 minutes]:\ndatabase/sql.(*DB).connectionOpener(0xc0003fc270, {0x13cce98, 0xc00020a280})\n\t/usr/local/go/src/database/sql/sql.go:1196 +0x93\ncreated by database/sql.OpenDB\n\t/usr/local/go/src/database/sql/sql.go:794 +0x188\n\ngoroutine 269 [select, 249 minutes]:\ndatabase/sql.(*DB).connectionOpener(0xc000372820, {0x13cce98, 0xc000137680})\n\t/usr/local/go/src/database/sql/sql.go:1196 +0x93\ncreated by database/sql.OpenDB\n\t/usr/local/go/src/database/sql/sql.go:794 +0x188\n\ngoroutine 226 [select, 249 minutes]:\ndatabase/sql.(*DB).connectionOpener(0xc0003fc000, {0x13cce98, 0xc00020a0c0})\n\t/usr/local/go/src/database/sql/sql.go:1196 +0x93\ncreated by database/sql.OpenDB\n\t/usr/local/go/src/database/sql/sql.go:794 +0x188\n\ngoroutine 964 [sleep, 10 minutes]:\ntime.Sleep(0x8bb2c97000)\n\t/usr/local/go/src/runtime/time.go:193 +0x12e\ngithub.com/spacemonkeygo/monkit/v3.(*ticker).run(0x1b53de0)\n\t/go/pkg/mod/github.com/spacemonkeygo/monkit/v3@v3.0.19/meter.go:203 +0x2c\ncreated by github.com/spacemonkeygo/monkit/v3.(*ticker).register\n\t/go/pkg/mod/github.com/spacemonkeygo/monkit/v3@v3.0.19/meter.go:195 +0x95\n\ngoroutine 363076 [syscall, 2 minutes]:\nsyscall.Syscall(0x107, 0xffffffffffffff9c, 0xc0cd9f40f0, 0x0)\n\t/usr/local/go/src/syscall/asm_linux_amd64.s:20 +0x5\nsyscall.unlinkat(0xf22840, {0xc00b572de0, 0x0}, 0x0)\n\t/usr/local/go/src/syscall/zsyscall_linux_amd64.go:127 +0xd4\nsyscall.Unlink(...)\n\t/usr/local/go/src/syscall/syscall_linux.go:185\nos.Remove.func1(...)\n\t/usr/local/go/src/os/file_unix.go:302\nos.ignoringEINTR(...)\n\t/usr/local/go/src/os/file_posix.go:246\nos.Remove({0xc00b572de0, 0x0})\n\t/usr/local/go/src/os/file_unix.go:301 +0x45\nstorj.io/storj/storage/filestore.(*blobWriter).Cancel(0xc01bc305a0, {0x13cd368, 0xc0cd9ebd60})\n\t/go/src/storj.io/storj/storage/filestore/blob.go:102 +0x1fb\nstorj.io/storj/storagenode/pieces.(*Writer).Cancel(0xc01be5e7e0, {0x13cd368, 0xc019958d20})\n\t/go/src/storj.io/storj/storagenode/pieces/readwrite.go:209 +0x1a8\nstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload.func4()\n\t/go/src/storj.io/storj/storagenode/piecestore/endpoint.go:310 +0x46\nstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload(0xc00018c000, {0x13dabf0, 0xc01bdd63f0})\n\t/go/src/storj.io/storj/storagenode/piecestore/endpoint.go:323 +0x1110\nstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1({0x102bca0, 0xc00018c000}, {0xc03789c8a0, 0x1d}, {0xfa1a60, 0xc031398540}, {0x20300c, 0x20300c})\n\t/go/pkg/mod/storj.io/common@v0.0.0-20221024150824-a2a5c611dacf/pb/piecestore2_drpc.pb.go:220 +0xb0\nstorj.io/drpc/drpcmux.(*Mux).HandleRPC(0xc0006c8a80, {0x13d6220, 0xc031398540}, {0xc03789c8a0, 0x1d})\n\t/go/pkg/mod/storj.io/drpc@v0.0.32/drpcmux/handle_rpc.go:33 +0x22d\nstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC(0xc00011e3c0, {0x13d61a0, 0xc031398500}, {0xc03789c8a0, 0x1d})\n\t/go/pkg/mod/storj.io/common@v0.0.0-20221024150824-a2a5c611dacf/rpc/rpctracing/handler.go:61 +0x30b\nstorj.io/common/experiment.(*Handler).HandleRPC(0xc00011c650, {0x13d62a0, 0xc03cc45d40}, {0xc03789c8a0, 0x1d})\n\t/go/pkg/mod/storj.io/common@v0.0.0-20221024150824-a2a5c611dacf/experiment/import.go:42 +0x170\nstorj.io/drpc/drpcserver.(*Server).handleRPC(0xc038100e00, 0x13ccf40, {0xc03789c8a0, 0x1b872b8})\n\t/go/pkg/mod/storj.io/drpc@v0.0.32/drpcserver/server.go:122 +0x42\nstorj.io/drpc/drpcserver.(*Server).ServeOne(0xc00018a0e0, {0x13cd640, 0xc00053a2a0}, {0x7f4dd171ff20, 0xc0371b7880})\n\t/go/pkg/mod/storj.io/drpc@v0.0.32/drpcserver/server.go:66 +0x1f3\nstorj.io/drpc/drpcserver.(*Server).Serve.func2({0x13cd640, 0xc00053a2a0})\n\t/go/pkg/mod/storj.io/drpc@v0.0.32/drpcserver/server.go:112 +0x58\nstorj.io/drpc/drpcctx.(*Tracker).track(0xc00053a2a0, 0xc0002006e0)\n\t/go/pkg/mod/storj.io/drpc@v0.0.32/drpcctx/transport.go:52 +0x2b\ncreated by storj.io/drpc/drpcctx.(*Tracker).Run\n\t/go/pkg/mod/storj.io/drpc@v0.0.32/drpcctx/transport.go:47 +0x92\n\ngoroutine 72 [select, 10 minutes]:\ndatabase/sql.(*DB).connectionCleaner(0xc00027e000, 0xc000553d50)\n\t/usr/local/go/src/database/sql/sql.go:1068 +0xbd\ncreated by database/sql.(*DB).startCleanerLocked\n\t/usr/local/go/src/database/sql/sql.go:1055 +0x105\n\ngoroutine 178848 [semacquire, 92 minutes]:\nsync.runtime_SemacquireMutex(0xc02153ee68, 0xaa, 0x18)\n\t/usr/local/go/src/runtime/sema.go:71 +0x25\nsync.(*Mutex).lockSlow(0xc000789a88)\n\t/usr/local/go/src/sync/mutex.go:138 +0x165\nsync.(*Mutex).Lock(...)\n\t/usr/local/go/src/sync/mutex.go:81\nstorj.io/storj/storagenode/orders.(*FileStore).BeginEnqueue.func1(0xc06b9e5660)\n\t/go/src/storj.io/storj/storagenode/orders/store.go:94 +0x94\nstorj.io/storj/storagenode/piecestore.(*Endpoint).beginSaveOrder.func1({0x13cd368, 0xc01ac403c0}, 0xc0229084b0)\n\t/go/src/storj.io/storj/storagenode/piecestore/endpoint.go:726 +0x106\nstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload(0xc00018c000, {0x13dabf0, 0xc0166f7c80})\n\t/go/src/storj.io/storj/storagenode/piecestore/endpoint.go:443 +0x1fe0\nstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1({0x102bca0, 0xc00018c000}, {0xc0062528a0, 0x1d}, {0xfa1a60, 0xc0055925a0}, {0x203001, 0x203001})\n\t/go/pkg/mod/storj.io/common@v0.0.0-20221024150824-a2a5c611dacf/pb/piecestore2_drpc.pb.go:220 +0xb0\nstorj.io/drpc/drpcmux.(*Mux).HandleRPC(0xc0006c8a80, {0x13d6220, 0xc0055925a0}, {0xc0062528a0, 0x1d})\n\t/go/pkg/mod/storj.io/drpc@v0.0.32/drpcmux/handle_rpc.go:33 +0x22d\nstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC(0xc00011e3c0, {0x13d61a0, 0xc005592560}, {0xc0062528a0, 0x1d})\n\t/go/pkg/mod/storj.io/common@v0.0.0-20221024150824-a2a5c611dacf/rpc/rpctracing/handler.go:61 +0x30b\nstorj.io/common/experiment.(*Handler).HandleRPC(0xc00011c650, {0x13d62a0, 0xc00ac59680}, {0xc0062528a0, 0x1d})\n\t/go/pkg/mod/storj.io/common@v0.0.0-20221024150824-a2a5c611dacf/experiment/import.go:42 +0x170\nstorj.io/drpc/drpcserver.(*Server).handleRPC(0xc00bd8fa40, 0x13ccf40, {0xc0062528a0, 0x1b872b8})\n\t/go/pkg/mod/storj.io/drpc@v0.0.32/drpcserver/server.go:122 +0x42\nstorj.io/drpc/drpcserver.(*Server).ServeOne(0xc00018a0e0, {0x13cd640, 0xc00053a2a0}, {0x7f4dd171ff20, 0xc004757500})\n\t/go/pkg/mod/storj.io/drpc@v0.0.32/drpcserver/server.go:66 +0x1f3\nstorj.io/drpc/drpcserver.(*Server).Serve.func2({0x13cd640, 0xc00053a2a0})\n\t/go/pkg/mod/storj.io/drpc@v0.0.32/drpcserver/server.go:112 +0x58\nstorj.io/drpc/drpcctx.(*Tracker).track(0xc00053a2a0, 0xc01f4e96e0)\n\t/go/pkg/mod/storj.io/drpc@v0.0.32/drpcctx/transport.go:52 +0x2b\ncreated by storj.io/drpc/drpcctx.(*Tracker).Run\n\t/go/pkg/mod/storj.io/drpc@v0.0.32/drpcctx/transport.go:47 +0x92\n\ngoroutine 245865 [semacquire, 52 minutes]:\nsync.runtime_SemacquireMutex(0xc094566e68, 0xaa, 0x18)\n\t/usr/local/go/src/runtime/sema.go:71 +0x25\nsync.(*Mutex).lockSlow(0xc000789a88)\n\t/usr/local/go/src/sync/mutex.go:138 +0x165\nsync.(*Mutex).Lock(...)\n\t/usr/local/go/src/sync/mutex.go:81\nstorj.io/storj/storagenode/orders.(*FileStore).BeginEnqueue.func1(0xc0787f4160)\n\t/go/src/storj.io/storj/storagenode/orders/store.go:94 +0x94\nstorj.io/storj/storagenode/piecestore.(*Endpoint).beginSaveOrder.func1({0x13cd368, 0xc021de17c0}, 0xc02df36e10)\n\t/go/src/storj.io/storj/storagenode/piecestore/endpoint.go:726 +0x106\nstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload(0xc00018c000, {0x13dabf0, 0xc027e0c410})\n\t/go/src/storj.io/storj/storagenode/piecestore/endpoint.go:443 +0x1fe0\nstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1({0x102bca0, 0xc00018c000}, {0xc03a357920, 0x1d}, {0xfa1a60, 0xc027e10b00}, {0x203009, 0x203009})\n\t/go/pkg/mod/storj.io/common@v0.0.0-20221024150824-a2a5c611dacf/pb/piecestore2_drpc.pb.go:220 +0xb0\nstorj.io/drpc/drpcmux.(*Mux).HandleRPC(0xc0006c8a80, {0x13d6220, 0xc027e10b00}, {0xc03a357920, 0x1d})\n\t/go/pkg/mod/storj.io/drpc@v0.0.32/drpcmux/handle_rpc.go:33 +0x22d\nstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC(0xc00011e3c0, {0x13d61a0, 0xc027e10ac0}, {0xc03a357920, 0x1d})\n\t/go/pkg/mod/storj.io/common@v0.0.0-20221024150824-a2a5c611dacf/rpc/rpctracing/handler.go:61 +0x30b\nstorj.io/common/experiment.(*Handler).HandleRPC(0xc00011c650, {0x13d62a0, 0xc023112900}, {0xc03a357920, 0x1d})\n\t/go/pkg/mod/storj.io/common@v0.0.0-20221024150824-a2a5c611dacf/experiment/import.go:42 +0x170\nstorj.io/drpc/drpcserver.(*Server).handleRPC(0xc00c8ee540, 0x13ccf40, {0xc03a357920, 0x1b872b8})\n\t/go/pkg/mod/storj.io/drpc@v0.0.32/drpcserver/server.go:122 +0x42\nstorj.io/drpc/drpcserver.(*Server).ServeOne(0xc00018a0e0, {0x13cd640, 0xc00053a2a0}, {0x7f4dd171ff20, 0xc0237a1500})\n\t/go/pkg/mod/storj.io/drpc@v0.0.32/drpcserver/server.go:66 +0x1f3\nstorj.io/drpc/drpcserver.(*Server).Serve.func2({0x13cd640, 0xc00053a2a0})\n\t/go/pkg/mod/storj.io/drpc@v0.0.32/drpcserver/server.go:112 +0x58\nstorj.io/drpc/drpcctx.(*Tracker).track(0xc00053a2a0, 0xc02a2b6390)\n\t/go/pkg/mod/storj.io/drpc@v0.0.32/drpcctx/transport.go:52 +0x2b\ncreated by storj.io/drpc/drpcctx.(*Tracker).Run\n\t/go/pkg/mod/storj.io/drpc@v0.0.32/drpcctx/transport.go:47 +0x92\n\ngoroutine 244194 [semacquire, 52 minutes]:\nsync.runtime_SemacquireMutex(0xc0009eee68, 0xaa, 0x18)\n\t/usr/local/go/src/runtime/sema.go:71 +0x25\nsync.(*Mutex).lockSlow(0xc000789a88)\n\t/usr/local/go/src/sync/mutex.go:138 +0x165\nsync.(*Mutex).Lock(...)\n\t/usr/local/go/src/sync/mutex.go:81\nstorj.io/storj/storagenode/orders.(*FileStore).BeginEnqueue.func1(0xc089289d10)\n\t/go/src/storj.io/storj/storagenode/orders/store.go:94 +0x94\nstorj.io/storj/storagenode/piecestore.(*Endpoint).beginSaveOrder.func1({0x13cd368, 0xc00c1df5e0}, 0xc01e40fb80)\n\t/go/src/storj.io/storj/storagenode/piecestore/endpoint.go:726 +0x106\nstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload(0xc00018c000, {0x13dabf0, 0xc0219c18d0})\n\t/go/src/storj.io/storj/storagenode/piecestore/endpoint.go:443 +0x1fe0\nstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1({0x102bca0, 0xc00018c000}, {0xc0158c5d20, 0x1d}, {0xfa1a60, 0xc025ad8480}, {0x203009, 0x203009})\n\t/go/pkg/mod/storj.io/common@v0.0.0-20221024150824-a2a5c611dacf/pb/piecestore2_drpc.pb.go:220 +0xb0\nstorj.io/drpc/drpcmux.(*Mux).HandleRPC(0xc0006c8a80, {0x13d6220, 0xc025ad8480}, {0xc0158c5d20, 0x1d})\n\t/go/pkg/mod/storj.io/drpc@v0.0.32/drpcmux/handle_rpc.go:33 +0x22d\nstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC(0xc00011e3c0, {0x13d61a0, 0xc025ad8420}, {0xc0158c5d20, 0x1d})\n\t/go/pkg/mod/storj.io/common@v0.0.0-20221024150824-a2a5c611dacf/rpc/rpctracing/handler.go:61 +0x30b\nstorj.io/common/experiment.(*Handler).HandleRPC(0xc00011c650, {0x13d62a0, 0xc023112240}, {0xc0158c5d20, 0x1d})\n\t/go/pkg/mod/storj.io/common@v0.0.0-20221024150824-a2a5c611dacf/experiment/import.go:42 +0x170\nstorj.io/drpc/drpcserver.(*Server).handleRPC(0xc01668e540, 0x13ccf40, {0xc0158c5d20, 0x1b872b8})\n\t/go/pkg/mod/storj.io/drpc@v0.0.32/drpcserver/server.go:122 +0x42\nstorj.io/drpc/drpcserver.(*Server).ServeOne(0xc00018a0e0, {0x13cd640, 0xc00053a2a0}, {0x7f4dd171ff20, 0xc0109f2000})\n\t/go/pkg/mod/storj.io/drpc@v0.0.32/drpcserver/server.go:66 +0x1f3\nstorj.io/drpc/drpcserver.(*Server).Serve.func2({0x13cd640, 0xc00053a2a0})\n\t/go/pkg/mod/storj.io/drpc@v0.0.32/drpcserver/server.go:112 +0x58\nstorj.io/drpc/drpcctx.(*Tracker).track(0xc00053a2a0, 0xc01ab020c0)\n\t/go/pkg/mod/storj.io/drpc@v0.0.32/drpcctx/transport.go:52 +0x2b\ncreated by storj.io/drpc/drpcctx.(*Tracker).Run\n\t/go/pkg/mod/storj.io/drpc@v0.0.32/drpcctx/transport.go:47 +0x92\n\ngoroutine 324516 [semacquire, 30 minutes]:\nsync.runtime_SemacquireMutex(0xc0aa322e68, 0xaa, 0x18)\n\t/usr/local/go/src/runtime/sema.go:71 +0x25\nsync.(*Mutex).lockSlow(0xc000789a88)\n\t/usr/local/go/src/sync/mutex.go:138 +0x165\nsync.(*Mutex).Lock(...)\n\t/usr/local/go/src/sync/mutex.go:81\nstorj.io/storj/storagenode/orders.(*FileStore).BeginEnqueue.func1(0xc01b433090)\n\t/go/src/storj.io/storj/storagenode/orders/store.go:94 +0x94\nstorj.io/storj/storagenode/piecestore.(*Endpoint).beginSaveOrder.func1({0x13cd368, 0xc03080bcc0}, 0xc031e75950)\n\t/go/src/storj.io/storj/storagenode/piecestore/endpoint.go:726 +0x106\nstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload(0xc00018c000, {0x13dabf0, 0xc00601e020})\n\t/go/src/storj.io/storj/storagenode/piecestore/endpoint.go:443 +0x1fe0\nstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1({0x102bca0, 0xc00018c000}, {0xc0235b03a0, 0x1d}, {0xfa1a60, 0xc041628bc0}, {0x203010, 0x203010})\n\t/go/pkg/mod/storj.io/common@v0.0.0-20221024150824-a2a5c611dacf/pb/piecestore2_drpc.pb.go:220 +0xb0\nstorj.io/drpc/drpcmux.(*Mux).HandleRPC(0xc0006c8a80, {0x13d6220, 0xc041628bc0}, {0xc0235b03a0, 0x1d})\n\t/go/pkg/mod/storj.io/drpc@v0.0.32/drpcmux/handle_rpc.go:33 +0x22d\nstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC(0xc00011e3c0, {0x13d61a0, 0xc041628a40}, {0xc0235b03a0, 0x1d})\n\t/go/pkg/mod/storj.io/common@v0.0.0-20221024150824-a2a5c611dacf/rpc/rpctracing/handler.go:61 +0x30b\nstorj.io/common/experiment.(*Handler).HandleRPC(0xc00011c650, {0x13d62a0, 0xc007db66c0}, {0xc0235b03a0, 0x1d})\n\t/go/pkg/mod/storj.io/common@v0.0.0-20221024150824-a2a5c611dacf/experiment/import.go:42 +0x170\nstorj.io/drpc/drpcserver.(*Server).handleRPC(0xc009f4ba40, 0x13ccf40, {0xc0235b03a0, 0x1b872b8})\n\t/go/pkg/mod/storj.io/drpc@v0.0.32/drpcserver/server.go:122 +0x42\nstorj.io/drpc/drpcserver.(*Server).ServeOne(0xc00018a0e0, {0x13cd640, 0xc00053a2a0},
7

Yeah I already feared that this may be a problem. I got more ram coming in 2 Days and hope this fixes the issue. Is there a way to configure a cache ssd to temporarily hold the files?

8

most of the io workload can be cached, it helps a great dealā€¦
adding more ram beyond 2-4GB pr node is useless.

only solution is to either speed up the storage, or moving the databases to an ssd which is also a feature that can be used, but isnā€™t used by default.

the ram is just a buffer, it will fill eventuallyā€¦ more does however help even out the issues when storage or cpu cannot keep upā€¦ then ram will fill until it canā€¦ or it crashesā€¦
a node that is underpower can eat just about every GB of RAM you can throw at it.

1 Like
9

(Removed: Apologies, I didnā€™t see your previous message.)

10

Please try to sync your clock first. If the problem is gone - then it was it.
If not - try to replace the image

docker stop -t 300 storagenode
docker rm storagenode

do the same for all your docker nodes
Then remove the image

docker image prune

Then download a latest one

docker pull storjlabs/storagenode:latest

And run your nodes back.

If crashing is happened only for one of your nodes - then please check its disk for errors and fix them.

1 Like
11

After adding more Ram to the System everything seems to be fine.
Thanks for your help everyone :slight_smile:

1 Like