Node keeps crashing after a few hours

Hey :smiley:, I recently added a few new nodes on my Windows System and now they keep mysteriously crashing after a few hours. For the first week I had no problems at all but now after a few hours they run into an error I can only fix by removing the container and adding it again. When the error occurs, the container keeps running and the node port is still reachable. The Dashboard on 127.0.0.1:14002 is no longer working and the ram fills up to 100%. The docker log is full of errors I cannot understand.
Does anyone has experienced this behaviour before? What could be the cause of it?
Thanks in advance!

Here a few lines of the errors:

[syscall, 2 minutes]:\nsyscall.Syscall(0x107, 0xffffffffffffff9c, 0xc045796b40, 0x0)\n\t/usr/local/go/src/syscall/asm_linux_amd64.s:20 +0x5\nsyscall.unlinkat(0xf22840, {0xc03ff40e10, 0x0}, 0x0)\n\t/usr/local/go/src/syscall/zsyscall_linux_amd64.go:127 +0xd4\nsyscall.Unlink(…)\n\t/usr/local/go/src/syscall/syscall_linux.go:185\nos.Remove.func1(…)\n\t/usr/local/go/src/os/file_unix.go:302\nos.ignoringEINTR(…)\n\t/usr/local/go/src/os/file_posix.go:246\nos.Remove({0xc03ff40e10, 0x0})\n\t/usr/local/go/src/os/file_unix.go:301 +0x45\nstorj.io/storj/storage/filestore.(*blobWriter).Cancel(0xc0343dd7a0, {0x13cd368, 0xc03ca508c0})\n\t/go/src/storj.io/storj/storage/filestore/blob.go:102 +0x1fb\nstorj.io/storj/storagenode/pieces.(*Writer).Cancel(0xc01bd9ff80, {0x13cd368, 0xc01d12d7c0})\n\t/go/src/storj.io/storj/storagenode/pieces/readwrite.go:209

Hi @alpha4701,

Please show more of the log surrounding the [syscall, 2 minutes] error. That will give us a better idea of what the cause may be. You can post the log between 3 backticks:

```
Log
```

Hi @jeremy here are the surroundings. The lines repeat over and over again in a similar way.

go:47 +0x92\n\ngoroutine 621847 [select]:\nstorj.io/drpc/drpcmigrate.(*ListenMux).routeConn(0xc00029ef50, {0x13df390, 0xc0ce284280})\n\t/go/pkg/mod/storj.io/drpc@v0.0.32/drpcmigrate/mux.go:161 +0x356\ncreated by storj.io/drpc/drpcmigrate.(*ListenMux).monitorBase\n\t/go/pkg/mod/storj.io/drpc@v0.0.32/drpcmigrate/mux.go:121 +0x29\n\ngoroutine 404938 [syscall, 2 minutes]:\nsyscall.Syscall(0x107, 0xffffffffffffff9c, 0xc00063b980, 0x0)\n\t/usr/local/go/src/syscall/asm_linux_amd64.s:20 +0x5\nsyscall.unlinkat(0xf22840, {0xc016c92ae0, 0x0}, 0x0)\n\t/usr/local/go/src/syscall/zsyscall_linux_amd64.go:127 +0xd4\nsyscall.Unlink(...)\n\t/usr/local/go/src/syscall/syscall_linux.go:185\nos.Remove.func1(...)\n\t/usr/local/go/src/os/file_unix.go:302\nos.ignoringEINTR(...)\n\t/usr/local/go/src/os/file_posix.go:246\nos.Remove({0xc016c92ae0, 0x0})\n\t/usr/local/go/src/os/file_unix.go:301 +0x45\nstorj.io/storj/storage/filestore.(*blobWriter).Cancel(0xc01bb67b00, {0x13cd368, 0xc0ce3adc20})\n\t/go/src/storj.io/storj/storage/filestore/blob.go:102 +0x1fb\nstorj.io/storj/storagenode/pieces.(*Writer).Cancel(0xc013dc4930, {0x13cd368, 0xc0523b03c0})\n\t/go/src/storj.io/storj/storagenode/pieces/readwrite.go:209 +0x1a8\nstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload.func4()\n\t/go/src/storj.io/storj/storagenode/piecestore/endpoint.go:310 +0x46\nstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload(0xc00018c000, {0x13dabf0, 0xc050d889b0})\n\t/go/src/storj.io/storj/storagenode/piecestore/endpoint.go:323 +0x1110\nstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1({0x102bca0, 0xc00018c000}, {0xc0521da5a0, 0x1d}, {0xfa1a60, 0xc04345c160}, {0x203010, 0x203010})\n\t/go/pkg/mod/storj.io/common@v0.0.0-20221024150824-a2a5c611dacf/pb/piecestore2_drpc.pb.go:220 +0xb0\nstorj.io/drpc/drpcmux.(*Mux).HandleRPC(0xc0006c8a80, {0x13d6220, 0xc04345c160}, {0xc0521da5a0, 0x1d})\n\t/go/pkg/mod/storj.io/drpc@v0.0.32/drpcmux/handle_rpc.go:33 +0x22d\nstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC(0xc00011e3c0, {0x13d61a0, 0xc04345c100}, {0xc0521da5a0, 0x1d})\n\t/go/pkg/mod/storj.io/common@v0.0.0-20221024150824-a2a5c611dacf/rpc/rpctracing/handler.go:61 +0x30b\nstorj.io/common/experiment.(*Handler).HandleRPC(0xc00011c650, {0x13d62a0, 0xc05ef93200}, {0xc0521da5a0, 0x1d})\n\t/go/pkg/mod/storj.io/common@v0.0.0-20221024150824-a2a5c611dacf/experiment/import.go:42 +0x170\nstorj.io/drpc/drpcserver.(*Server).handleRPC(0xc0617fcfc0, 0x13ccf40, {0xc0521da5a0, 0x1b872b8})\n\t/go/pkg/mod/storj.io/drpc@v0.0.32/drpcserver/server.go:122 +0x42\nstorj.io/drpc/drpcserver.(*Server).ServeOne(0xc00018a0e0, {0x13cd640, 0xc00053a2a0}, {0x7f4dd171ff20, 0xc034769880})\n\t/go/pkg/mod/storj.io/drpc@v0.0.32/drpcserver/server.go:66 +0x1f3\nstorj.io/drpc/drpcserver.(*Server).Serve.func2({0x13cd640, 0xc00053a2a0})\n\t/go/pkg/mod/storj.io/drpc@v0.0.32/drpcserver/server.go:112 +0x58\nstorj.io/drpc/drpcctx.(*Tracker).track(0xc00053a2a0, 0xc050d88200)\n\t/go/pkg/mod/storj.io/drpc@v0.0.32/drpcctx/transport.go:52 +0x2b\ncreated by storj.io/drpc/drpcctx.(*Tracker).Run\n\t/go/pkg/mod/storj.io/drpc@v0.0.32/drpcctx/transport.go:47 +0x92\n\ngoroutine 404941 [syscall, 2 minutes]:\nsyscall.Syscall(0x107, 0xffffffffffffff9c, 0xc059ef7800, 0x0)\n\t/usr/local/go/src/syscall/asm_linux_amd64.s:20 +0x5\nsyscall.unlinkat(0xf22840, {0xc01516b410, 0x0}, 0x0)\n\t/usr/local/go/src/syscall/zsyscall_linux_amd64.go:127 +0xd4\nsyscall.Unlink(...)\n\t/usr/local/go/src/syscall/syscall_linux.go:185\nos.Remove.func1(...)\n\t/usr/local/go/src/os/file_unix.go:302\nos.ignoringEINTR(...)\n\t/usr/local/go/src/os/file_posix.go:246\nos.Remove({0xc01516b410, 0x0})\n\t/usr/local/go/src/os/file_unix.go:301 +0x45\nstorj.io/storj/storage/filestore.(*blobWriter).Cancel(0xc01b64af00, {0x13cd368, 0xc035d5c3c0})\n\t/go/src/storj.io/storj/storage/filestore/blob.go:102 +0x1fb\nstorj.io/storj/storagenode/pieces.(*Writer).Cancel(0xc05f69f030, {0x13cd368, 0xc00a727680})\n\t/go/src/storj.io/storj/storagenode/pieces/readwrite.go:209 +0x1a8\nstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload.func4()\n\t/go/src/storj.io/storj/storagenode/piecestore/endpoint.go:310 +0x46\nstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload(0xc00018c000, {0x13dabf0, 0xc051e7f1a0})\n\t/go/src/storj.io/storj/storagenode/piecestore/endpoint.go:323 +0x1110\nstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1({0x102bca0, 0xc00018c000}, {0xc00b1b0b60, 0x1d}, {0xfa1a60, 0xc05b0dfde0}, {0x203016, 0x203016})\n\t/go/pkg/mod/storj.io/common@v0.0.0-20221024150824-a2a5c611dacf/pb/piecestore2_drpc.pb.go:220 +0xb0\nstorj.io/drpc/drpcmux.(*Mux).HandleRPC(0xc0006c8a80, {0x13d6220, 0xc05b0dfde0}, {0xc00b1b0b60, 0x1d})\n\t/go/pkg/mod/storj.io/drpc@v0.0.32/drpcmux/handle_rpc.go:33 +0x22d\nstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC(0xc00011e3c0, {0x13d61a0, 0xc05b0dfda0}, {0xc00b1b0b60, 0x1d})\n\t/go/pkg/mod/storj.io/common@v0.0.0-20221024150824-a2a5c611dacf/rpc/rpctracing/handler.go:61 +0x30b\nstorj.io/common/experiment.(*Handler).HandleRPC(0xc00011c650, {0x13d62a0, 0xc05a634fc0}, {0xc00b1b0b60, 0x1d})\n\t/go/pkg/mod/storj.io/common@v0.0.0-20221024150824-a2a5c611dacf/experiment/import.go:42 +0x170\nstorj.io/drpc/drpcserver.(*Server).handleRPC(0xc0617fd180, 0x13ccf40, {0xc00b1b0b60, 0x1b872b8})\n\t/go/pkg/mod/storj.io/drpc@v0.0.32/drpcserver/server.go:122 +0x42\nstorj.io/drpc/drpcserver.(*Server).ServeOne(0xc00018a0e0, {0x13cd640, 0xc00053a2a0}, {0x7f4dd171ff20, 0xc034769c00})\n\t/go/pkg/mod/storj.io/drpc@v0.0.32/drpcserver/server.go:66 +0x1f3\nstorj.io/drpc/drpcserver.(*Server).Serve.func2({0x13cd640, 0xc00053a2a0})\n\t/go/pkg/mod/storj.io/drpc@v0.0.32/drpcserver/server.go:112 +0x58\nstorj.io/drpc/drpcctx.(*Tracker).track(0xc00053a2a0, 0xc050d89570)\n\t/go/pkg/mod/storj.io/drpc@v0.0.32/drpcctx/transport.go:52 +0x2b\ncreated by storj.io/drpc/drpcctx.(*Tracker).Run\n\t/go/pkg/mod/storj.io/drpc@v0.0.32/drpcctx/transport.go:47 +0x92\n\ngoroutine 404350 [syscall, 2 minutes]:\nsyscall.Syscall(0x107, 0xffffffffffffff9c, 0xc0ce2f03f0, 0x0)\n\t/usr/local/go/src/syscall/asm_linux_amd64.s:20 +0x5\nsyscall.unlinkat(0xf22840, {0xc0340b7920, 0x0}, 0x0)\n\t/usr/local/go/src/syscall/zsyscall_linux_amd64.go:127 +0xd4\nsyscall.Unlink(...)\n\t/usr/local/go/src/syscall/syscall_linux.go:185\nos.Remove.func1(...)\n\t/usr/local/go/src/os/file_unix.go:302\nos.ignoringEINTR(...)\n\t/usr/local/go/src/os/file_posix.go:246\nos.Remove({0xc0340b7920, 0x0})\n\t/usr/local/go/src/os/file_unix.go:301 +0x45\nstorj.io/storj/storage/filestore.(*blobWriter).Cancel(0xc009c7d4a0, {0x13cd368, 0xc0137d7680})\n\t/go/src/storj.io/storj/storage/filestore/blob.go:102 +0x1fb\nstorj.io/storj/storagenode/pieces.(*Writer).Cancel(0xc004b68c40, {0x13cd368, 0xc03688f9a0})\n\t/go/src/storj.io/storj/storagenode/pieces/readwrite.go:209 +0x1a8\nstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload.func4()\n\t/go/src/storj.io/storj/storagenode/piecestore/endpoint.go:310 +0x46\nstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload(0xc00018c000, {0x13dabf0, 0xc051e7f670})\n\t/go/src/storj.io/storj/storagenode/piecestore/endpoint.go:323 +0x1110\nstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1({0x102bca0, 0xc00018c000}, {0xc0064e8260, 0x1d}, {0xfa1a60, 0xc02a99cca0}, {0x20300a, 0x20300a})\n\t/go/pkg/mod/storj.io/common@v0.0.0-20221024150824-a2a5c611dacf/pb/piecestore2_drpc.pb
.go:220 +0xb0\nstorj.io/drpc/drpcmux.(*Mux).HandleRPC(0xc0006c8a80, {0x13d6220, 0xc02a99cca0}, {0xc0064e8260, 0x1d})\n\t/go/pkg/mod/storj.io/drpc@v0.0.32/drpcmux/handle_rpc.go:33 +0x22d\nstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC(0xc00011e3c0, {0x13d61a0, 0xc02a99c9e0}, {0xc0064e8260, 0x1d})\n\t/go/pkg/mod/storj.io/common@v0.0.0-20221024150824-a2a5c611dacf/rpc/rpctracing/handler.go:61 +0x30b\nstorj.io/common/experiment.(*Handler).HandleRPC(0xc00011c650, {0x13d62a0, 0xc05a635200}, {0xc0064e8260, 0x1d})\n\t/go/pkg/mod/storj.io/common@v0.0.0-20221024150824-a2a5c611dacf/experiment/import.go:42 +0x170\nstorj.io/drpc/drpcserver.(*Server).handleRPC(0xc04c42a000, 0x13ccf40, {0xc0064e8260, 0x1b872b8})\n\t/go/pkg/mod/storj.io/drpc@v0.0.32/drpcserver/server.go:122 +0x42\nstorj.io/drpc/drpcserver.(*Server).ServeOne(0xc00018a0e0, {0x13cd640, 0xc00053a2a0}, {0x7f4dd171ff20, 0xc061255c00})\n\t/go/pkg/mod/storj.io/drpc@v0.0.32/drpcserver/server.go:66 +0x1f3\nstorj.io/drpc/drpcserver.(*Server).Serve.func2({0x13cd640, 0xc00053a2a0})\n\t/go/pkg/mod/storj.io/drpc@v0.0.32/drpcserver/server.go:112 +0x58\nstorj.io/drpc/drpcctx.(*Tracker).track(0xc00053a2a0, 0xc050d895a0)\n\t/go/pkg/mod/storj.io/drpc@v0.0.32/drpcctx/transport.go:52 +0x2b\ncreated by storj.io/drpc/drpcctx.(*Tracker).Run\n\t/go/pkg/mod/storj.io/drpc@v0.0.32/drpcctx/transport.go:47 +0x92\n\ngoroutine 405071 [syscall, 2 minutes]:\nsyscall.Syscall(0x107, 0xffffffffffffff9c, 0xc045796c60, 0x0)\n\t/usr/local/go/src/syscall/asm_linux_amd64.s:20 +0x5\nsyscall.unlinkat(0xf22840, {0xc016c92630, 0x0}, 0x0)\n\t/usr/local/go/src/syscall/zsyscall_linux_amd64.go:127 +0xd4\nsyscall.Unlink(...)\n\t/usr/local/go/src/syscall/syscall_linux.go:185\nos.Remove.func1(...)\n\t/usr/local/go/src/os/file_unix.go:302\nos.ignoringEINTR(...)\n\t/usr/local/go/src/os/file_posix.go:246\nos.Remove({0xc016c92630, 0x0})\n\t/usr/local/go/src/os/file_unix.go:301 +0x45\nstorj.io/storj/storage/filestore.(*blobWriter).Cancel(0xc01fdc8300, {0x13cd368, 0xc03ca51040})\n\t/go/src/storj.io/storj/storage/filestore/blob.go:102 +0x1fb\nstorj.io/storj/storagenode/pieces.(*Writer).Cancel(0xc00bb63420, {0x13cd368, 0xc0523b1900})\n\t/go/src/storj.io/storj/storagenode/pieces/readwrite.go:209 +0x1a8\nstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload.func4()\n\t/go/src/storj.io/storj/storagenode/piecestore/endpoint.go:310 +0x46\nstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload(0xc00018c000, {0x13dabf0, 0xc050d89f60})\n\t/go/src/storj.io/storj/storagenode/piecestore/endpoint.go:323 +0x1110\nstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1({0x102bca0, 0xc00018c000}, {0xc01bdf53a0, 0x1d}, {0xfa1a60, 0xc04345dc80}, {0x203010, 0x203010})\n\t/go/pkg/mod/storj.io/common@v0.0.0-20221024150824-a2a5c611dacf/pb/piecestore2_drpc.pb.go:220 +0xb0\nstorj.io/drpc/drpcmux.(*Mux).HandleRPC(0xc0006c8a80, {0x13d6220, 0xc04345dc80}, {0xc01bdf53a0, 0x1d})\n\t/go/pkg/mod/storj.io/drpc@v0.0.32/drpcmux/handle_rpc.go:33 +0x22d\nstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC(0xc00011e3c0, {0x13d61a0, 0xc04345dc40}, {0xc01bdf53a0, 0x1d})\n\t/go/pkg/mod/storj.io/common@v0.0.0-20221024150824-a2a5c611dacf/rpc/rpctracing/handler.go:61 +0x30b\nstorj.io/common/experiment.(*Handler).HandleRPC(0xc00011c650, {0x13d62a0, 0xc05ef93440}, {0xc01bdf53a0, 0x1d})\n\t/go/pkg/mod/storj.io/common@v0.0.0-20221024150824-a2a5c611dacf/experiment/import.go:42 +0x170\nstorj.io/drpc/drpcserver.(*Server).handleRPC(0xc061872fc0, 0x13ccf40, {0xc01bdf53a0, 0x1b872b8})\n\t/go/pkg/mod/storj.io/drpc@v0.0.32/drpcserver/server.go:122 +0x42\nstorj.io/drpc/drpcserver.(*Server).ServeOne(0xc00018a0e0, {0x13cd640, 0xc00053a2a0}, {0x7f4dd171ff20, 0xc056dec000})\n\t/go/pkg/mod/storj.io/drpc@v0.0.32/drpcserver/server.go:66 +0x1f3\nstorj.io/drpc/drpcserver.(*Server).Serve.func2({0x13cd640, 0xc00053a2a0})\n\t/go/pkg/mod/storj.io/drpc@v0.0.32/drpcserver/server.go:112 +0x58\nstorj.io/drpc/drpcctx.(*Tracker).track(0xc00053a2a0, 0xc0107efb70)\n\t/go/pkg/mod/storj.io/drpc@v0.0.32/drpcctx/transport.go:52 +0x2b\ncreated by storj.io/drpc/drpcctx.(*Tracker).Run\n\t/go/pkg/mod/storj.io/drpc@v0.0.32/drpcctx/transport.

Thank you. What does the log look like before the first occurrence of this type of error? Does “panic” appear anywhere in your logs?

ram will fill up if the storagenodes cpu time is to limited (overloaded cpu) or if the storage cannot keep up…

“panic” does not appear anywhere.
This is the first Error. After the it occurs there appear more and more error messages until there are only errors.

2022-11-21T14:02:21.687Z	INFO	piecedeleter	delete piece sent to trash	{"Process": "storagenode", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Piece ID": "E2HNU6NCIDJ2W2FHKEDCPLBGUWUIQMFLLJRBBUHXROYD23AD2HXQ"}
2022-11-21T14:02:22.090Z	INFO	piecedeleter	delete piece sent to trash	{"Process": "storagenode", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Piece ID": "LETI3C4RDHX6CPZTCK7ABVBL24CS4TQXU6AXJJA7ZMHS6UUXVJMQ"}
2022-11-21T14:02:22.190Z	INFO	piecestore	upload started	{"Process": "storagenode", "Piece ID": "THH443NUHGJIZ6Q4COYHCJ4DH3WFV7JWU7TGZF4EB4Q3FARSFNSA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Available Space": 1866150438912}
2022-11-21T14:02:22.604Z	INFO	piecestore	upload started	{"Process": "storagenode", "Piece ID": "AVHOF55H42NWE2Y7FSLG6MLSNKK2I7V7YROSURDGZYM4I2E4J2JA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Available Space": 1866150438912}
2022-11-21T14:02:22.603Z	INFO	piecestore	upload started	{"Process": "storagenode", "Piece ID": "DZQCU4VAE7MXUX3GX4SNISKHIWSTAQZAR5474YNETAE7Q5HF3JEQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT", "Available Space": 1866150438912}
2022-11-21T14:02:22.610Z	INFO	piecestore	upload started	{"Process": "storagenode", "Piece ID": "BHHVPUFZVKNA6LCWEROQC5NZLHMNFN5AKVOG27GRBOHYHOHENNJA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Available Space": 1866150438912}
2022-11-21T14:02:22.603Z	INFO	piecestore	upload started	{"Process": "storagenode", "Piece ID": "LHNYBHV63FO5I7CBPA53FMT3XNIWOVYTVHZ77BAX2X4IGTM2ACGA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Available Space": 1866150438912}
2022-11-21T14:02:22.892Z	INFO	piecestore	upload started	{"Process": "storagenode", "Piece ID": "EAMTGGK6ZACCTJ4MLDJPGKT4T5N4KSYRI5NRZWBOKUQVAOJDEPDA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT_REPAIR", "Available Space": 1866150438912}
2022-11-21T14:02:24.097Z	ERROR	piecedeleter	could not send delete piece to trash	{"Process": "storagenode", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Piece ID": "MT5L6WPIZEPFV7TUUOKKLEC5F7OCAOOVD5RWWOR7ARQGE52RW5QA", "error": "pieces error: filestore error: rename config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/mt/5l6wpizepfv7tuuokklec5f7ocaoovd5rwwor7arqge52rw5qa.sj1 config/storage/trash/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/mt/5l6wpizepfv7tuuokklec5f7ocaoovd5rwwor7arqge52rw5qa.sj1: input/output error", "errorVerbose": "pieces error: filestore error: rename config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/mt/5l6wpizepfv7tuuokklec5f7ocaoovd5rwwor7arqge52rw5qa.sj1 config/storage/trash/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/mt/5l6wpizepfv7tuuokklec5f7ocaoovd5rwwor7arqge52rw5qa.sj1: input/output error\n\tstorj.io/storj/storage/filestore.(*blobStore).Trash:140\n\tstorj.io/storj/storagenode/pieces.(*BlobsUsageCache).Trash:295\n\tstorj.io/storj/storagenode/pieces.(*Store).Trash:365\n\tstorj.io/storj/storagenode/pieces.(*Deleter).deleteOrTrash:185\n\tstorj.io/storj/storagenode/pieces.(*Deleter).work:135\n\tstorj.io/storj/storagenode/pieces.(*Deleter).Run.func1:72\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75"}
...
2022-11-21T14:03:54.869Z	ERROR	piecestore	upload failed	{"Process": "storagenode", "Piece ID": "DXGSGR4GO25HMYW3XNUTUAUFVV52K26TCN2JP3MPFADICFJZF6SQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "error": "order: grace period passed for order limit", "errorVerbose": "order: grace period passed for order limit\n\tstorj.io/storj/storagenode/orders.(*FileStore).BeginEnqueue:86\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).beginSaveOrder:705\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:321\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:220\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:61\n\tstorj.io/common/experiment.(*Handler).HandleRPC:42\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:122\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:112\n\tstorj.io/drpc/drpcctx.(*Tracker).track:52", "Size": 0}
2022-11-21T14:03:54.872Z	ERROR	piecestore	upload failed	{"Process": "storagenode", "Piece ID": "XSBTWFWIZ5NJIURUG4IRXL3IZHLHCNIECBK6HGXFMKKELT3J6J6A", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "error": "order: grace period passed for order limit", "errorVerbose": "order: grace period passed for order limit\n\tstorj.io/storj/storagenode/orders.(*FileStore).BeginEnqueue:86\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).beginSaveOrder:705\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:321\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:220\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:61\n\tstorj.io/common/experiment.(*Handler).HandleRPC:42\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:122\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:112\n\tstorj.io/drpc/drpcctx.(*Tracker).track:52", "Size": 0}
2022-11-21T14:03:54.919Z	ERROR	piecestore	upload failed	{"Process": "storagenode", "Piece ID": "NO7FHV4QES3LFRWPRJ6EEFSBHTJMTUO33ZO5ILEFPF4WV2TCADWA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "error": "order: grace period passed for order limit", "errorVerbose": "order: grace period passed for order limit\n\tstorj.io/storj/storagenode/orders.(*FileStore).BeginEnqueue:86\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).beginSaveOrder:705\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:321\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:220\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:61\n\tstorj.io/common/experiment.(*Handler).HandleRPC:42\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:122\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:112\n\tstorj.io/drpc/drpcctx.(*Tracker).track:52", "Size": 0}
2022-11-21T14:03:54.924Z	ERROR	piecestore	upload failed	{"Process": "storagenode", "Piece ID": "KSVIVQFYJV44YJ6QGEPKWKSFAJ5OIGC4BNKHREGX4M3RKYE3X73Q", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "error": "order: grace period passed for order limit", "errorVerbose": "order: grace period passed for order limit\n\tstorj.io/storj/storagenode/orders.(*FileStore).BeginEnqueue:86\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).beginSaveOrder:705\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:321\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:220\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:61\n\tstorj.io/common/experiment.(*Handler).HandleRPC:42\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:122\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:112\n\tstorj.io/drpc/drpcctx.(*Tracker).track:52", "Size": 0}
2022-11-21T14:03:54.926Z	ERROR	piecestore	upload failed	{"Process": "storagenode", "Piece ID": "SX55RO5FVD7KJQXYOYCBMZLGZG5SNQA5Z4AGIQLVNBUDP633SZLQ", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Action": "PUT", "error": "order: grace period passed for order limit", "errorVerbose": "order: grace period passed for order limit\n\tstorj.io/storj/storagenode/orders.(*FileStore).BeginEnqueue:86\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).beginSaveOrder:705\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:321\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:220\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:61\n\tstorj.io/common/experiment.(*Handler).HandleRPC:42\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:122\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:112\n\tstorj.io/drpc/drpcctx.(*Tracker).track:52", "Size": 0}
2022-11-21T14:03:54.942Z	ERROR	piecestore	upload failed	{"Process": "storagenode", "Piece ID": "K6FA2QRIOVL6VFUZDUQOSC6I4BKVUBHSK2TLJTWBF7UB26QVAOZA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "error": "order: grace period passed for order limit", "errorVerbose": "order: grace period passed for order limit\n\tstorj.io/storj/storagenode/orders.(*FileStore).BeginEnqueue:86\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).beginSaveOrder:705\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:321\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:220\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:61\n\tstorj.io/common/experiment.(*Handler).HandleRPC:42\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:122\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:112\n\tstorj.io/drpc/drpcctx.(*Tracker).track:52", "Size": 0}

After a while this results in this (as the error is way too long this is only the beginning):

2022-11-21T14:03:39.033Z	INFO	services	slow shutdown	{"Process": "storagenode", "stack": "goroutine 1211 [running]:\nstorj.io/storj/private/lifecycle.(*Group).logStackTrace.func1()\n\t/go/src/storj.io/storj/private/lifecycle/group.go:107 +0x85\nsync.(*Once).doSlow(0xc098afc000, 0xc0cdbe2600)\n\t/usr/local/go/src/sync/once.go:68 +0xd2\nsync.(*Once).Do(...)\n\t/usr/local/go/src/sync/once.go:59\nstorj.io/storj/private/lifecycle.(*Group).logStackTrace(0xc0007c46b8)\n\t/go/src/storj.io/storj/private/lifecycle/group.go:104 +0x45\nstorj.io/storj/private/lifecycle.(*Group).Run.func1({0x13ccf40, 0xc00053b770})\n\t/go/src/storj.io/storj/private/lifecycle/group.go:77 +0x2e5\nruntime/pprof.Do({0x13cd368, 0xc0004be5a0}, {{0xc0003c2280, 0x0, 0x0}}, 0xc0004e0cc0)\n\t/usr/local/go/src/runtime/pprof/runtime.go:40 +0xa3\ncreated by storj.io/storj/private/lifecycle.(*Group).Run\n\t/go/src/storj.io/storj/private/lifecycle/group.go:64 +0x630\n\ngoroutine 1 [semacquire, 249 minutes]:\nsync.runtime_Semacquire(0x0)\n\t/usr/local/go/src/runtime/sema.go:56 +0x25\nsync.(*WaitGroup).Wait(0xc0003b8620)\n\t/usr/local/go/src/sync/waitgroup.go:130 +0x71\ngolang.org/x/sync/errgroup.(*Group).Wait(0xc0004e0140)\n\t/go/pkg/mod/golang.org/x/sync@v0.0.0-20220819030929-7fc1605a5dde/errgroup/errgroup.go:53 +0x27\nstorj.io/storj/storagenode.(*Peer).Run(0xc0001c6000, {0x13cce98, 0xc0002929c0})\n\t/go/src/storj.io/storj/storagenode/peer.go:911 +0x49d\nmain.cmdRun(0xf314e0, {0xc000283080, 0x1ad6e80, 0x441cee})\n\t/go/src/storj.io/storj/cmd/storagenode/main.go:251 +0xe85\nstorj.io/private/process.cleanup.func1.4({0x13cd368, 0xc0004be1e0})\n\t/go/pkg/mod/storj.io/private@v0.0.0-20221011183246-586e5f48357a/process/exec_conf.go:377 +0x14f\nstorj.io/private/process.cleanup.func1(0x1ad6e80, {0xc0002be300, 0x0, 0xc})\n\t/go/pkg/mod/storj.io/private@v0.0.0-20221011183246-586e5f48357a/process/exec_conf.go:395 +0x1ca8\ngithub.com/spf13/cobra.(*Command).execute(0x1ad6e80, {0xc0002be240, 0xc, 0xc})\n\t/go/pkg/mod/github.com/spf13/cobra@v1.1.3/command.go:852 +0x60e\ngithub.com/spf13/cobra.(*Command).ExecuteC(0x1ad6c00)\n\t/go/pkg/mod/github.com/spf13/cobra@v1.1.3/command.go:960 +0x3ad\ngithub.com/spf13/cobra.(*Command).Execute(...)\n\t/go/pkg/mod/github.com/spf13/cobra@v1.1.3/command.go:897\nstorj.io/private/process.ExecWithCustomConfigAndLogger(0x1ad6c00, 0xd0, 0xedf360, 0xc0000001a0)\n\t/go/pkg/mod/storj.io/private@v0.0.0-20221011183246-586e5f48357a/process/exec_conf.go:92 +0x190\nmain.main()\n\t/go/src/storj.io/storj/cmd/storagenode/main.go:478 +0x30\n\ngoroutine 13 [select]:\nstorj.io/monkit-jaeger.(*UDPCollector).Run(0xc0002be3c0, {0x13cce98, 0xc000292940})\n\t/go/pkg/mod/storj.io/monkit-jaeger@v0.0.0-20220915074555-d100d7589f41/udp.go:160 +0x7ff\nstorj.io/private/process.cleanup.func1.2()\n\t/go/pkg/mod/storj.io/private@v0.0.0-20221011183246-586e5f48357a/process/exec_conf.go:328 +0x25\ngolang.org/x/sync/errgroup.(*Group).Go.func1()\n\t/go/pkg/mod/golang.org/x/sync@v0.0.0-20220819030929-7fc1605a5dde/errgroup/errgroup.go:75 +0x64\ncreated by golang.org/x/sync/errgroup.(*Group).Go\n\t/go/pkg/mod/golang.org/x/sync@v0.0.0-20220819030929-7fc1605a5dde/errgroup/errgroup.go:72 +0xa9\n\ngoroutine 15 [syscall, 249 minutes]:\nos/signal.signal_recv()\n\t/usr/local/go/src/runtime/sigqueue.go:169 +0x98\nos/signal.loop()\n\t/usr/local/go/src/os/signal/signal_unix.go:24 +0x19\ncreated by os/signal.Notify.func1.1\n\t/usr/local/go/src/os/signal/signal.go:151 +0x2c\n\ngoroutine 53 [chan receive, 249 minutes]:\nstorj.io/private/process.Ctx.func1()\n\t/go/pkg/mod/storj.io/private@v0.0.0-20221011183246-586e5f48357a/process/exec_conf.go:119 +0x3a\ncreated by storj.io/private/process.Ctx\n\t/go/pkg/mod/storj.io/private@v0.0.0-20221011183246-586e5f48357a/process/exec_conf.go:118 +0x2de\n\ngoroutine 71 [select, 249 minutes]:\ndatabase/sql.(*DB).connectionOpener(0xc00027e000, {0x13cce98, 0xc000292000})\n\t/usr/local/go/src/database/sql/sql.go:1196 +0x93\ncreated by database/sql.OpenDB\n\t/usr/local/go/src/database/sql/sql.go:794 +0x188\n\ngoroutine 312985 [semacquire, 32 minutes]:\nsync.runtime_SemacquireMutex(0xc0276c8e68, 0xaa, 0x18)\n\t/usr/local/go/src/runtime/sema.go:71 +0x25\nsync.(*Mutex).lockSlow(0xc000789a88)\n\t/usr/local/go/src/sync/mutex.go:138 +0x165\nsync.(*Mutex).Lock(...)\n\t/usr/local/go/src/sync/mutex.go:81\nstorj.io/storj/storagenode/orders.(*FileStore).BeginEnqueue.func1(0xc0a60d7070)\n\t/go/src/storj.io/storj/storagenode/orders/store.go:94 +0x94\nstorj.io/storj/storagenode/piecestore.(*Endpoint).beginSaveOrder.func1({0x13cd368, 0xc00543d2c0}, 0xc0116b9bd0)\n\t/go/src/storj.io/storj/storagenode/piecestore/endpoint.go:726 +0x106\nstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload(0xc00018c000, {0x13dabf0, 0xc0061f9780})\n\t/go/src/storj.io/storj/storagenode/piecestore/endpoint.go:443 +0x1fe0\nstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1({0x102bca0, 0xc00018c000}, {0xc021ea2f40, 0x1d}, {0xfa1a60, 0xc00a432c20}, {0x203002, 0x203002})\n\t/go/pkg/mod/storj.io/common@v0.0.0-20221024150824-a2a5c611dacf/pb/piecestore2_drpc.pb.go:220 +0xb0\nstorj.io/drpc/drpcmux.(*Mux).HandleRPC(0xc0006c8a80, {0x13d6220, 0xc00a432c20}, {0xc021ea2f40, 0x1d})\n\t/go/pkg/mod/storj.io/drpc@v0.0.32/drpcmux/handle_rpc.go:33 +0x22d\nstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC(0xc00011e3c0, {0x13d61a0, 0xc00a432ac0}, {0xc021ea2f40, 0x1d})\n\t/go/pkg/mod/storj.io/common@v0.0.0-20221024150824-a2a5c611dacf/rpc/rpctracing/handler.go:61 +0x30b\nstorj.io/common/experiment.(*Handler).HandleRPC(0xc00011c650, {0x13d62a0, 0xc00f1f4b40}, {0xc021ea2f40, 0x1d})\n\t/go/pkg/mod/storj.io/common@v0.0.0-20221024150824-a2a5c611dacf/experiment/import.go:42 +0x170\nstorj.io/drpc/drpcserver.(*Server).handleRPC(0xc033fc3180, 0x13ccf40, {0xc021ea2f40, 0x1b872b8})\n\t/go/pkg/mod/storj.io/drpc@v0.0.32/drpcserver/server.go:122 +0x42\nstorj.io/drpc/drpcserver.(*Server).ServeOne(0xc00018a0e0, {0x13cd640, 0xc00053a2a0}, {0x7f4dd171ff20, 0xc014647880})\n\t/go/pkg/mod/storj.io/drpc@v0.0.32/drpcserver/server.go:66 +0x1f3\nstorj.io/drpc/drpcserver.(*Server).Serve.func2({0x13cd640, 0xc00053a2a0})\n\t/go/pkg/mod/storj.io/drpc@v0.0.32/drpcserver/server.go:112 +0x58\nstorj.io/drpc/drpcctx.(*Tracker).track(0xc00053a2a0, 0xc01fd91f70)\n\t/go/pkg/mod/storj.io/drpc@v0.0.32/drpcctx/transport.go:52 +0x2b\ncreated by storj.io/drpc/drpcctx.(*Tracker).Run\n\t/go/pkg/mod/storj.io/drpc@v0.0.32/drpcctx/transport.go:47 +0x92\n\ngoroutine 248452 [semacquire, 52 minutes]:\nsync.runtime_SemacquireMutex(0xc006b80e68, 0xaa, 0x18)\n\t/usr/local/go/src/runtime/sema.go:71 +0x25\nsync.(*Mutex).lockSlow(0xc000789a88)\n\t/usr/local/go/src/sync/mutex.go:138 +0x165\nsync.(*Mutex).Lock(...)\n\t/usr/local/go/src/sync/mutex.go:81\nstorj.io/storj/storagenode/orders.(*FileStore).BeginEnqueue.func1(0xc01ab02d70)\n\t/go/src/storj.io/storj/storagenode/orders/store.go:94 +0x94\nstorj.io/storj/storagenode/piecestore.(*Endpoint).beginSaveOrder.func1({0x13cd368, 0xc0057e21e0}, 0xc02df36ff0)\n\t/go/src/storj.io/storj/storagenode/piecestore/endpoint.go:726 +0x106\nstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload(0xc00018c000, {0x13dabf0, 0xc014f709b0})\n\t/go/src/storj.io/storj/storagenode/piecestore/endpoint.go:443 +0x1fe0\nstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1({0x102bca0, 0xc00018c000}, {0xc01ba094e0, 0x1d}, {0xfa1a60, 0xc03d12d440}, {0x20300f, 0x20300f})\n\t/go/pkg/mod/storj.io/common@v0.0.0-20221024150824-a2a5c611dacf/pb/piecestore2_drpc.pb.go:220 +0xb0\nstorj.io/drpc/drpcmux.(*Mux).HandleRPC(0xc0006c8a80, {0x13d6220, 0xc03d12d440}, {0xc01ba094e0, 0x1d})\n\t/go/pkg/mod/storj.io/drpc@v0.0.32/drpcmux/handle_rpc.go:33 +0x22d\nstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC(0xc00011e3c0, {0x13d61a0, 0xc03d12d400}, {0xc01ba094e0, 0x1d})\n\t/go/pkg/mod/storj.io/common@v0.0.0-20221024150824-a2a5c611dacf/rpc/rpctracing/handler.go:61 +0x30b\nstorj.io/common/experiment.(*Handler).HandleRPC(0xc00011c650, {0x13d62a0, 0xc01682f680}, {0xc01ba094e0, 0x1d})\n\t/go/pkg/mod/storj.io/common@v0.0.0-20221024150824-a2a5c611dacf/experiment/import.go:42 +0x170\nstorj.io/drpc/drpcserver.(*Server).handleRPC(0xc01946b6c0, 0x13ccf40, {0xc01ba094e0, 0x1b872b8})\n\t/go/pkg/mod/storj.io/drpc@v0.0.32/drpcserver/server.go:122 +0x42\nstorj.io/drpc/drpcserver.(*Server).ServeOne(0xc00018a0e0, {0x13cd640, 0xc00053a2a0}, {0x7f4dd171ff20, 0xc039c3d880})\n\t/go/pkg/mod/storj.io/drpc@v0.0.32/drpcserver/server.go:66 +0x1f3\nstorj.io/drpc/drpcserver.(*Server).Serve.func2({0x13cd640, 0xc00053a2a0})\n\t/go/pkg/mod/storj.io/drpc@v0.0.32/drpcserver/server.go:112 +0x58\nstorj.io/drpc/drpcctx.(*Tracker).track(0xc00053a2a0, 0xc00d9f9210)\n\t/go/pkg/mod/storj.io/drpc@v0.0.32/drpcctx/transport.go:52 +0x2b\ncreated by storj.io/drpc/drpcctx.(*Tracker).Run\n\t/go/pkg/mod/storj.io/drpc@v0.0.32/drpcctx/transport.go:47 +0x92\n\ngoroutine 228 [select, 249 minutes]:\ndatabase/sql.(*DB).connectionOpener(0xc0003fc270, {0x13cce98, 0xc00020a280})\n\t/usr/local/go/src/database/sql/sql.go:1196 +0x93\ncreated by database/sql.OpenDB\n\t/usr/local/go/src/database/sql/sql.go:794 +0x188\n\ngoroutine 269 [select, 249 minutes]:\ndatabase/sql.(*DB).connectionOpener(0xc000372820, {0x13cce98, 0xc000137680})\n\t/usr/local/go/src/database/sql/sql.go:1196 +0x93\ncreated by database/sql.OpenDB\n\t/usr/local/go/src/database/sql/sql.go:794 +0x188\n\ngoroutine 226 [select, 249 minutes]:\ndatabase/sql.(*DB).connectionOpener(0xc0003fc000, {0x13cce98, 0xc00020a0c0})\n\t/usr/local/go/src/database/sql/sql.go:1196 +0x93\ncreated by database/sql.OpenDB\n\t/usr/local/go/src/database/sql/sql.go:794 +0x188\n\ngoroutine 964 [sleep, 10 minutes]:\ntime.Sleep(0x8bb2c97000)\n\t/usr/local/go/src/runtime/time.go:193 +0x12e\ngithub.com/spacemonkeygo/monkit/v3.(*ticker).run(0x1b53de0)\n\t/go/pkg/mod/github.com/spacemonkeygo/monkit/v3@v3.0.19/meter.go:203 +0x2c\ncreated by github.com/spacemonkeygo/monkit/v3.(*ticker).register\n\t/go/pkg/mod/github.com/spacemonkeygo/monkit/v3@v3.0.19/meter.go:195 +0x95\n\ngoroutine 363076 [syscall, 2 minutes]:\nsyscall.Syscall(0x107, 0xffffffffffffff9c, 0xc0cd9f40f0, 0x0)\n\t/usr/local/go/src/syscall/asm_linux_amd64.s:20 +0x5\nsyscall.unlinkat(0xf22840, {0xc00b572de0, 0x0}, 0x0)\n\t/usr/local/go/src/syscall/zsyscall_linux_amd64.go:127 +0xd4\nsyscall.Unlink(...)\n\t/usr/local/go/src/syscall/syscall_linux.go:185\nos.Remove.func1(...)\n\t/usr/local/go/src/os/file_unix.go:302\nos.ignoringEINTR(...)\n\t/usr/local/go/src/os/file_posix.go:246\nos.Remove({0xc00b572de0, 0x0})\n\t/usr/local/go/src/os/file_unix.go:301 +0x45\nstorj.io/storj/storage/filestore.(*blobWriter).Cancel(0xc01bc305a0, {0x13cd368, 0xc0cd9ebd60})\n\t/go/src/storj.io/storj/storage/filestore/blob.go:102 +0x1fb\nstorj.io/storj/storagenode/pieces.(*Writer).Cancel(0xc01be5e7e0, {0x13cd368, 0xc019958d20})\n\t/go/src/storj.io/storj/storagenode/pieces/readwrite.go:209 +0x1a8\nstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload.func4()\n\t/go/src/storj.io/storj/storagenode/piecestore/endpoint.go:310 +0x46\nstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload(0xc00018c000, {0x13dabf0, 0xc01bdd63f0})\n\t/go/src/storj.io/storj/storagenode/piecestore/endpoint.go:323 +0x1110\nstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1({0x102bca0, 0xc00018c000}, {0xc03789c8a0, 0x1d}, {0xfa1a60, 0xc031398540}, {0x20300c, 0x20300c})\n\t/go/pkg/mod/storj.io/common@v0.0.0-20221024150824-a2a5c611dacf/pb/piecestore2_drpc.pb.go:220 +0xb0\nstorj.io/drpc/drpcmux.(*Mux).HandleRPC(0xc0006c8a80, {0x13d6220, 0xc031398540}, {0xc03789c8a0, 0x1d})\n\t/go/pkg/mod/storj.io/drpc@v0.0.32/drpcmux/handle_rpc.go:33 +0x22d\nstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC(0xc00011e3c0, {0x13d61a0, 0xc031398500}, {0xc03789c8a0, 0x1d})\n\t/go/pkg/mod/storj.io/common@v0.0.0-20221024150824-a2a5c611dacf/rpc/rpctracing/handler.go:61 +0x30b\nstorj.io/common/experiment.(*Handler).HandleRPC(0xc00011c650, {0x13d62a0, 0xc03cc45d40}, {0xc03789c8a0, 0x1d})\n\t/go/pkg/mod/storj.io/common@v0.0.0-20221024150824-a2a5c611dacf/experiment/import.go:42 +0x170\nstorj.io/drpc/drpcserver.(*Server).handleRPC(0xc038100e00, 0x13ccf40, {0xc03789c8a0, 0x1b872b8})\n\t/go/pkg/mod/storj.io/drpc@v0.0.32/drpcserver/server.go:122 +0x42\nstorj.io/drpc/drpcserver.(*Server).ServeOne(0xc00018a0e0, {0x13cd640, 0xc00053a2a0}, {0x7f4dd171ff20, 0xc0371b7880})\n\t/go/pkg/mod/storj.io/drpc@v0.0.32/drpcserver/server.go:66 +0x1f3\nstorj.io/drpc/drpcserver.(*Server).Serve.func2({0x13cd640, 0xc00053a2a0})\n\t/go/pkg/mod/storj.io/drpc@v0.0.32/drpcserver/server.go:112 +0x58\nstorj.io/drpc/drpcctx.(*Tracker).track(0xc00053a2a0, 0xc0002006e0)\n\t/go/pkg/mod/storj.io/drpc@v0.0.32/drpcctx/transport.go:52 +0x2b\ncreated by storj.io/drpc/drpcctx.(*Tracker).Run\n\t/go/pkg/mod/storj.io/drpc@v0.0.32/drpcctx/transport.go:47 +0x92\n\ngoroutine 72 [select, 10 minutes]:\ndatabase/sql.(*DB).connectionCleaner(0xc00027e000, 0xc000553d50)\n\t/usr/local/go/src/database/sql/sql.go:1068 +0xbd\ncreated by database/sql.(*DB).startCleanerLocked\n\t/usr/local/go/src/database/sql/sql.go:1055 +0x105\n\ngoroutine 178848 [semacquire, 92 minutes]:\nsync.runtime_SemacquireMutex(0xc02153ee68, 0xaa, 0x18)\n\t/usr/local/go/src/runtime/sema.go:71 +0x25\nsync.(*Mutex).lockSlow(0xc000789a88)\n\t/usr/local/go/src/sync/mutex.go:138 +0x165\nsync.(*Mutex).Lock(...)\n\t/usr/local/go/src/sync/mutex.go:81\nstorj.io/storj/storagenode/orders.(*FileStore).BeginEnqueue.func1(0xc06b9e5660)\n\t/go/src/storj.io/storj/storagenode/orders/store.go:94 +0x94\nstorj.io/storj/storagenode/piecestore.(*Endpoint).beginSaveOrder.func1({0x13cd368, 0xc01ac403c0}, 0xc0229084b0)\n\t/go/src/storj.io/storj/storagenode/piecestore/endpoint.go:726 +0x106\nstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload(0xc00018c000, {0x13dabf0, 0xc0166f7c80})\n\t/go/src/storj.io/storj/storagenode/piecestore/endpoint.go:443 +0x1fe0\nstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1({0x102bca0, 0xc00018c000}, {0xc0062528a0, 0x1d}, {0xfa1a60, 0xc0055925a0}, {0x203001, 0x203001})\n\t/go/pkg/mod/storj.io/common@v0.0.0-20221024150824-a2a5c611dacf/pb/piecestore2_drpc.pb.go:220 +0xb0\nstorj.io/drpc/drpcmux.(*Mux).HandleRPC(0xc0006c8a80, {0x13d6220, 0xc0055925a0}, {0xc0062528a0, 0x1d})\n\t/go/pkg/mod/storj.io/drpc@v0.0.32/drpcmux/handle_rpc.go:33 +0x22d\nstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC(0xc00011e3c0, {0x13d61a0, 0xc005592560}, {0xc0062528a0, 0x1d})\n\t/go/pkg/mod/storj.io/common@v0.0.0-20221024150824-a2a5c611dacf/rpc/rpctracing/handler.go:61 +0x30b\nstorj.io/common/experiment.(*Handler).HandleRPC(0xc00011c650, {0x13d62a0, 0xc00ac59680}, {0xc0062528a0, 0x1d})\n\t/go/pkg/mod/storj.io/common@v0.0.0-20221024150824-a2a5c611dacf/experiment/import.go:42 +0x170\nstorj.io/drpc/drpcserver.(*Server).handleRPC(0xc00bd8fa40, 0x13ccf40, {0xc0062528a0, 0x1b872b8})\n\t/go/pkg/mod/storj.io/drpc@v0.0.32/drpcserver/server.go:122 +0x42\nstorj.io/drpc/drpcserver.(*Server).ServeOne(0xc00018a0e0, {0x13cd640, 0xc00053a2a0}, {0x7f4dd171ff20, 0xc004757500})\n\t/go/pkg/mod/storj.io/drpc@v0.0.32/drpcserver/server.go:66 +0x1f3\nstorj.io/drpc/drpcserver.(*Server).Serve.func2({0x13cd640, 0xc00053a2a0})\n\t/go/pkg/mod/storj.io/drpc@v0.0.32/drpcserver/server.go:112 +0x58\nstorj.io/drpc/drpcctx.(*Tracker).track(0xc00053a2a0, 0xc01f4e96e0)\n\t/go/pkg/mod/storj.io/drpc@v0.0.32/drpcctx/transport.go:52 +0x2b\ncreated by storj.io/drpc/drpcctx.(*Tracker).Run\n\t/go/pkg/mod/storj.io/drpc@v0.0.32/drpcctx/transport.go:47 +0x92\n\ngoroutine 245865 [semacquire, 52 minutes]:\nsync.runtime_SemacquireMutex(0xc094566e68, 0xaa, 0x18)\n\t/usr/local/go/src/runtime/sema.go:71 +0x25\nsync.(*Mutex).lockSlow(0xc000789a88)\n\t/usr/local/go/src/sync/mutex.go:138 +0x165\nsync.(*Mutex).Lock(...)\n\t/usr/local/go/src/sync/mutex.go:81\nstorj.io/storj/storagenode/orders.(*FileStore).BeginEnqueue.func1(0xc0787f4160)\n\t/go/src/storj.io/storj/storagenode/orders/store.go:94 +0x94\nstorj.io/storj/storagenode/piecestore.(*Endpoint).beginSaveOrder.func1({0x13cd368, 0xc021de17c0}, 0xc02df36e10)\n\t/go/src/storj.io/storj/storagenode/piecestore/endpoint.go:726 +0x106\nstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload(0xc00018c000, {0x13dabf0, 0xc027e0c410})\n\t/go/src/storj.io/storj/storagenode/piecestore/endpoint.go:443 +0x1fe0\nstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1({0x102bca0, 0xc00018c000}, {0xc03a357920, 0x1d}, {0xfa1a60, 0xc027e10b00}, {0x203009, 0x203009})\n\t/go/pkg/mod/storj.io/common@v0.0.0-20221024150824-a2a5c611dacf/pb/piecestore2_drpc.pb.go:220 +0xb0\nstorj.io/drpc/drpcmux.(*Mux).HandleRPC(0xc0006c8a80, {0x13d6220, 0xc027e10b00}, {0xc03a357920, 0x1d})\n\t/go/pkg/mod/storj.io/drpc@v0.0.32/drpcmux/handle_rpc.go:33 +0x22d\nstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC(0xc00011e3c0, {0x13d61a0, 0xc027e10ac0}, {0xc03a357920, 0x1d})\n\t/go/pkg/mod/storj.io/common@v0.0.0-20221024150824-a2a5c611dacf/rpc/rpctracing/handler.go:61 +0x30b\nstorj.io/common/experiment.(*Handler).HandleRPC(0xc00011c650, {0x13d62a0, 0xc023112900}, {0xc03a357920, 0x1d})\n\t/go/pkg/mod/storj.io/common@v0.0.0-20221024150824-a2a5c611dacf/experiment/import.go:42 +0x170\nstorj.io/drpc/drpcserver.(*Server).handleRPC(0xc00c8ee540, 0x13ccf40, {0xc03a357920, 0x1b872b8})\n\t/go/pkg/mod/storj.io/drpc@v0.0.32/drpcserver/server.go:122 +0x42\nstorj.io/drpc/drpcserver.(*Server).ServeOne(0xc00018a0e0, {0x13cd640, 0xc00053a2a0}, {0x7f4dd171ff20, 0xc0237a1500})\n\t/go/pkg/mod/storj.io/drpc@v0.0.32/drpcserver/server.go:66 +0x1f3\nstorj.io/drpc/drpcserver.(*Server).Serve.func2({0x13cd640, 0xc00053a2a0})\n\t/go/pkg/mod/storj.io/drpc@v0.0.32/drpcserver/server.go:112 +0x58\nstorj.io/drpc/drpcctx.(*Tracker).track(0xc00053a2a0, 0xc02a2b6390)\n\t/go/pkg/mod/storj.io/drpc@v0.0.32/drpcctx/transport.go:52 +0x2b\ncreated by storj.io/drpc/drpcctx.(*Tracker).Run\n\t/go/pkg/mod/storj.io/drpc@v0.0.32/drpcctx/transport.go:47 +0x92\n\ngoroutine 244194 [semacquire, 52 minutes]:\nsync.runtime_SemacquireMutex(0xc0009eee68, 0xaa, 0x18)\n\t/usr/local/go/src/runtime/sema.go:71 +0x25\nsync.(*Mutex).lockSlow(0xc000789a88)\n\t/usr/local/go/src/sync/mutex.go:138 +0x165\nsync.(*Mutex).Lock(...)\n\t/usr/local/go/src/sync/mutex.go:81\nstorj.io/storj/storagenode/orders.(*FileStore).BeginEnqueue.func1(0xc089289d10)\n\t/go/src/storj.io/storj/storagenode/orders/store.go:94 +0x94\nstorj.io/storj/storagenode/piecestore.(*Endpoint).beginSaveOrder.func1({0x13cd368, 0xc00c1df5e0}, 0xc01e40fb80)\n\t/go/src/storj.io/storj/storagenode/piecestore/endpoint.go:726 +0x106\nstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload(0xc00018c000, {0x13dabf0, 0xc0219c18d0})\n\t/go/src/storj.io/storj/storagenode/piecestore/endpoint.go:443 +0x1fe0\nstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1({0x102bca0, 0xc00018c000}, {0xc0158c5d20, 0x1d}, {0xfa1a60, 0xc025ad8480}, {0x203009, 0x203009})\n\t/go/pkg/mod/storj.io/common@v0.0.0-20221024150824-a2a5c611dacf/pb/piecestore2_drpc.pb.go:220 +0xb0\nstorj.io/drpc/drpcmux.(*Mux).HandleRPC(0xc0006c8a80, {0x13d6220, 0xc025ad8480}, {0xc0158c5d20, 0x1d})\n\t/go/pkg/mod/storj.io/drpc@v0.0.32/drpcmux/handle_rpc.go:33 +0x22d\nstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC(0xc00011e3c0, {0x13d61a0, 0xc025ad8420}, {0xc0158c5d20, 0x1d})\n\t/go/pkg/mod/storj.io/common@v0.0.0-20221024150824-a2a5c611dacf/rpc/rpctracing/handler.go:61 +0x30b\nstorj.io/common/experiment.(*Handler).HandleRPC(0xc00011c650, {0x13d62a0, 0xc023112240}, {0xc0158c5d20, 0x1d})\n\t/go/pkg/mod/storj.io/common@v0.0.0-20221024150824-a2a5c611dacf/experiment/import.go:42 +0x170\nstorj.io/drpc/drpcserver.(*Server).handleRPC(0xc01668e540, 0x13ccf40, {0xc0158c5d20, 0x1b872b8})\n\t/go/pkg/mod/storj.io/drpc@v0.0.32/drpcserver/server.go:122 +0x42\nstorj.io/drpc/drpcserver.(*Server).ServeOne(0xc00018a0e0, {0x13cd640, 0xc00053a2a0}, {0x7f4dd171ff20, 0xc0109f2000})\n\t/go/pkg/mod/storj.io/drpc@v0.0.32/drpcserver/server.go:66 +0x1f3\nstorj.io/drpc/drpcserver.(*Server).Serve.func2({0x13cd640, 0xc00053a2a0})\n\t/go/pkg/mod/storj.io/drpc@v0.0.32/drpcserver/server.go:112 +0x58\nstorj.io/drpc/drpcctx.(*Tracker).track(0xc00053a2a0, 0xc01ab020c0)\n\t/go/pkg/mod/storj.io/drpc@v0.0.32/drpcctx/transport.go:52 +0x2b\ncreated by storj.io/drpc/drpcctx.(*Tracker).Run\n\t/go/pkg/mod/storj.io/drpc@v0.0.32/drpcctx/transport.go:47 +0x92\n\ngoroutine 324516 [semacquire, 30 minutes]:\nsync.runtime_SemacquireMutex(0xc0aa322e68, 0xaa, 0x18)\n\t/usr/local/go/src/runtime/sema.go:71 +0x25\nsync.(*Mutex).lockSlow(0xc000789a88)\n\t/usr/local/go/src/sync/mutex.go:138 +0x165\nsync.(*Mutex).Lock(...)\n\t/usr/local/go/src/sync/mutex.go:81\nstorj.io/storj/storagenode/orders.(*FileStore).BeginEnqueue.func1(0xc01b433090)\n\t/go/src/storj.io/storj/storagenode/orders/store.go:94 +0x94\nstorj.io/storj/storagenode/piecestore.(*Endpoint).beginSaveOrder.func1({0x13cd368, 0xc03080bcc0}, 0xc031e75950)\n\t/go/src/storj.io/storj/storagenode/piecestore/endpoint.go:726 +0x106\nstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload(0xc00018c000, {0x13dabf0, 0xc00601e020})\n\t/go/src/storj.io/storj/storagenode/piecestore/endpoint.go:443 +0x1fe0\nstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1({0x102bca0, 0xc00018c000}, {0xc0235b03a0, 0x1d}, {0xfa1a60, 0xc041628bc0}, {0x203010, 0x203010})\n\t/go/pkg/mod/storj.io/common@v0.0.0-20221024150824-a2a5c611dacf/pb/piecestore2_drpc.pb.go:220 +0xb0\nstorj.io/drpc/drpcmux.(*Mux).HandleRPC(0xc0006c8a80, {0x13d6220, 0xc041628bc0}, {0xc0235b03a0, 0x1d})\n\t/go/pkg/mod/storj.io/drpc@v0.0.32/drpcmux/handle_rpc.go:33 +0x22d\nstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC(0xc00011e3c0, {0x13d61a0, 0xc041628a40}, {0xc0235b03a0, 0x1d})\n\t/go/pkg/mod/storj.io/common@v0.0.0-20221024150824-a2a5c611dacf/rpc/rpctracing/handler.go:61 +0x30b\nstorj.io/common/experiment.(*Handler).HandleRPC(0xc00011c650, {0x13d62a0, 0xc007db66c0}, {0xc0235b03a0, 0x1d})\n\t/go/pkg/mod/storj.io/common@v0.0.0-20221024150824-a2a5c611dacf/experiment/import.go:42 +0x170\nstorj.io/drpc/drpcserver.(*Server).handleRPC(0xc009f4ba40, 0x13ccf40, {0xc0235b03a0, 0x1b872b8})\n\t/go/pkg/mod/storj.io/drpc@v0.0.32/drpcserver/server.go:122 +0x42\nstorj.io/drpc/drpcserver.(*Server).ServeOne(0xc00018a0e0, {0x13cd640, 0xc00053a2a0}, 

Yeah I already feared that this may be a problem. I got more ram coming in 2 Days and hope this fixes the issue. Is there a way to configure a cache ssd to temporarily hold the files?

most of the io workload can be cached, it helps a great deal…
adding more ram beyond 2-4GB pr node is useless.

only solution is to either speed up the storage, or moving the databases to an ssd which is also a feature that can be used, but isn’t used by default.

the ram is just a buffer, it will fill eventually… more does however help even out the issues when storage or cpu cannot keep up… then ram will fill until it can… or it crashes…
a node that is underpower can eat just about every GB of RAM you can throw at it.

1 Like

(Removed: Apologies, I didn’t see your previous message.)

Please try to sync your clock first. If the problem is gone - then it was it.
If not - try to replace the image

docker stop -t 300 storagenode
docker rm storagenode

do the same for all your docker nodes
Then remove the image

docker image prune

Then download a latest one

docker pull storjlabs/storagenode:latest

And run your nodes back.

If crashing is happened only for one of your nodes - then please check its disk for errors and fix them.

1 Like

After adding more Ram to the System everything seems to be fine.
Thanks for your help everyone :slight_smile:

1 Like