Getting Put and Get on all Nodes

[batelis]

So I get alot of put and get errors in logs for all of my nodes they crash after like 12h of running time, cant seem to find any new information about put and get errors so lets talk about it?
Logs of past few hours

2024-10-11T05:23:22+03:00	ERROR	piecestore	upload failed	{"Piece ID": "274XERN5JVETEI6EM2HYJ6NDLPHSDPIKRLOOZK7NFEOKRX5R3LRQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT", "Remote Address": "79.127.226.103:45664", "Size": 1835008, "error": "unexpected EOF", "errorVerbose": "unexpected EOF\n\tstorj.io/common/rpc/rpcstatus.Error:85\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:574\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:294\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:62\n\tstorj.io/common/experiment.(*Handler).HandleRPC:43\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:166\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:108\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:156\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35"}
2024-10-11T05:24:59+03:00	ERROR	piecestore	download failed	{"Piece ID": "NKXDKS2RJJUA7HZJH25E7ZVDJ226ROXHD2WQGHJJMUY5GDWBN6YA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "Offset": 0, "Size": 137472, "Remote Address": "79.127.219.34:41926", "error": "write tcp 192.168.1.245:28967->79.127.219.34:41926: wsasend: An existing connection was forcibly closed by the remote host.", "errorVerbose": "write tcp 192.168.1.245:28967->79.127.219.34:41926: wsasend: An existing connection was forcibly closed by the remote host.\n\tstorj.io/drpc/drpcstream.(*Stream).rawFlushLocked:428\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:489\n\tstorj.io/common/pb.(*drpcPiecestore_DownloadStream).Send:408\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).sendData.func1:946\n\tstorj.io/storj/storagenode/piecestore.withTimeout[...]:1224\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).sendData:944\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func7:844\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78"}
2024-10-11T05:25:01+03:00	ERROR	piecestore	upload internal error	{"error": "EOF"}
2024-10-11T05:25:09+03:00	ERROR	piecestore	download failed	{"Piece ID": "NKXDKS2RJJUA7HZJH25E7ZVDJ226ROXHD2WQGHJJMUY5GDWBN6YA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "Offset": 0, "Size": 137472, "Remote Address": "109.61.92.72:39082", "error": "write tcp 192.168.1.245:28967->109.61.92.72:39082: wsasend: An existing connection was forcibly closed by the remote host.", "errorVerbose": "write tcp 192.168.1.245:28967->109.61.92.72:39082: wsasend: An existing connection was forcibly closed by the remote host.\n\tstorj.io/drpc/drpcstream.(*Stream).rawFlushLocked:428\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:489\n\tstorj.io/common/pb.(*drpcPiecestore_DownloadStream).Send:408\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).sendData.func1:946\n\tstorj.io/storj/storagenode/piecestore.withTimeout[...]:1224\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).sendData:944\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func7:844\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78"}
2024-10-11T05:30:28+03:00	ERROR	piecestore	upload failed	{"Piece ID": "U4LNYV2OXUM2DLL7NVPN7IBOVUXLU6MQYKPKDTDXT5RVRKXNGE7A", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT", "Remote Address": "37.138.114.8:44980", "Size": 1245184, "error": "unexpected EOF", "errorVerbose": "unexpected EOF\n\tstorj.io/common/rpc/rpcstatus.Error:85\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:574\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:294\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:62\n\tstorj.io/common/experiment.(*Handler).HandleRPC:43\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:166\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:108\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:156\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35"}
2024-10-11T05:37:15+03:00	ERROR	piecestore	upload failed	{"Piece ID": "RQQTDNRLWULFMKVMXAKX2SPFS7ATV5NNCFWHC62NR4GJNCDUEJYQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Remote Address": "79.127.205.229:40026", "Size": 196608, "error": "unexpected EOF", "errorVerbose": "unexpected EOF\n\tstorj.io/common/rpc/rpcstatus.Error:85\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:574\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:294\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:62\n\tstorj.io/common/experiment.(*Handler).HandleRPC:43\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:166\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:108\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:156\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35"}
2024-10-11T05:39:43+03:00	ERROR	piecestore	upload failed	{"Piece ID": "2B7BYP7GKCT4RFMUIRTSZYJWHSF355AJ7MU63OMZQXHHBKRDNZBA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT_REPAIR", "Remote Address": "199.102.71.25:44098", "Size": 196608, "error": "unexpected EOF", "errorVerbose": "unexpected EOF\n\tstorj.io/common/rpc/rpcstatus.Error:85\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:574\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:294\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:62\n\tstorj.io/common/experiment.(*Handler).HandleRPC:43\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:166\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:108\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:156\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35"}
2024-10-11T05:43:11+03:00	ERROR	piecestore	download failed	{"Piece ID": "NKXDKS2RJJUA7HZJH25E7ZVDJ226ROXHD2WQGHJJMUY5GDWBN6YA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "Offset": 0, "Size": 137472, "Remote Address": "109.61.92.71:47136", "error": "write tcp 192.168.1.245:28967->109.61.92.71:47136: wsasend: An existing connection was forcibly closed by the remote host.", "errorVerbose": "write tcp 192.168.1.245:28967->109.61.92.71:47136: wsasend: An existing connection was forcibly closed by the remote host.\n\tstorj.io/drpc/drpcstream.(*Stream).rawFlushLocked:428\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:489\n\tstorj.io/common/pb.(*drpcPiecestore_DownloadStream).Send:408\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).sendData.func1:946\n\tstorj.io/storj/storagenode/piecestore.withTimeout[...]:1224\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).sendData:944\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func7:844\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78"}
2024-10-11T05:47:13+03:00	ERROR	piecestore	upload failed	{"Piece ID": "WVJNQQWJIWZPKLGQLL4J4GTD7SOZYPSDP2MNJ7UKGDP3VI77AK3Q", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Remote Address": "79.127.219.40:44672", "Size": 131072, "error": "unexpected EOF", "errorVerbose": "unexpected EOF\n\tstorj.io/common/rpc/rpcstatus.Error:85\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:574\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:294\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:62\n\tstorj.io/common/experiment.(*Handler).HandleRPC:43\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:166\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:108\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:156\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35"}
2024-10-11T05:51:45+03:00	ERROR	piecestore	upload failed	{"Piece ID": "QC46W42NF5QYDMGSFSCTIXVBWQBSPO7LNZBQUM4LJTT3IVO6LHBA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT_REPAIR", "Remote Address": "199.102.71.64:53436", "Size": 1769472, "error": "unexpected EOF", "errorVerbose": "unexpected EOF\n\tstorj.io/common/rpc/rpcstatus.Error:85\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:574\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:294\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:62\n\tstorj.io/common/experiment.(*Handler).HandleRPC:43\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:166\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:108\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:156\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35"}
2024-10-11T06:01:58+03:00	ERROR	piecestore	download failed	{"Piece ID": "NKXDKS2RJJUA7HZJH25E7ZVDJ226ROXHD2WQGHJJMUY5GDWBN6YA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "Offset": 0, "Size": 137472, "Remote Address": "109.61.92.80:40806", "error": "write tcp 192.168.1.245:28967->109.61.92.80:40806: wsasend: An existing connection was forcibly closed by the remote host.", "errorVerbose": "write tcp 192.168.1.245:28967->109.61.92.80:40806: wsasend: An existing connection was forcibly closed by the remote host.\n\tstorj.io/drpc/drpcstream.(*Stream).rawFlushLocked:428\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:489\n\tstorj.io/common/pb.(*drpcPiecestore_DownloadStream).Send:408\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).sendData.func1:946\n\tstorj.io/storj/storagenode/piecestore.withTimeout[...]:1224\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).sendData:944\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func7:844\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78"}
2024-10-11T06:07:57+03:00	ERROR	piecestore	download failed	{"Piece ID": "GNP4G5ME5VMVIHCCZIP4ZDHLQK77LOCQEPDUVJIYONLJVOSAZHEQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "Offset": 0, "Size": 2319104, "Remote Address": "103.214.68.73:58250", "error": "write tcp 192.168.1.245:28967->103.214.68.73:58250: wsasend: An existing connection was forcibly closed by the remote host.", "errorVerbose": "write tcp 192.168.1.245:28967->103.214.68.73:58250: wsasend: An existing connection was forcibly closed by the remote host.\n\tstorj.io/drpc/drpcstream.(*Stream).rawFlushLocked:428\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:489\n\tstorj.io/common/pb.(*drpcPiecestore_DownloadStream).Send:408\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).sendData.func1:946\n\tstorj.io/storj/storagenode/piecestore.withTimeout[...]:1224\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).sendData:944\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func7:844\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78"}
2024-10-11T06:10:23+03:00	ERROR	piecestore	download failed	{"Piece ID": "NKXDKS2RJJUA7HZJH25E7ZVDJ226ROXHD2WQGHJJMUY5GDWBN6YA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "Offset": 0, "Size": 137472, "Remote Address": "79.127.205.225:57502", "error": "write tcp 192.168.1.245:28967->79.127.205.225:57502: wsasend: An existing connection was forcibly closed by the remote host.", "errorVerbose": "write tcp 192.168.1.245:28967->79.127.205.225:57502: wsasend: An existing connection was forcibly closed by the remote host.\n\tstorj.io/drpc/drpcstream.(*Stream).rawFlushLocked:428\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:489\n\tstorj.io/common/pb.(*drpcPiecestore_DownloadStream).Send:408\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).sendData.func1:946\n\tstorj.io/storj/storagenode/piecestore.withTimeout[...]:1224\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).sendData:944\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func7:844\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78"}
2024-10-11T06:12:54+03:00	ERROR	piecestore	download failed	{"Piece ID": "NKXDKS2RJJUA7HZJH25E7ZVDJ226ROXHD2WQGHJJMUY5GDWBN6YA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "Offset": 0, "Size": 137472, "Remote Address": "79.127.205.226:58078", "error": "write tcp 192.168.1.245:28967->79.127.205.226:58078: wsasend: An existing connection was forcibly closed by the remote host.", "errorVerbose": "write tcp 192.168.1.245:28967->79.127.205.226:58078: wsasend: An existing connection was forcibly closed by the remote host.\n\tstorj.io/drpc/drpcstream.(*Stream).rawFlushLocked:428\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:489\n\tstorj.io/common/pb.(*drpcPiecestore_DownloadStream).Send:408\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).sendData.func1:946\n\tstorj.io/storj/storagenode/piecestore.withTimeout[...]:1224\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).sendData:944\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func7:844\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78"}
2024-10-11T06:13:12+03:00	ERROR	piecestore	download failed	{"Piece ID": "CJJDW5SUFCSLCQMX66JFOMDX3FTWJH5VELCF2OYX3353YTS4J4KA", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET", "Offset": 0, "Size": 2319104, "Remote Address": "103.214.68.73:40578", "error": "write tcp 192.168.1.245:28967->103.214.68.73:40578: wsasend: An existing connection was forcibly closed by the remote host.", "errorVerbose": "write tcp 192.168.1.245:28967->103.214.68.73:40578: wsasend: An existing connection was forcibly closed by the remote host.\n\tstorj.io/drpc/drpcstream.(*Stream).rawFlushLocked:428\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:489\n\tstorj.io/common/pb.(*drpcPiecestore_DownloadStream).Send:408\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).sendData.func1:946\n\tstorj.io/storj/storagenode/piecestore.withTimeout[...]:1224\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).sendData:944\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func7:844\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78"}
2024-10-11T06:15:11+03:00	ERROR	piecestore	download failed	{"Piece ID": "KQDWXYDFYGW4PN3XQOPI4CXWH527D6WSBQXR5UVNSWS3KD6DYRUA", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET", "Offset": 0, "Size": 2319104, "Remote Address": "45.86.229.103:44026", "error": "write tcp 192.168.1.245:28967->45.86.229.103:44026: wsasend: An existing connection was forcibly closed by the remote host.", "errorVerbose": "write tcp 192.168.1.245:28967->45.86.229.103:44026: wsasend: An existing connection was forcibly closed by the remote host.\n\tstorj.io/drpc/drpcstream.(*Stream).rawFlushLocked:428\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:489\n\tstorj.io/common/pb.(*drpcPiecestore_DownloadStream).Send:408\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).sendData.func1:946\n\tstorj.io/storj/storagenode/piecestore.withTimeout[...]:1224\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).sendData:944\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func7:844\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78"}
2024-10-11T06:22:25+03:00	ERROR	piecestore	download failed	{"Piece ID": "NKXDKS2RJJUA7HZJH25E7ZVDJ226ROXHD2WQGHJJMUY5GDWBN6YA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "Offset": 0, "Size": 137472, "Remote Address": "79.127.205.227:41682", "error": "write tcp 192.168.1.245:28967->79.127.205.227:41682: wsasend: An existing connection was forcibly closed by the remote host.", "errorVerbose": "write tcp 192.168.1.245:28967->79.127.205.227:41682: wsasend: An existing connection was forcibly closed by the remote host.\n\tstorj.io/drpc/drpcstream.(*Stream).rawFlushLocked:428\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:489\n\tstorj.io/common/pb.(*drpcPiecestore_DownloadStream).Send:408\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).sendData.func1:946\n\tstorj.io/storj/storagenode/piecestore.withTimeout[...]:1224\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).sendData:944\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func7:844\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78"}
2024-10-11T06:24:33+03:00	ERROR	piecestore	upload internal error	{"error": "EOF"}
2024-10-11T06:27:14+03:00	ERROR	piecestore	upload failed	{"Piece ID": "3R4AGHWS3TQQY2IX6VXU5KESBQMBUBW5U7W344CVPGN27H3YSL5Q", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Remote Address": "79.127.205.231:60506", "Size": 131072, "error": "unexpected EOF", "errorVerbose": "unexpected EOF\n\tstorj.io/common/rpc/rpcstatus.Error:85\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:574\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:294\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:62\n\tstorj.io/common/experiment.(*Handler).HandleRPC:43\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:166\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:108\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:156\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35"}
2024-10-11T06:31:26+03:00	ERROR	piecestore	upload failed	{"Piece ID": "WNKJ3K5X6MNTOMRVNHIN7Z52JTUT3VVIHXACYJKETU5DNMEQWBEA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT_REPAIR", "Remote Address": "199.102.71.57:38654", "Size": 0, "error": "unexpected EOF", "errorVerbose": "unexpected EOF\n\tstorj.io/common/rpc/rpcstatus.Error:85\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:574\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:294\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:62\n\tstorj.io/common/experiment.(*Handler).HandleRPC:43\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:166\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:108\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:156\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35"}
2024-10-11T06:38:39+03:00	ERROR	piecestore	download failed	{"Piece ID": "3FJ7RXTTUH25AY6YUWIWESY6KQOLBORR2XAN4G6ENNMWYCN6ETUA", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET", "Offset": 0, "Size": 2319104, "Remote Address": "103.214.68.73:52132", "error": "write tcp 192.168.1.245:28967->103.214.68.73:52132: wsasend: An existing connection was forcibly closed by the remote host.", "errorVerbose": "write tcp 192.168.1.245:28967->103.214.68.73:52132: wsasend: An existing connection was forcibly closed by the remote host.\n\tstorj.io/drpc/drpcstream.(*Stream).rawFlushLocked:428\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:489\n\tstorj.io/common/pb.(*drpcPiecestore_DownloadStream).Send:408\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).sendData.func1:946\n\tstorj.io/storj/storagenode/piecestore.withTimeout[...]:1224\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).sendData:944\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func7:844\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78"}
2024-10-11T06:38:46+03:00	ERROR	piecestore	upload internal error	{"error": "manager closed: read tcp 192.168.1.245:28967->193.34.77.90:59826: wsarecv: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.", "errorVerbose": "manager closed: read tcp 192.168.1.245:28967->193.34.77.90:59826: wsarecv: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.\n\tgithub.com/jtolio/noiseconn.(*Conn).readMsg:225\n\tgithub.com/jtolio/noiseconn.(*Conn).Read:171\n\tstorj.io/drpc/drpcwire.(*Reader).read:68\n\tstorj.io/drpc/drpcwire.(*Reader).ReadPacketUsing:113\n\tstorj.io/drpc/drpcmanager.(*Manager).manageReader:230"}
2024-10-11T06:38:46+03:00	ERROR	piecestore	upload failed	{"Piece ID": "K7MOSEAWJ5FSTXOSGPJZ2GYIDTHOIFVBR4TYHZ2AKX25X47K2BEQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT", "Remote Address": "193.34.77.90:59826", "Size": 917504, "error": "manager closed: read tcp 192.168.1.245:28967->193.34.77.90:59826: wsarecv: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.", "errorVerbose": "manager closed: read tcp 192.168.1.245:28967->193.34.77.90:59826: wsarecv: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.\n\tgithub.com/jtolio/noiseconn.(*Conn).readMsg:225\n\tgithub.com/jtolio/noiseconn.(*Conn).Read:171\n\tstorj.io/drpc/drpcwire.(*Reader).read:68\n\tstorj.io/drpc/drpcwire.(*Reader).ReadPacketUsing:113\n\tstorj.io/drpc/drpcmanager.(*Manager).manageReader:230"}
2024-10-11T06:40:17+03:00	ERROR	piecestore	upload failed	{"Piece ID": "V76JGDBZCF6I5SZENOG5MGQGYBEGDCXDQUXHXSIQWA27HUP33PQA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Remote Address": "79.127.205.226:59742", "Size": 262144, "error": "unexpected EOF", "errorVerbose": "unexpected EOF\n\tstorj.io/common/rpc/rpcstatus.Error:85\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:574\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:294\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:62\n\tstorj.io/common/experiment.(*Handler).HandleRPC:43\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:166\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:108\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:156\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35"}
2024-10-11T06:49:54+03:00	ERROR	piecestore	upload failed	{"Piece ID": "P26PHNL5URQADPVHWH57T43KIJZFZMR3JY2YSTPAWBK2JBUNTWGA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Remote Address": "79.127.219.39:52606", "Size": 131072, "error": "unexpected EOF", "errorVerbose": "unexpected EOF\n\tstorj.io/common/rpc/rpcstatus.Error:85\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:574\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:294\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:62\n\tstorj.io/common/experiment.(*Handler).HandleRPC:43\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:166\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:108\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:156\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35"}
2024-10-11T06:52:39+03:00	ERROR	piecestore	upload failed	{"Piece ID": "TOXVZ7WBK5PFLMJOBB6FTSTAURCFMRV6J5XTWQQARE232SSRLSYA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Remote Address": "79.127.201.219:39310", "Size": 262144, "error": "unexpected EOF", "errorVerbose": "unexpected EOF\n\tstorj.io/common/rpc/rpcstatus.Error:85\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:574\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:294\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:62\n\tstorj.io/common/experiment.(*Handler).HandleRPC:43\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:166\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:108\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:156\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35"}
2024-10-11T07:02:19+03:00	ERROR	piecestore	upload failed	{"Piece ID": "JIWYMK6NQMHSS3F5JTTPNHC6HSENLQC6YTETQMMMRPOJK6WFOCPA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Remote Address": "121.127.47.26:56804", "Size": 131072, "error": "unexpected EOF", "errorVerbose": "unexpected EOF\n\tstorj.io/common/rpc/rpcstatus.Error:85\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:574\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:294\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:62\n\tstorj.io/common/experiment.(*Handler).HandleRPC:43\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:166\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:108\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:156\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35"}
2024-10-11T07:10:20+03:00	ERROR	piecestore	download failed	{"Piece ID": "NDCUFVBID23U2QJJXWUL3P5QVG3PENEKZAH2RMXSFJSHCTQKKH5Q", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET", "Offset": 0, "Size": 2319104, "Remote Address": "45.86.229.103:44198", "error": "write tcp 192.168.1.245:28967->45.86.229.103:44198: wsasend: An existing connection was forcibly closed by the remote host.", "errorVerbose": "write tcp 192.168.1.245:28967->45.86.229.103:44198: wsasend: An existing connection was forcibly closed by the remote host.\n\tstorj.io/drpc/drpcstream.(*Stream).rawFlushLocked:428\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:489\n\tstorj.io/common/pb.(*drpcPiecestore_DownloadStream).Send:408\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).sendData.func1:946\n\tstorj.io/storj/storagenode/piecestore.withTimeout[...]:1224\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).sendData:944\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func7:844\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78"}
2024-10-11T07:13:02+03:00	ERROR	piecestore	download failed	{"Piece ID": "4BYVIHAXOAUVSEMTCWEILK2P57SGPMTBDFELORTCWENPDVD3RBPA", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Action": "GET", "Offset": 325632, "Size": 2560, "Remote Address": "79.127.201.217:58218", "error": "context deadline exceeded", "errorVerbose": "context deadline exceeded\n\tstorj.io/common/rpc/rpcstatus.Wrap:76\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func8:904\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:915\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:302\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:62\n\tstorj.io/common/experiment.(*Handler).HandleRPC:43\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:166\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:108\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:156\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35"}
2024-10-11T07:22:53+03:00	ERROR	piecestore	upload failed	{"Piece ID": "H2T6IYSPH6MZ4ZMHG3V4Y5PLH54Y36L7VIB6QXUFR2EBTRWQHSVA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Remote Address": "79.127.205.232:38458", "Size": 196608, "error": "unexpected EOF", "errorVerbose": "unexpected EOF\n\tstorj.io/common/rpc/rpcstatus.Error:85\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:574\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:294\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:62\n\tstorj.io/common/experiment.(*Handler).HandleRPC:43\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:166\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:108\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:156\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35"}
2024-10-11T09:10:11+03:00	ERROR	piecestore	download failed	{"Piece ID": "HQB6MJLLGTUQPGAYQ5WQGL7F6IJ6XULAN2CDC45DM4VQCVTDLCJA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "Offset": 0, "Size": 11264, "Remote Address": "109.61.92.81:45096", "error": "trust: rpc: tcp connector failed: rpc: dial tcp 34.150.199.48:7777: operation was canceled", "errorVerbose": "trust: rpc: tcp connector failed: rpc: dial tcp 34.150.199.48:7777: operation was canceled\n\tstorj.io/common/rpc.HybridConnector.DialContext.func1:190"}
2024-10-11T09:15:50+03:00	ERROR	piecestore	download failed	{"Piece ID": "NKXDKS2RJJUA7HZJH25E7ZVDJ226ROXHD2WQGHJJMUY5GDWBN6YA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "Offset": 0, "Size": 137472, "Remote Address": "79.127.205.225:50994", "error": "write tcp 192.168.1.245:28967->79.127.205.225:50994: wsasend: An existing connection was forcibly closed by the remote host.", "errorVerbose": "write tcp 192.168.1.245:28967->79.127.205.225:50994: wsasend: An existing connection was forcibly closed by the remote host.\n\tstorj.io/drpc/drpcstream.(*Stream).rawFlushLocked:428\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:489\n\tstorj.io/common/pb.

[Alexey]

The usual long tail cancelation errors. Your node is too slow for these customers and they drop it.
You cannot win all races, because your node cannot be close to everyone customer in the world.

However, dropped GET_REPAIR suggests that your node have issues to read data from the disk in time. Likely you need an optimization there: do not use a network filesystems, do not use SMR drives, do not use NTFS under Linux, do not use BTRFS without an SSD as a cache layer, do not use RAID any kind. If you have to use RAID, then you need to add an SSD as a special device/cache layer or at least much more RAM.

[batelis]

Im using normal hdds diffrent sizes NTFS(on windows) file system, 1 node per drive and I did add 200gb of virtual ram.

[Alexey]

Ok then. You may do some optimizations though:

Windows Node Best Configs

Hi! Perhaps the most popular suggestion would be do not use Windows and NTFS, if that is not a server version or perhaps do not use it at all and use Linux/FreeBSD instead without using of NTFS However, you may optimize your NTFS: Enable automatic defragmentation for the disk, if you disabled it (it’s enabled by default) If you have managed UPS, then you may enable a write cache on the disk (both checkboxes). if you do not have a managed UPS, you should disable these checkboxes instead. D…

[batelis]

Ill try that, thank you

[Vadim]

SMR hdds? they are really slow.

[batelis]

Mostly seagate exos and some wd

[EasyRhino]

look at the last couple hundred lines of the file and see if some segments are successfully “uploaded” or “downloaded”. If some are succeeding then the node is still operating, although maybe the disk load is high. steps to reduce load (there are NTFS tips out there, or adding extra SSD caching options) might help with a higher success ratio, especially during backend operations like filewalkers and garbage collection.

if they are all timing out then there is a problem on the storage backend. For me this usually happens because my storage is mounted NFS shares (which you are not supposed to do) and it would mean the NAS died while the node was still nominally operating. Eventually a lot of stalled uploads (writes) will cause the node software to consume a ton of RAM and then crash.

[batelis]

And I think i found the problem, since there is no fatal errors in logs,(did a scan on all hdds and ssd for errors and fixed them) the only thing that could be crashing the node would be Vadims toolbox with the new update that updates all nodes. And it doesnt even work I left the updater running and it never updated my nodes but kept crashing them one by one.

[Vadim]

Then it try to update, but for some reason it cant stop normaly node, what version of node do you have?

[batelis]

When I installed your new version of toolbox I had my first node on v1.113.2 and others on v1.112 I started the updater and it did nothing. After that I manually updated my nodes now all on v1.113.2.

[Vadim]

my nodes are on 1.114 for long time, so it should update also your nodes, check the updater logs, it should be in program files, advanced_node_updater or something similar name

[batelis]

2024-10-16T10:25:33+03:00	INFO	Current binary version	{"Service": "storagenode-updater", "Version": "v1.113.2"}
2024-10-16T10:25:33+03:00	INFO	New version is being rolled out but hasn't made it to this node yet	{"Service": "storagenode-updater"}
2024-10-16T10:40:32+03:00	INFO	Downloading versions.	{"Server Address": "https://version.storj.io"}
2024-10-16T10:40:33+03:00	INFO	Current binary version	{"Service": "storagenode", "Version": "v1.113.2"}
2024-10-16T10:40:33+03:00	INFO	New version is being rolled out but hasn't made it to this node yet	{"Service": "storagenode"}
2024-10-16T10:40:33+03:00	INFO	Current binary version	{"Service": "storagenode-updater", "Version": "v1.113.2"}
2024-10-16T10:40:33+03:00	INFO	New version is being rolled out but hasn't made it to this node yet	{"Service": "storagenode-updater"}
2024-10-16T10:55:32+03:00	INFO	Downloading versions.	{"Server Address": "https://version.storj.io"}
2024-10-16T10:55:32+03:00	INFO	Current binary version	{"Service": "storagenode", "Version": "v1.113.2"}
2024-10-16T10:55:32+03:00	INFO	New version is being rolled out but hasn't made it to this node yet	{"Service": "storagenode"}
2024-10-16T10:55:33+03:00	INFO	Current binary version	{"Service": "storagenode-updater", "Version": "v1.113.2"}
2024-10-16T10:55:33+03:00	INFO	New version is being rolled out but hasn't made it to this node yet	{"Service": "storagenode-updater"}
2024-10-16T11:10:32+03:00	INFO	Downloading versions.	{"Server Address": "https://version.storj.io"}
2024-10-16T11:10:32+03:00	INFO	Current binary version	{"Service": "storagenode", "Version": "v1.113.2"}
2024-10-16T11:10:32+03:00	INFO	New version is being rolled out but hasn't made it to this node yet	{"Service": "storagenode"}
2024-10-16T11:10:33+03:00	INFO	Current binary version	{"Service": "storagenode-updater", "Version": "v1.113.2"}
2024-10-16T11:10:33+03:00	INFO	New version is being rolled out but hasn't made it to this node yet	{"Service": "storagenode-updater"}
2024-10-16T11:25:32+03:00	INFO	Downloading versions.	{"Server Address": "https://version.storj.io"}
2024-10-16T11:25:33+03:00	INFO	Current binary version	{"Service": "storagenode", "Version": "v1.113.2"}
2024-10-16T11:25:33+03:00	INFO	New version is being rolled out but hasn't made it to this node yet	{"Service": "storagenode"}
2024-10-16T11:25:33+03:00	INFO	Current binary version	{"Service": "storagenode-updater", "Version": "v1.113.2"}
2024-10-16T11:25:33+03:00	INFO	New version is being rolled out but hasn't made it to this node yet	{"Service": "storagenode-updater"}

[Vadim]

this is storj updater log, But i tell about toolbox updater

[batelis]

Ohhh sorry

10/9/2024 12:11:59 AM  1Discovered old verion, need update: storagenode
10/9/2024 12:11:59 AM  2Discovered old verion, need update: storagenode1
10/9/2024 12:11:59 AM  2Discovered old verion, need update: storagenode2
10/9/2024 12:11:59 AM  2Discovered old verion, need update: storagenode3
10/9/2024 12:11:59 AM  2Discovered old verion, need update: storagenode4
10/9/2024 12:12:00 AM  !Downloaded new file, extracting
10/9/2024 12:12:01 AM  Extracting complete.
10/9/2024 12:12:01 AM  Stoping: storagenode
10/9/2024 12:18:52 AM  )storagenode cant stop, too many atempts
10/9/2024 12:18:52 AM  Stoping: storagenode1
10/9/2024 12:25:42 AM  *storagenode1 cant stop, too many atempts
10/9/2024 12:25:42 AM  Stoping: storagenode2
10/9/2024 12:32:32 AM  *storagenode2 cant stop, too many atempts
10/9/2024 12:32:32 AM  Stoping: storagenode3
10/9/2024 12:39:23 AM  *storagenode3 cant stop, too many atempts
10/9/2024 12:39:23 AM  Stoping: storagenode4
10/9/2024 12:46:13 AM  *storagenode4 cant stop, too many atempts
10/9/2024 12:46:13 AM  Next update after: 42 h
10/9/2024 11:55:36 PM  &Last storagenode version is: 1.114.6
10/9/2024 11:55:36 PM  1Discovered old verion, need update: storagenode
10/9/2024 11:55:36 PM  2Discovered old verion, need update: storagenode1
10/9/2024 11:55:36 PM  2Discovered old verion, need update: storagenode2
10/9/2024 11:55:36 PM  2Discovered old verion, need update: storagenode3
10/9/2024 11:55:36 PM  2Discovered old verion, need update: storagenode4
10/9/2024 11:55:37 PM  !Downloaded new file, extracting
10/9/2024 11:55:38 PM  Extracting complete.
10/9/2024 11:55:38 PM  Stoping: storagenode
10/10/2024 12:02:28 AM  )storagenode cant stop, too many atempts
10/10/2024 12:02:28 AM  Stoping: storagenode1
10/10/2024 12:09:18 AM  *storagenode1 cant stop, too many atempts
10/10/2024 12:09:18 AM  Stoping: storagenode2
10/10/2024 12:16:09 AM  *storagenode2 cant stop, too many atempts
10/10/2024 12:16:09 AM  Stoping: storagenode3
10/10/2024 12:22:59 AM  *storagenode3 cant stop, too many atempts
10/10/2024 12:22:59 AM  Stoping: storagenode4
10/10/2024 12:29:49 AM  *storagenode4 cant stop, too many atempts
10/10/2024 12:29:49 AM  Next update after: 31 h
10/11/2024 7:29:50 AM  &Last storagenode version is: 1.114.6
10/11/2024 7:29:50 AM  1Discovered old verion, need update: storagenode
10/11/2024 7:29:50 AM  2Discovered old verion, need update: storagenode1
10/11/2024 7:29:50 AM  2Discovered old verion, need update: storagenode2
10/11/2024 7:29:50 AM  2Discovered old verion, need update: storagenode3
10/11/2024 7:29:50 AM  2Discovered old verion, need update: storagenode4
10/11/2024 7:29:50 AM  Stoping: storagenode
10/11/2024 7:36:40 AM  )storagenode cant stop, too many atempts
10/11/2024 7:36:40 AM  Stoping: storagenode1
10/11/2024 7:43:30 AM  *storagenode1 cant stop, too many atempts
10/11/2024 7:43:30 AM  Stoping: storagenode2
10/11/2024 7:50:21 AM  *storagenode2 cant stop, too many atempts
10/11/2024 7:50:21 AM  Stoping: storagenode3
10/11/2024 7:57:11 AM  *storagenode3 cant stop, too many atempts
10/11/2024 7:57:11 AM  Stoping: storagenode4
10/11/2024 8:04:01 AM  *storagenode4 cant stop, too many atempts
10/11/2024 8:04:01 AM  Next update after: 67 h
10/11/2024 5:10:34 PM  &Last storagenode version is: 1.114.6
10/11/2024 5:10:34 PM  1Discovered old verion, need update: storagenode
10/11/2024 5:10:34 PM  2Discovered old verion, need update: storagenode1
10/11/2024 5:10:34 PM  2Discovered old verion, need update: storagenode2
10/11/2024 5:10:34 PM  2Discovered old verion, need update: storagenode3
10/11/2024 5:10:34 PM  2Discovered old verion, need update: storagenode4
10/11/2024 5:10:34 PM  Stoping: storagenode
10/11/2024 5:17:24 PM  )storagenode cant stop, too many atempts
10/11/2024 5:17:24 PM  Stoping: storagenode1
10/11/2024 5:24:15 PM  *storagenode1 cant stop, too many atempts
10/11/2024 5:24:15 PM  Stoping: storagenode2
10/11/2024 5:31:05 PM  *storagenode2 cant stop, too many atempts
10/11/2024 5:31:05 PM  Stoping: storagenode3
10/11/2024 5:37:55 PM  *storagenode3 cant stop, too many atempts
10/11/2024 5:37:55 PM  Stoping: storagenode4
10/11/2024 5:44:46 PM  *storagenode4 cant stop, too many atempts
10/11/2024 5:44:46 PM  Next update after: 35 h
10/13/2024 4:44:46 AM  &Last storagenode version is: 1.114.6
10/13/2024 4:44:46 AM  1Discovered old verion, need update: storagenode
10/13/2024 4:44:46 AM  2Discovered old verion, need update: storagenode1
10/13/2024 4:44:46 AM  2Discovered old verion, need update: storagenode2
10/13/2024 4:44:46 AM  2Discovered old verion, need update: storagenode3
10/13/2024 4:44:46 AM  2Discovered old verion, need update: storagenode4
10/13/2024 4:44:46 AM  Stoping: storagenode
10/13/2024 4:51:37 AM  )storagenode cant stop, too many atempts
10/13/2024 4:51:37 AM  Stoping: storagenode1
10/13/2024 4:58:27 AM  *storagenode1 cant stop, too many atempts
10/13/2024 4:58:27 AM  Stoping: storagenode2
10/13/2024 5:05:17 AM  *storagenode2 cant stop, too many atempts
10/13/2024 5:05:17 AM  Stoping: storagenode3
10/13/2024 5:12:07 AM  *storagenode3 cant stop, too many atempts
10/13/2024 5:12:07 AM  Stoping: storagenode4
10/13/2024 5:18:58 AM  *storagenode4 cant stop, too many atempts
10/13/2024 5:18:58 AM  Next update after: 32 h

So it looks like it does work but something doesnt

[Vadim]

your node for some reasons cant stop normally, in 400s time. this is main problem. if you see that your node is stopped, then restart updater, it will update even stopped nodes.

[batelis]

when I press stop node in toolbox it takes like 2sec and it stops

[Vadim]

what version do you have, i remember there was some problem version, that not detected that node stopped
try download last one.
Releases · TonyTosol/Storj-Node-Toolbox (github.com)
you also need to update updater.
stop updater.
run cmd as Admin, ther write sc.exe delete advanced_storagenode_updater
then install it in toolbox again.

[Alexey]

Has anyone tried setting up a separate storagenode-updater for each node?

[batelis]

Just installed the newest version Ill report if something wrong happens