Online score at 100% but Suspension is decreasing

ptC7H12 · November 7, 2023, 8:15pm

Hello one of my nodes has following issues:

Online score is at 100% but suspension is decresing:

The Audit of us1 is due to an shutdown a few weeks ago, but slowly increasing.

But what is about that suspension?

I’ve git 6 Nodes on same host (all 6 with same settings. All other 5 are working well and without issues, only one this one the suspensions suddelny started to drop…

That are the logs:

2023-11-07T18:16:18Z    ERROR   piecestore      upload failed   {"process": "storagenode", "Piece ID": "D653UNS56KGB3DMFOHXO2SUX3BFLBIUR27V2OANMID3N7VF6QOIA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "error": "manager closed: unexpected EOF", "errorVerbose": "manager closed: unexpected EOF\n\tgithub.com/jtolio/noiseconn.(*Conn).readMsg:225\n\tgithub.com/jtolio/noiseconn.(*Conn).Read:171\n\tstorj.io/drpc/drpcwire.(*Reader).ReadPacketUsing:96\n\tstorj.io/drpc/drpcmanager.(*Manager).manageReader:226", "Size": 131072, "Remote Address": "23.237.191.42:34542"}
2023-11-07T18:17:30Z    ERROR   contact:service ping satellite failed   {"process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "attempts": 1, "error": "ping satellite: rpc: tcp connector failed: rpc: dial tcp: lookup saltlake.tardigrade.io on 192.168.1.1:53: read udp 172.17.0.3:45929->192.168.1.1:53: i/o timeout", "errorVerbose": "ping satellite: rpc: tcp connector failed: rpc: dial tcp: lookup saltlake.tardigrade.io on 192.168.1.1:53: read udp 172.17.0.3:45929->192.168.1.1:53: i/o timeout\n\tstorj.io/common/rpc.HybridConnector.DialContext.func1:190"}
2023-11-07T18:17:30Z    ERROR   contact:service ping satellite failed   {"process": "storagenode", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "attempts": 1, "error": "ping satellite: rpc: tcp connector failed: rpc: dial tcp: lookup ap1.storj.io on 192.168.1.1:53: read udp 172.17.0.3:48793->192.168.1.1:53: i/o timeout", "errorVerbose": "ping satellite: rpc: tcp connector failed: rpc: dial tcp: lookup ap1.storj.io on 192.168.1.1:53: read udp 172.17.0.3:48793->192.168.1.1:53: i/o timeout\n\tstorj.io/common/rpc.HybridConnector.DialContext.func1:190"}
2023-11-07T18:17:30Z    ERROR   contact:service ping satellite failed   {"process": "storagenode", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "attempts": 1, "error": "ping satellite: rpc: tcp connector failed: rpc: dial tcp: lookup eu1.storj.io on 192.168.1.1:53: read udp 172.17.0.3:55239->192.168.1.1:53: i/o timeout", "errorVerbose": "ping satellite: rpc: tcp connector failed: rpc: dial tcp: lookup eu1.storj.io on 192.168.1.1:53: read udp 172.17.0.3:55239->192.168.1.1:53: i/o timeout\n\tstorj.io/common/rpc.HybridConnector.DialContext.func1:190"}
2023-11-07T18:17:30Z    ERROR   contact:service ping satellite failed   {"process": "storagenode", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "attempts": 1, "error": "ping satellite: rpc: tcp connector failed: rpc: dial tcp: lookup us1.storj.io on 192.168.1.1:53: read udp 172.17.0.3:39355->192.168.1.1:53: i/o timeout", "errorVerbose": "ping satellite: rpc: tcp connector failed: rpc: dial tcp: lookup us1.storj.io on 192.168.1.1:53: read udp 172.17.0.3:39355->192.168.1.1:53: i/o timeout\n\tstorj.io/common/rpc.HybridConnector.DialContext.func1:190"}
2023-11-07T18:18:06Z    ERROR   piecestore      upload failed   {"process": "storagenode", "Piece ID": "P47JRC2WT2OK6S47BMGVX3TTHQIS5H2C5O4NGD3TCGCUDLCQ5NEQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "error": "manager closed: unexpected EOF", "errorVerbose": "manager closed: unexpected EOF\n\tgithub.com/jtolio/noiseconn.(*Conn).readMsg:225\n\tgithub.com/jtolio/noiseconn.(*Conn).Read:171\n\tstorj.io/drpc/drpcwire.(*Reader).ReadPacketUsing:96\n\tstorj.io/drpc/drpcmanager.(*Manager).manageReader:226", "Size": 131072, "Remote Address": "23.237.191.122:32964"}
2023-11-07T18:18:35Z    ERROR   piecestore      upload failed   {"process": "storagenode", "Piece ID": "O7SFBE3MCU6A5C62UV2C53CXEDP7LRI6T5JL4EJ6FIGXTWADCXKQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "error": "manager closed: unexpected EOF", "errorVerbose": "manager closed: unexpected EOF\n\tgithub.com/jtolio/noiseconn.(*Conn).readMsg:225\n\tgithub.com/jtolio/noiseconn.(*Conn).Read:171\n\tstorj.io/drpc/drpcwire.(*Reader).ReadPacketUsing:96\n\tstorj.io/drpc/drpcmanager.(*Manager).manageReader:226", "Size": 131072, "Remote Address": "5.161.117.79:49798"}
2023-11-07T19:06:46Z    ERROR   piecestore      upload failed   {"process": "storagenode", "Piece ID": "S66DGAFYMBBSAWGN546RAIQ6LSQ3CNM7X4ZC6EMFWJXY2V7ZTVCA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "error": "manager closed: unexpected EOF", "errorVerbose": "manager closed: unexpected EOF\n\tgithub.com/jtolio/noiseconn.(*Conn).readMsg:225\n\tgithub.com/jtolio/noiseconn.(*Conn).Read:171\n\tstorj.io/drpc/drpcwire.(*Reader).ReadPacketUsing:96\n\tstorj.io/drpc/drpcmanager.(*Manager).manageReader:226", "Size": 131072, "Remote Address": "50.7.22.194:41810"}
2023-11-07T19:06:53Z    ERROR   piecestore      download failed {"process": "storagenode", "Piece ID": "33BTM2ALILDKBQVCSPDQWNRDD5J5RKYRIPQGCFCERW5M3H26M7HQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "Remote Address": "5.161.143.41:36720", "error": "order created too long ago: OrderCreation 2023-11-07 18:02:21.271343061 +0000 UTC < SystemClock 2023-11-07 19:06:53.355718495 +0000 UTC m=+514175.541673280", "errorVerbose": "order created too long ago: OrderCreation 2023-11-07 18:02:21.271343061 +0000 UTC < SystemClock 2023-11-07 19:06:53.355718495 +0000 UTC m=+514175.541673280\n\tstorj.io/common/rpc/rpcstatus.Errorf:89\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).verifyOrderLimit:45\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:616\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:251\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:61\n\tstorj.io/common/experiment.(*Handler).HandleRPC:42\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:124\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:114\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35"}
2023-11-07T19:07:40Z    ERROR   piecestore      upload failed   {"process": "storagenode", "Piece ID": "LAKI24DQ2ZEID3I6WBNFJDWXBFK25H67VGAXTIAN2MODXSZCAFLQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "error": "manager closed: unexpected EOF", "errorVerbose": "manager closed: unexpected EOF\n\tgithub.com/jtolio/noiseconn.(*Conn).readMsg:225\n\tgithub.com/jtolio/noiseconn.(*Conn).Read:171\n\tstorj.io/drpc/drpcwire.(*Reader).ReadPacketUsing:96\n\tstorj.io/drpc/drpcmanager.(*Manager).manageReader:226", "Size": 393216, "Remote Address": "79.127.203.194:59676"}
2023-11-07T19:07:47Z    ERROR   piecestore      upload failed   {"process": "storagenode", "Piece ID": "3QILYWQTW4SHJ6MPR3GPB2X4BJ3U5QLE6U3GRAJCMOIWPR56GI3A", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "error": "manager closed: unexpected EOF", "errorVerbose": "manager closed: unexpected EOF\n\tgithub.com/jtolio/noiseconn.(*Conn).readMsg:225\n\tgithub.com/jtolio/noiseconn.(*Conn).Read:171\n\tstorj.io/drpc/drpcwire.(*Reader).ReadPacketUsing:96\n\tstorj.io/drpc/drpcmanager.(*Manager).manageReader:226", "Size": 131072, "Remote Address": "79.127.220.98:38914"}
2023-11-07T19:07:53Z    ERROR   piecestore      upload failed   {"process": "storagenode", "Piece ID": "BMO2DXNMJZB3NMHL4IP4FZEV7JXBYQ5OLREQIRTY5YNZK5RDT72Q", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "error": "manager closed: unexpected EOF", "errorVerbose": "manager closed: unexpected EOF\n\tgithub.com/jtolio/noiseconn.(*Conn).readMsg:225\n\tgithub.com/jtolio/noiseconn.(*Conn).Read:171\n\tstorj.io/drpc/drpcwire.(*Reader).ReadPacketUsing:96\n\tstorj.io/drpc/drpcmanager.(*Manager).manageReader:226", "Size": 262144, "Remote Address": "5.161.117.79:58730"}
2023-11-07T19:08:27Z    ERROR   piecestore      upload failed   {"process": "storagenode", "Piece ID": "UT4X46SC6DS6KRHPIGE5F7HZINEV2LMERGB2LM5CBVZN5ZAOU5CQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "error": "manager closed: unexpected EOF", "errorVerbose": "manager closed: unexpected EOF\n\tgithub.com/jtolio/noiseconn.(*Conn).readMsg:225\n\tgithub.com/jtolio/noiseconn.(*Conn).Read:171\n\tstorj.io/drpc/drpcwire.(*Reader).ReadPacketUsing:96\n\tstorj.io/drpc/drpcmanager.(*Manager).manageReader:226", "Size": 393216, "Remote Address": "23.237.232.130:47082"}
2023-11-07T19:08:48Z    ERROR   piecestore      upload failed   {"process": "storagenode", "Piece ID": "JJ6H2NXDE4XKROOP5BJZL76HHWQ4KHQOV6KEKBO6UOUHS54UYYVA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "error": "manager closed: unexpected EOF", "errorVerbose": "manager closed: unexpected EOF\n\tgithub.com/jtolio/noiseconn.(*Conn).readMsg:225\n\tgithub.com/jtolio/noiseconn.(*Conn).Read:171\n\tstorj.io/drpc/drpcwire.(*Reader).ReadPacketUsing:96\n\tstorj.io/drpc/drpcmanager.(*Manager).manageReader:226", "Size": 131072, "Remote Address": "5.161.211.187:11138"}
2023-11-07T19:09:02Z    ERROR   piecestore      upload failed   {"process": "storagenode", "Piece ID": "CHZB5HDQ72FLS6CFX3PJVMLIW2EZ545BLBEDPFUMHSJ6OFVSDXEA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "error": "manager closed: unexpected EOF", "errorVerbose": "manager closed: unexpected EOF\n\tgithub.com/jtolio/noiseconn.(*Conn).readMsg:225\n\tgithub.com/jtolio/noiseconn.(*Conn).Read:171\n\tstorj.io/drpc/drpcwire.(*Reader).ReadPacketUsing:96\n\tstorj.io/drpc/drpcmanager.(*Manager).manageReader:226", "Size": 393216, "Remote Address": "23.237.232.130:45936"}
2023-11-07T19:09:06Z    ERROR   piecestore      upload failed   {"process": "storagenode", "Piece ID": "AIMRSHC6XI3RWZ45QLCAQXMFOGAXMLB4G2UE5ZNUDDJDVXCCKQKQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "error": "manager closed: unexpected EOF", "errorVerbose": "manager closed: unexpected EOF\n\tgithub.com/jtolio/noiseconn.(*Conn).readMsg:225\n\tgithub.com/jtolio/noiseconn.(*Conn).Read:171\n\tstorj.io/drpc/drpcwire.(*Reader).ReadPacketUsing:96\n\tstorj.io/drpc/drpcmanager.(*Manager).manageReader:226", "Size": 720896, "Remote Address": "89.187.181.16:25588"}
2023-11-07T19:09:09Z    ERROR   piecestore      download failed {"process": "storagenode", "Piece ID": "R77O4P35QCARTA4NLZMRYF737QATCA4USYSKERN2Y6XXVJN7AJHA", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET", "Remote Address": "50.7.230.34:42156", "error": "order created too long ago: OrderCreation 2023-11-07 18:07:04.935793629 +0000 UTC < SystemClock 2023-11-07 19:09:09.95666747 +0000 UTC m=+514312.142622261", "errorVerbose": "order created too long ago: OrderCreation 2023-11-07 18:07:04.935793629 +0000 UTC < SystemClock 2023-11-07 19:09:09.95666747 +0000 UTC m=+514312.142622261\n\tstorj.io/common/rpc/rpcstatus.Errorf:89\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).verifyOrderLimit:45\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:616\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:251\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:61\n\tstorj.io/common/experiment.(*Handler).HandleRPC:42\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:124\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:114\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35"}
2023-11-07T19:10:23Z    ERROR   piecestore      upload failed   {"process": "storagenode", "Piece ID": "QJRR7SAZJHIAJC2BCYWODUQVMICFRYYBRWH77T2CRY342QQAPLHQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "error": "manager closed: unexpected EOF", "errorVerbose": "manager closed: unexpected EOF\n\tgithub.com/jtolio/noiseconn.(*Conn).readMsg:225\n\tgithub.com/jtolio/noiseconn.(*Conn).Read:171\n\tstorj.io/drpc/drpcwire.(*Reader).ReadPacketUsing:96\n\tstorj.io/drpc/drpcmanager.(*Manager).manageReader:226", "Size": 917504, "Remote Address": "23.237.232.130:55352"}
2023-11-07T19:10:53Z    ERROR   piecestore      upload failed   {"process": "storagenode", "Piece ID": "CZFAJQXAPMPKAUAG2LCSHCM6SGWCTXINYCZFSMYIM2EECXMZYPLQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "error": "manager closed: unexpected EOF", "errorVerbose": "manager closed: unexpected EOF\n\tgithub.com/jtolio/noiseconn.(*Conn).readMsg:225\n\tgithub.com/jtolio/noiseconn.(*Conn).Read:171\n\tstorj.io/drpc/drpcwire.(*Reader).ReadPacketUsing:96\n\tstorj.io/drpc/drpcmanager.(*Manager).manageReader:226", "Size": 131072, "Remote Address": "79.127.220.98:48196"}
2023-11-07T19:11:46Z    ERROR   piecestore      upload failed   {"process": "storagenode", "Piece ID": "22V2VO2WAD2DWZPHANQ64FXM2R7QYN6FNZG3Q7Z3LXBQE5MXTVDQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "error": "manager closed: unexpected EOF", "errorVerbose": "manager closed: unexpected EOF\n\tgithub.com/jtolio/noiseconn.(*Conn).readMsg:225\n\tgithub.com/jtolio/noiseconn.(*Conn).Read:171\n\tstorj.io/drpc/drpcwire.(*Reader).ReadPacketUsing:96\n\tstorj.io/drpc/drpcmanager.(*Manager).manageReader:226", "Size": 131072, "Remote Address": "5.161.211.187:35284"}
2023-11-07T19:12:10Z    ERROR   piecestore      download failed {"process": "storagenode", "Piece ID": "DYAJGQGYWXJKROE7A7JRKPSOPSY6Z7U7KSL7V3CDRKQ25F3DXBYA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "Remote Address": "23.237.191.146:34062", "error": "order created too long ago: OrderCreation 2023-11-07 18:10:29.907284054 +0000 UTC < SystemClock 2023-11-07 19:12:10.898919632 +0000 UTC m=+514493.084874422", "errorVerbose": "order created too long ago: OrderCreation 2023-11-07 18:10:29.907284054 +0000 UTC < SystemClock 2023-11-07 19:12:10.898919632 +0000 UTC m=+514493.084874422\n\tstorj.io/common/rpc/rpcstatus.Errorf:89\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).verifyOrderLimit:45\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:616\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:251\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:61\n\tstorj.io/common/experiment.(*Handler).HandleRPC:42\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:124\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:114\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35"}
2023-11-07T19:12:42Z    ERROR   piecestore      upload failed   {"process": "storagenode", "Piece ID": "HMGCX7Q3T6IWOK4WE6OAECZ5M75TVMJ4VCOJCZETPI64XJ62CDEA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "error": "manager closed: unexpected EOF", "errorVerbose": "manager closed: unexpected EOF\n\tgithub.com/jtolio/noiseconn.(*Conn).readMsg:225\n\tgithub.com/jtolio/noiseconn.(*Conn).Read:171\n\tstorj.io/drpc/drpcwire.(*Reader).ReadPacketUsing:96\n\tstorj.io/drpc/drpcmanager.(*Manager).manageReader:226", "Size": 131072, "Remote Address": "5.161.117.79:50830"}
2023-11-07T19:13:28Z    ERROR   piecestore      upload failed   {"process": "storagenode", "Piece ID": "WJKVYE6OMQUFYG22BFPG4EIBL6LPQCSDRHUHZKOEEIAAVEMVZVRQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "error": "manager closed: unexpected EOF", "errorVerbose": "manager closed: unexpected EOF\n\tgithub.com/jtolio/noiseconn.(*Conn).readMsg:225\n\tgithub.com/jtolio/noiseconn.(*Conn).Read:171\n\tstorj.io/drpc/drpcwire.(*Reader).ReadPacketUsing:96\n\tstorj.io/drpc/drpcmanager.(*Manager).manageReader:226", "Size": 131072, "Remote Address": "79.127.203.194:45598"}
2023-11-07T19:13:51Z    ERROR   piecestore      upload failed   {"process": "storagenode", "Piece ID": "S3K6CX66H5PXUNLFSVNSPE3PFNWNXGN46UZ22ROYOSAO2I4TXBFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "error": "manager closed: unexpected EOF", "errorVerbose": "manager closed: unexpected EOF\n\tgithub.com/jtolio/noiseconn.(*Conn).readMsg:225\n\tgithub.com/jtolio/noiseconn.(*Conn).Read:171\n\tstorj.io/drpc/drpcwire.(*Reader).ReadPacketUsing:96\n\tstorj.io/drpc/drpcmanager.(*Manager).manageReader:226", "Size": 131072, "Remote Address": "5.161.211.187:14392"}
2023-11-07T19:14:11Z    ERROR   piecestore      upload failed   {"process": "storagenode", "Piece ID": "MO25FSLKADKWDKX5UF4BXZHIQ7352V6XTCNZCLKIZCUDZZAIVE6Q", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "error": "manager closed: unexpected EOF", "errorVerbose": "manager closed: unexpected EOF\n\tgithub.com/jtolio/noiseconn.(*Conn).readMsg:225\n\tgithub.com/jtolio/noiseconn.(*Conn).Read:171\n\tstorj.io/drpc/drpcwire.(*Reader).ReadPacketUsing:96\n\tstorj.io/drpc/drpcmanager.(*Manager).manageReader:226", "Size": 786432, "Remote Address": "89.187.181.16:58360"}
2023-11-07T19:14:42Z    ERROR   piecestore      upload failed   {"process": "storagenode", "Piece ID": "TAUOJ3A4FHHHYPCMV7IJSMA2T3S6DNUSMHR5MRKXLGLN6SDKJ2ZQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "error": "manager closed: unexpected EOF", "errorVerbose": "manager closed: unexpected EOF\n\tgithub.com/jtolio/noiseconn.(*Conn).readMsg:225\n\tgithub.com/jtolio/noiseconn.(*Conn).Read:171\n\tstorj.io/drpc/drpcwire.(*Reader).ReadPacketUsing:96\n\tstorj.io/drpc/drpcmanager.(*Manager).manageReader:226", "Size": 262144, "Remote Address": "23.237.191.42:35556"}
2023-11-07T19:15:52Z    ERROR   piecestore      upload failed   {"process": "storagenode", "Piece ID": "22FMN6K5OGUZIZIUSYB3M7EVBC4TBCMEUO7WYKOGSQBNDEGN5DRA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "error": "manager closed: unexpected EOF", "errorVerbose": "manager closed: unexpected EOF\n\tgithub.com/jtolio/noiseconn.(*Conn).readMsg:225\n\tgithub.com/jtolio/noiseconn.(*Conn).Read:171\n\tstorj.io/drpc/drpcwire.(*Reader).ReadPacketUsing:96\n\tstorj.io/drpc/drpcmanager.(*Manager).manageReader:226", "Size": 262144, "Remote Address": "5.161.192.57:32610"}
2023-11-07T19:15:53Z    ERROR   piecestore      upload failed   {"process": "storagenode", "Piece ID": "IULE6Y2BLPNG2YFWEBKUTAOV46EWM5QTQ36STM67HTIR4JZXTNLQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "error": "manager closed: unexpected EOF", "errorVerbose": "manager closed: unexpected EOF\n\tgithub.com/jtolio/noiseconn.(*Conn).readMsg:225\n\tgithub.com/jtolio/noiseconn.(*Conn).Read:171\n\tstorj.io/drpc/drpcwire.(*Reader).ReadPacketUsing:96\n\tstorj.io/drpc/drpcmanager.(*Manager).manageReader:226", "Size": 131072, "Remote Address": "5.161.117.79:40270"}
2023-11-07T19:15:53Z    ERROR   piecestore      upload failed   {"process": "storagenode", "Piece ID": "AQUUJSU64HSG7P5WORBIUUJ2VJAEMLVKNH5X5UG35UT2JZXHKBUA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "error": "manager closed: unexpected EOF", "errorVerbose": "manager closed: unexpected EOF\n\tgithub.com/jtolio/noiseconn.(*Conn).readMsg:225\n\tgithub.com/jtolio/noiseconn.(*Conn).Read:171\n\tstorj.io/drpc/drpcwire.(*Reader).ReadPacketUsing:96\n\tstorj.io/drpc/drpcmanager.(*Manager).manageReader:226", "Size": 131072, "Remote Address": "79.127.203.193:50782"}
2023-11-07T19:17:14Z    ERROR   piecestore      upload failed   {"process": "storagenode", "Piece ID": "NHHWEX7G3IT4FOXNWAXBWWKWYO23E2TF2VJA6NWLA6GPJJJHH3ZA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "error": "manager closed: unexpected EOF", "errorVerbose": "manager closed: unexpected EOF\n\tgithub.com/jtolio/noiseconn.(*Conn).readMsg:225\n\tgithub.com/jtolio/noiseconn.(*Conn).Read:171\n\tstorj.io/drpc/drpcwire.(*Reader).ReadPacketUsing:96\n\tstorj.io/drpc/drpcmanager.(*Manager).manageReader:226", "Size": 262144, "Remote Address": "5.161.220.231:58474"}
2023-11-07T19:17:15Z    ERROR   piecestore      upload failed   {"process": "storagenode", "Piece ID": "OAPLGISYPL27W32JR2TZZ5PZFEPYQWCTRWMJBEF4XWFJ3IZ7I4TQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT", "error": "manager closed: unexpected EOF", "errorVerbose": "manager closed: unexpected EOF\n\tstorj.io/drpc/drpcmanager.(*Manager).manageReader:231", "Size": 311296, "Remote Address": "144.24.180.15:52892"}
2023-11-07T19:17:27Z    ERROR   Error retrieving version info.  {"Process": "storagenode-updater", "error": "version checker client: Get \"https://version.storj.io\": dial tcp: lookup version.storj.io on 192.168.1.1:53: read udp 172.17.0.3:46483->192.168.1.1:53: i/o timeout", "errorVerbose": "version checker client: Get \"https://version.storj.io\": dial tcp: lookup version.storj.io on 192.168.1.1:53: read udp 172.17.0.3:46483->192.168.1.1:53: i/o timeout\n\tstorj.io/storj/private/version/checker.(*Client).All:68\n\tmain.loopFunc:21\n\tstorj.io/common/sync2.(*Cycle).Run:160\n\tmain.cmdRun:136\n\tstorj.io/private/process.cleanup.func1.4:402\n\tstorj.io/private/process.cleanup.func1:420\n\tgithub.com/spf13/cobra.(*Command).execute:852\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:960\n\tgithub.com/spf13/cobra.(*Command).Execute:897\n\tstorj.io/private/process.ExecWithCustomOptions:113\n\tstorj.io/private/process.ExecWithCustomConfigAndLogger:78\n\tmain.main:20\n\truntime.main:267"}
2023-11-07T19:18:24Z    ERROR   piecestore      upload failed   {"process": "storagenode", "Piece ID": "LNIXGYLMPXZWFOFDR3ZFVMFT6HAGAL2RUUQHMXREFGNKHTCY32CQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "error": "manager closed: unexpected EOF", "errorVerbose": "manager closed: unexpected EOF\n\tgithub.com/jtolio/noiseconn.(*Conn).readMsg:225\n\tgithub.com/jtolio/noiseconn.(*Conn).Read:171\n\tstorj.io/drpc/drpcwire.(*Reader).ReadPacketUsing:96\n\tstorj.io/drpc/drpcmanager.(*Manager).manageReader:226", "Size": 393216, "Remote Address": "79.127.220.97:54906"}

Knowledge · November 7, 2023, 8:29pm

Are you using a DDNS provider? It could be a slow/bad DNS on their end, we see this quite a bit. You would do better to switch DDNS providers to fix it if so.

ptC7H12 · November 7, 2023, 8:34pm

No, this Node has a static ip, thats why it is strange - as mentioned all other nodes are working fine…

Knowledge · November 7, 2023, 8:35pm

You could try changing your public DNS provider and see if that improves things.

ptC7H12 · November 7, 2023, 8:37pm

Thanks, I will try and report back!

Alexey · November 8, 2023, 8:49am

Both Audit and Suspension scores falling only when your node failing audits, it’s not related to downtime. To answer on audit request your node must be online, otherwise the online score will be affected.
I would suggest to check your logs to figure out, why your node is failing audits:

daki82 · November 8, 2023, 9:02am

Mind to tell us the specs of hard and software? Helping would be much more easy.

ptC7H12 · November 9, 2023, 9:38am

As I checked and did not find any issues I just restarted my host and Suspension score is rising since then.

Thank you all for your help!

ptC7H12 · December 4, 2023, 5:36pm

No I’ve got other errors on my Node:

2023-11-28T04:07:43Z    ERROR   piecestore      download failed {"process": "storagenode", "Piece ID": "LQB4BAIGBAXGIMQS4CUZB5YZHP6MCQELANTVLLTZXY44LBPINXNA", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Action": "GET_REPAIR", "Offset": 0, "Size": 0, "Remote Address": "167.235.26.51:34318", "error": "write tcp 172.17.0.3:28967->167.235.26.51:34318: write: broken pipe", "errorVerbose": "write tcp 172.17.0.3:28967->167.235.26.51:34318: write: broken pipe\n\tstorj.io/drpc/drpcstream.(*Stream).rawFlushLocked:401\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:462\n\tstorj.io/common/pb.(*drpcPiecestore_DownloadStream).Send:349\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).sendData.func1:861\n\tstorj.io/common/rpc/rpctimeout.Run.func1:22"}
2023-11-29T01:39:27Z    ERROR   piecestore      download failed {"process": "storagenode", "Piece ID": "SPDB7WACMTU4RBWVOKD3O5IOEQWW4AJLGPWAMQX7FU7DDFYP6RLA", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Action": "GET_REPAIR", "Offset": 0, "Size": 0, "Remote Address": "49.12.194.191:60070", "error": "write tcp 172.17.0.3:28967->49.12.194.191:60070: write: broken pipe", "errorVerbose": "write tcp 172.17.0.3:28967->49.12.194.191:60070: write: broken pipe\n\tstorj.io/drpc/drpcstream.(*Stream).rawFlushLocked:401\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:462\n\tstorj.io/common/pb.(*drpcPiecestore_DownloadStream).Send:349\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).sendData.func1:861\n\tstorj.io/common/rpc/rpctimeout.Run.func1:22"}
2023-11-29T10:32:19Z    ERROR   piecestore      download failed {"process": "storagenode", "Piece ID": "2ST3ZSZ722BLNKAEY3VYEK4M3DYEXME7JLLEEISCUWTQ2JVE6KFA", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Action": "GET_REPAIR", "Offset": 0, "Size": 0, "Remote Address": "94.130.111.189:56512", "error": "write tcp 172.17.0.3:28967->94.130.111.189:56512: write: broken pipe", "errorVerbose": "write tcp 172.17.0.3:28967->94.130.111.189:56512: write: broken pipe\n\tstorj.io/drpc/drpcstream.(*Stream).rawFlushLocked:401\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:462\n\tstorj.io/common/pb.(*drpcPiecestore_DownloadStream).Send:349\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).sendData.func1:861\n\tstorj.io/common/rpc/rpctimeout.Run.func1:22"}
2023-11-29T12:09:55Z    ERROR   piecestore      download failed {"process": "storagenode", "Piece ID": "7C3FGLRVWHIEBZUNGPKLKWHMBFLO3Y4ONSNRWRLCMWXJQEBQXDGA", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Action": "GET_REPAIR", "Offset": 0, "Size": 0, "Remote Address": "116.203.40.151:52876", "error": "write tcp 172.17.0.3:28967->116.203.40.151:52876: write: broken pipe", "errorVerbose": "write tcp 172.17.0.3:28967->116.203.40.151:52876: write: broken pipe\n\tstorj.io/drpc/drpcstream.(*Stream).rawFlushLocked:401\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:462\n\tstorj.io/common/pb.(*drpcPiecestore_DownloadStream).Send:349\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).sendData.func1:861\n\tstorj.io/common/rpc/rpctimeout.Run.func1:22"}
2023-12-01T16:32:28Z    ERROR   piecestore      download failed {"process": "storagenode", "Piece ID": "XZ546ZYRQFJPBDFCETZCPKTGO7HJCVOCPKML2OMGJVFXXZREPRGQ", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Action": "GET_REPAIR", "Offset": 0, "Size": 0, "Remote Address": "116.203.117.228:44606", "error": "write tcp 172.17.0.3:28967->116.203.117.228:44606: write: broken pipe", "errorVerbose": "write tcp 172.17.0.3:28967->116.203.117.228:44606: write: broken pipe\n\tstorj.io/drpc/drpcstream.(*Stream).rawFlushLocked:401\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:462\n\tstorj.io/common/pb.(*drpcPiecestore_DownloadStream).Send:349\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).sendData.func1:861\n\tstorj.io/common/rpc/rpctimeout.Run.func1:22"}
2023-12-03T12:59:48Z    ERROR   piecestore      download failed {"process": "storagenode", "Piece ID": "MGY276YEIIZJCYWRZ36UW3CNF57PW7TLNO2G7AP7E32SZ6USWQXQ", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Action": "GET_REPAIR", "Offset": 0, "Size": 0, "Remote Address": "49.12.194.191:60958", "error": "write tcp 172.17.0.3:28967->49.12.194.191:60958: write: broken pipe", "errorVerbose": "write tcp 172.17.0.3:28967->49.12.194.191:60958: write: broken pipe\n\tstorj.io/drpc/drpcstream.(*Stream).rawFlushLocked:401\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:462\n\tstorj.io/common/pb.(*drpcPiecestore_DownloadStream).Send:349\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).sendData.func1:861\n\tstorj.io/common/rpc/rpctimeout.Run.func1:22"}

When I restart Node, the suspension score increases for some time, then it starts to decrease again…
It seems as it is only on us1
Any hints? What can cause this errors?

daki82 · December 4, 2023, 7:43pm

is us1 your biggest ingress provider?
google says its something on file or networking level. how are the 6 nodes connected?, mainboard driver?

ptC7H12 · December 4, 2023, 7:51pm

How do you get, that 6 nodes are connected? Because of the 6 error lines? Oh, there are much more, but all nearly the same. There is only one Node.

And us1 and eu1 are very close to each other. DBs are on SSD

daki82 · December 4, 2023, 8:57pm

Because you said so…

Alexey · December 5, 2023, 5:03am

These errors are usual context canceled errors, your node loose the race for pieces.
You need to search for errors related to GET_AUDIT and GET_REPAIR, all others cannot affect a suspension score.

ptC7H12 · December 5, 2023, 6:26am

Sorry, I was so focused on on my error, that ignored what I I had written before

That was the log command I’d used. There are no errors about AUDIT or repair:
docker logs storagenode 2>&1 | grep -E "GET_AUDIT|GET_REPAIR" | grep failed

And the errors ebove are GET_REPAIR errors