Loosing score on audit

mer44 · July 27, 2022, 7:07am

I ran a search:
docker logs storagenode --since “2022-07-25T22:00” | grep -E “GET_AUDIT|GET_REPAIR”

and I get only the following errors:

use of closed network connection
write tcp 172.17.0.3:28967->5.161.44.25:52488: write: broken pipe

Why am I loosing audit scores?
5% on two satelites and 10% on one to be precise.

Bivvo · July 27, 2022, 7:21am

can you please share the full log result of your selection?
It seems that you have some kind of network unavailability during an audit session.

mer44 · July 27, 2022, 7:30am

2022-07-26T05:45:15.742Z        ERROR   piecestore      download failed {"Process": "storagenode", "Piece ID": "RFQ2HG77GRCSENF74S73MNR6CCBCLOBGV6E7YVTEYLIEW7WL3LEQ", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Action": "GET_REPAIR", "error": "write tcp 172.17.0.3:28967->167.235.19.47:40518: write: broken pipe", "errorVerbose": "write tcp 172.17.0.3:28967->167.235.19.47:40518: write: broken pipe\n\tstorj.io/drpc/drpcstream.(*Stream).rawFlushLocked:352\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:404\n\tstorj.io/common/pb.(*drpcPiecestore_DownloadStream).Send:317\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func4:571\n\tstorj.io/common/rpc/rpctimeout.Run.func1:22"} 2022-07-26T07:36:41.878Z        ERROR   piecestore      download failed {"Process": "storagenode", "Piece ID": "EEVXCKW5UUZ6LBC77BXCNL5364RSDG76WB3X33LLERWWHZAJQIFQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET_REPAIR", "error": "write tcp 172.17.0.3:28967->5.161.74.111:55816: write: broken pipe", "errorVerbose": "write tcp 172.17.0.3:28967->5.161.74.111:55816: write: broken pipe\n\tstorj.io/drpc/drpcstream.(*Stream).rawFlushLocked:352\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:404\n\tstorj.io/common/pb.(*drpcPiecestore_DownloadStream).Send:317\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func4:571\n\tstorj.io/common/rpc/rpctimeout.Run.func1:22"} 2022-07-26T07:55:27.001Z        ERROR   piecestore      download failed {"Process": "storagenode", "Piece ID": "HQFLE67QTPY4ETINK2DCZON2SFHJHCGIE6VCWT33B7CAOZWJHFYA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET_REPAIR", "error": "write tcp 172.17.0.3:28967->5.161.74.111:45530: write: broken pipe", "errorVerbose": "write tcp 172.17.0.3:28967->5.161.74.111:45530: write: broken pipe\n\tstorj.io/drpc/drpcstream.(*Stream).rawFlushLocked:352\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:404\n\tstorj.io/common/pb.(*drpcPiecestore_DownloadStream).Send:317\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func4:571\n\tstorj.io/common/rpc/rpctimeout.Run.func1:22"} 2022-07-26T10:05:43.550Z        ERROR   piecestore      download failed {"Process": "storagenode", "Piece ID": "APLNPOAKRUIIFXJ6EJO3IYKLTN2SJQWKOY7YUQCHU4YGJMERE4HQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET_REPAIR", "error": "write tcp 172.17.0.3:28967->157.90.17.108:47572: write: broken pipe", "errorVerbose": "write tcp 172.17.0.3:28967->157.90.17.108:47572: write: broken pipe\n\tstorj.io/drpc/drpcstream.(*Stream).rawWriteLocked:322\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:400\n\tstorj.io/common/pb.(*drpcPiecestore_DownloadStream).Send:317\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func5.1:620\n\tstorj.io/common/rpc/rpctimeout.Run.func1:22"} 2022-07-26T11:20:28.184Z        ERROR   piecestore      download failed {"Process": "storagenode", "Piece ID": "4KM5UNVFJX5IQEXVKOK7QMAX6IGD3HIFSNFNTYXS674AV4QLG7FA", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Action": "GET_REPAIR", "error": "write tcp 172.17.0.3:28967->49.12.194.191:55854: write: broken pipe", "errorVerbose": "write tcp 172.17.0.3:28967->49.12.194.191:55854: write: broken pipe\n\tstorj.io/drpc/drpcstream.(*Stream).rawFlushLocked:352\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:404\n\tstorj.io/common/pb.(*drpcPiecestore_DownloadStream).Send:317\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func4:571\n\tstorj.io/common/rpc/rpctimeout.Run.func1:22"} 2022-07-26T11:24:13.737Z        ERROR   piecestore      download failed {"Process": "storagenode", "Piece ID": "6QJUTJOSLPPXTBBQFUBSIDF67MY65DTGZAB7ALTNICCIY25XZPUA", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Action": "GET_REPAIR", "error": "write tcp 172.17.0.3:28967->167.235.19.43:43548: write: broken pipe", "errorVerbose": "write tcp 172.17.0.3:28967->167.235.19.43:43548: write: broken pipe\n\tstorj.io/drpc/drpcstream.(*Stream).rawFlushLocked:352\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:404\n\tstorj.io/common/pb.(*drpcPiecestore_DownloadStream).Send:317\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func4:571\n\tstorj.io/common/rpc/rpctimeout.Run.func1:22"} 2022-07-26T11:25:10.583Z        ERROR   piecestore      download failed {"Process": "storagenode", "Piece ID": "TFQ6EBWEZXZBZMPZ5D6JILFO3F27JV27GMBTGOSIYAYM4VCNZFUQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET_REPAIR", "error": "write tcp 172.17.0.3:28967->5.161.50.62:60494: write: broken pipe", "errorVerbose": "write tcp 172.17.0.3:28967->5.161.50.62:60494: write: broken pipe\n\tstorj.io/drpc/drpcstream.(*Stream).rawFlushLocked:352\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:404\n\tstorj.io/common/pb.(*drpcPiecestore_DownloadStream).Send:317\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func4:571\n\tstorj.io/common/rpc/rpctimeout.Run.func1:22"} 2022-07-26T11:43:56.780Z        ERROR   piecestore      download failed {"Process": "storagenode", "Piece ID": "ELDGHPXZ7VQ65SH5H23GCVJDPC67L6CQABN5NIWVLXQ4DUFZ6KSQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET_REPAIR", "error": "write tcp 172.17.0.3:28967->5.161.44.25:51066: write: broken pipe", "errorVerbose": "write tcp 172.17.0.3:28967->5.161.44.25:51066: write: broken pipe\n\tstorj.io/drpc/drpcstream.(*Stream).rawWriteLocked:322\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:400\n\tstorj.io/common/pb.(*drpcPiecestore_DownloadStream).Send:317\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func5.1:620\n\tstorj.io/common/rpc/rpctimeout.Run.func1:22"} 2022-07-26T17:41:35.916Z        ERROR   piecestore      download failed {"Process": "storagenode", "Piece ID": "SWXA7GOLVSGZLRNDK64MUJB4KFI6M5TVJEGTZMXYL2CFB5NDLJLA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET_REPAIR", "error": "write tcp 172.17.0.3:28967->5.161.50.62:47634: write: broken pipe", "errorVerbose": "write tcp 172.17.0.3:28967->5.161.50.62:47634: write: broken pipe\n\tstorj.io/drpc/drpcstream.(*Stream).rawFlushLocked:352\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:404\n\tstorj.io/common/pb.(*drpcPiecestore_DownloadStream).Send:317\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func4:571\n\tstorj.io/common/rpc/rpctimeout.Run.func1:22"} 2022-07-26T17:41:35.917Z        ERROR   piecestore      download failed {"Process": "storagenode", "Piece ID": "RAYLQMWDAPMYEDZXN4F34KI3V5C7IFZVK4RLL5TH3FIAX63DB6MA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET_REPAIR", "error": "write tcp 172.17.0.3:28967->5.161.74.40:55242: write: broken pipe", "errorVerbose": "write tcp 172.17.0.3:28967->5.161.74.40:55242: write: broken pipe\n\tstorj.io/drpc/drpcstream.(*Stream).rawFlushLocked:352\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:404\n\tstorj.io/common/pb.(*drpcPiecestore_DownloadStream).Send:317\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func4:571\n\tstorj.io/common/rpc/rpctimeout.Run.func1:22"} 2022-07-26T18:08:19.770Z        ERROR   piecestore      download failed {"Process": "storagenode", "Piece ID": "7KTNLWAJPSPFZSQ6YAVE7V53RULBVUMQMFTVUFSFNM3LS6KBI2LQ", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Action": "GET_REPAIR", "error": "write tcp 172.17.0.3:28967->167.235.19.47:51720: write: broken pipe", "errorVerbose": "write tcp 172.17.0.3:28967->167.235.19.47:51720: write: broken pipe\n\tstorj.io/drpc/drpcstream.(*Stream).rawFlushLocked:352\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:404\n\tstorj.io/common/pb.(*drpcPiecestore_DownloadStream).Send:317\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func4:571\n\tstorj.io/common/rpc/rpctimeout.Run.func1:22"} 2022-07-26T18:21:55.715Z        ERROR   piecestore      download failed {"Process": "storagenode", "Piece ID": "UHEQPMWSKRKKVWEY56RIBIOWUX2MWJD7FR5PUXQH2FUDZUJ5KGRQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET_REPAIR", "error": "use of closed network connection", "errorVerbose": "use of closed network connection\n\tstorj.io/drpc/drpcstream.(*Stream).rawWriteLocked:322\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:400\n\tstorj.io/common/pb.(*drpcPiecestore_DownloadStream).Send:317\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func5.1:620\n\tstorj.io/common/rpc/rpctimeout.Run.func1:22"} 2022-07-26T19:13:56.019Z        ERROR   piecestore      download failed {"Process": "storagenode", "Piece ID": "XO47BXISYGRQZ2GFM4SMX3K4OUWPRIN3S4OKHV4IY6R6VHP5HIGA", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Action": "GET_REPAIR", "error": "write tcp 172.17.0.3:28967->167.235.19.47:53016: write: broken pipe", "errorVerbose": "write tcp 172.17.0.3:28967->167.235.19.47:53016: write: broken pipe\n\tstorj.io/drpc/drpcstream.(*Stream).rawFlushLocked:352\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:404\n\tstorj.io/common/pb.(*drpcPiecestore_DownloadStream).Send:317\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func4:571\n\tstorj.io/common/rpc/rpctimeout.Run.func1:22"} 2022-07-26T23:57:51.182Z        ERROR   piecestore      download failed {"Process": "storagenode", "Piece ID": "TBL7OUZINTWLX2OZOELSEDUAUMRD5JFYZS4PEZXNECVLORWHEWMA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET_REPAIR", "error": "write tcp 172.17.0.3:28967->5.161.44.25:39932: write: broken pipe", "errorVerbose": "write tcp 172.17.0.3:28967->5.161.44.25:39932: write: broken pipe\n\tstorj.io/drpc/drpcstream.(*Stream).rawFlushLocked:352\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:404\n\tstorj.io/common/pb.(*drpcPiecestore_DownloadStream).Send:317\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func4:571\n\tstorj.io/common/rpc/rpctimeout.Run.func1:22"} 2022-07-26T23:59:48.783Z        ERROR   piecestore      download failed {"Process": "storagenode", "Piece ID": "IEZFSCICXWILMY5QR37S3SIHAPIIRYWG4C3466GP2ZUX7K5CQ4YA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET_REPAIR", "error": "write tcp 172.17.0.3:28967->5.161.74.40:54706: write: broken pipe", "errorVerbose": "write tcp 172.17.0.3:28967->5.161.74.40:54706: write: broken pipe\n\tstorj.io/drpc/drpcstream.(*Stream).rawFlushLocked:352\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:404\n\tstorj.io/common/pb.(*drpcPiecestore_DownloadStream).Send:317\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func4:571\n\tstorj.io/common/rpc/rpctimeout.Run.func1:22"} 2022-07-27T04:18:21.261Z        ERROR   piecestore      download failed {"Process": "storagenode", "Piece ID": "XNIOK2LAQQO6KAEC2UG7GRAVHI2FVM6NMK6HM7QFXAJJH2NANLAA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET_REPAIR", "error": "write tcp 172.17.0.3:28967->5.161.44.25:52488: write: broken pipe", "errorVerbose": "write tcp 172.17.0.3:28967->5.161.44.25:52488: write: broken pipe\n\tstorj.io/drpc/drpcstream.(*Stream).rawWriteLocked:322\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:400\n\tstorj.io/common/pb.(*drpcPiecestore_DownloadStream).Send:317\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func5.1:620\n\tstorj.io/common/rpc/rpctimeout.Run.func1:22"} 2022-07-27T05:04:55.582Z        ERROR   piecestore      download failed {"Process": "storagenode", "Piece ID": "RWLZI26HS6E4FIKED7ZJAV7R6L6QTSDQ75OU3QSL7BLJ3SZPRVZA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET_REPAIR", "error": "use of closed network connection", "errorVerbose": "use of closed network connection\n\tstorj.io/drpc/drpcstream.(*Stream).rawWriteLocked:322\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:400\n\tstorj.io/common/pb.(*drpcPiecestore_DownloadStream).Send:317\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func5.1:620\n\tstorj.io/common/rpc/rpctimeout.Run.func1:22"}

Bivvo · July 27, 2022, 7:31am

thank you, can you modify the formatting of your code and put it in ``` at the beginning and at the end? This will support reading. Thx.

mer44 · July 27, 2022, 7:37am

So what is the issue?

It really shouldn’t be so hard to find out. Storj should give alerts saying
" 5% of audit lost
reason: … "

Or even just provide exact logs of the failed audit in the dashboard.

Bivvo · July 27, 2022, 7:51am

First let me format your input:

2022-07-26T07:36:41.878Z ERROR piecestore download failed {"Process": "storagenode", "Piece ID": "EEVXCKW5UUZ6LBC77BXCNL5364RSDG76WB3X33LLERWWHZAJQIFQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET_REPAIR", "error": "write tcp 172.17.0.3:28967->5.161.74.111:55816: write: broken pipe", "errorVerbose": "write tcp 172.17.0.3:28967->5.161.74.111:55816: write: broken pipe\n\tstorj.io/drpc/drpcstream.(*Stream).rawFlushLocked:352\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:404\n\tstorj.io/common/pb.(*drpcPiecestore_DownloadStream).Send:317\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func4:571\n\tstorj.io/common/rpc/rpctimeout.Run.func1:22"} 
2022-07-26T07:55:27.001Z ERROR piecestore download failed {"Process": "storagenode", "Piece ID": "HQFLE67QTPY4ETINK2DCZON2SFHJHCGIE6VCWT33B7CAOZWJHFYA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET_REPAIR", "error": "write tcp 172.17.0.3:28967->5.161.74.111:45530: write: broken pipe", "errorVerbose": "write tcp 172.17.0.3:28967->5.161.74.111:45530: write: broken pipe\n\tstorj.io/drpc/drpcstream.(*Stream).rawFlushLocked:352\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:404\n\tstorj.io/common/pb.(*drpcPiecestore_DownloadStream).Send:317\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func4:571\n\tstorj.io/common/rpc/rpctimeout.Run.func1:22"} 
2022-07-26T10:05:43.550Z ERROR piecestore download failed {"Process": "storagenode", "Piece ID": "APLNPOAKRUIIFXJ6EJO3IYKLTN2SJQWKOY7YUQCHU4YGJMERE4HQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET_REPAIR", "error": "write tcp 172.17.0.3:28967->157.90.17.108:47572: write: broken pipe", "errorVerbose": "write tcp 172.17.0.3:28967->157.90.17.108:47572: write: broken pipe\n\tstorj.io/drpc/drpcstream.(*Stream).rawWriteLocked:322\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:400\n\tstorj.io/common/pb.(*drpcPiecestore_DownloadStream).Send:317\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func5.1:620\n\tstorj.io/common/rpc/rpctimeout.Run.func1:22"} 
2022-07-26T11:20:28.184Z ERROR piecestore download failed {"Process": "storagenode", "Piece ID": "4KM5UNVFJX5IQEXVKOK7QMAX6IGD3HIFSNFNTYXS674AV4QLG7FA", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Action": "GET_REPAIR", "error": "write tcp 172.17.0.3:28967->49.12.194.191:55854: write: broken pipe", "errorVerbose": "write tcp 172.17.0.3:28967->49.12.194.191:55854: write: broken pipe\n\tstorj.io/drpc/drpcstream.(*Stream).rawFlushLocked:352\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:404\n\tstorj.io/common/pb.(*drpcPiecestore_DownloadStream).Send:317\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func4:571\n\tstorj.io/common/rpc/rpctimeout.Run.func1:22"} 
2022-07-26T11:24:13.737Z ERROR piecestore download failed {"Process": "storagenode", "Piece ID": "6QJUTJOSLPPXTBBQFUBSIDF67MY65DTGZAB7ALTNICCIY25XZPUA", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Action": "GET_REPAIR", "error": "write tcp 172.17.0.3:28967->167.235.19.43:43548: write: broken pipe", "errorVerbose": "write tcp 172.17.0.3:28967->167.235.19.43:43548: write: broken pipe\n\tstorj.io/drpc/drpcstream.(*Stream).rawFlushLocked:352\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:404\n\tstorj.io/common/pb.(*drpcPiecestore_DownloadStream).Send:317\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func4:571\n\tstorj.io/common/rpc/rpctimeout.Run.func1:22"} 
2022-07-26T11:25:10.583Z ERROR piecestore download failed {"Process": "storagenode", "Piece ID": "TFQ6EBWEZXZBZMPZ5D6JILFO3F27JV27GMBTGOSIYAYM4VCNZFUQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET_REPAIR", "error": "write tcp 172.17.0.3:28967->5.161.50.62:60494: write: broken pipe", "errorVerbose": "write tcp 172.17.0.3:28967->5.161.50.62:60494: write: broken pipe\n\tstorj.io/drpc/drpcstream.(*Stream).rawFlushLocked:352\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:404\n\tstorj.io/common/pb.(*drpcPiecestore_DownloadStream).Send:317\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func4:571\n\tstorj.io/common/rpc/rpctimeout.Run.func1:22"} 
2022-07-26T11:43:56.780Z ERROR piecestore download failed {"Process": "storagenode", "Piece ID": "ELDGHPXZ7VQ65SH5H23GCVJDPC67L6CQABN5NIWVLXQ4DUFZ6KSQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET_REPAIR", "error": "write tcp 172.17.0.3:28967->5.161.44.25:51066: write: broken pipe", "errorVerbose": "write tcp 172.17.0.3:28967->5.161.44.25:51066: write: broken pipe\n\tstorj.io/drpc/drpcstream.(*Stream).rawWriteLocked:322\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:400\n\tstorj.io/common/pb.(*drpcPiecestore_DownloadStream).Send:317\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func5.1:620\n\tstorj.io/common/rpc/rpctimeout.Run.func1:22"} 
2022-07-26T17:41:35.916Z ERROR piecestore download failed {"Process": "storagenode", "Piece ID": "SWXA7GOLVSGZLRNDK64MUJB4KFI6M5TVJEGTZMXYL2CFB5NDLJLA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET_REPAIR", "error": "write tcp 172.17.0.3:28967->5.161.50.62:47634: write: broken pipe", "errorVerbose": "write tcp 172.17.0.3:28967->5.161.50.62:47634: write: broken pipe\n\tstorj.io/drpc/drpcstream.(*Stream).rawFlushLocked:352\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:404\n\tstorj.io/common/pb.(*drpcPiecestore_DownloadStream).Send:317\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func4:571\n\tstorj.io/common/rpc/rpctimeout.Run.func1:22"} 
2022-07-26T17:41:35.917Z ERROR piecestore download failed {"Process": "storagenode", "Piece ID": "RAYLQMWDAPMYEDZXN4F34KI3V5C7IFZVK4RLL5TH3FIAX63DB6MA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET_REPAIR", "error": "write tcp 172.17.0.3:28967->5.161.74.40:55242: write: broken pipe", "errorVerbose": "write tcp 172.17.0.3:28967->5.161.74.40:55242: write: broken pipe\n\tstorj.io/drpc/drpcstream.(*Stream).rawFlushLocked:352\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:404\n\tstorj.io/common/pb.(*drpcPiecestore_DownloadStream).Send:317\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func4:571\n\tstorj.io/common/rpc/rpctimeout.Run.func1:22"} 
2022-07-26T18:08:19.770Z ERROR piecestore download failed {"Process": "storagenode", "Piece ID": "7KTNLWAJPSPFZSQ6YAVE7V53RULBVUMQMFTVUFSFNM3LS6KBI2LQ", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Action": "GET_REPAIR", "error": "write tcp 172.17.0.3:28967->167.235.19.47:51720: write: broken pipe", "errorVerbose": "write tcp 172.17.0.3:28967->167.235.19.47:51720: write: broken pipe\n\tstorj.io/drpc/drpcstream.(*Stream).rawFlushLocked:352\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:404\n\tstorj.io/common/pb.(*drpcPiecestore_DownloadStream).Send:317\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func4:571\n\tstorj.io/common/rpc/rpctimeout.Run.func1:22"} 
2022-07-26T18:21:55.715Z ERROR piecestore download failed {"Process": "storagenode", "Piece ID": "UHEQPMWSKRKKVWEY56RIBIOWUX2MWJD7FR5PUXQH2FUDZUJ5KGRQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET_REPAIR", "error": "use of closed network connection", "errorVerbose": "use of closed network connection\n\tstorj.io/drpc/drpcstream.(*Stream).rawWriteLocked:322\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:400\n\tstorj.io/common/pb.(*drpcPiecestore_DownloadStream).Send:317\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func5.1:620\n\tstorj.io/common/rpc/rpctimeout.Run.func1:22"} 
2022-07-26T19:13:56.019Z ERROR piecestore download failed {"Process": "storagenode", "Piece ID": "XO47BXISYGRQZ2GFM4SMX3K4OUWPRIN3S4OKHV4IY6R6VHP5HIGA", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Action": "GET_REPAIR", "error": "write tcp 172.17.0.3:28967->167.235.19.47:53016: write: broken pipe", "errorVerbose": "write tcp 172.17.0.3:28967->167.235.19.47:53016: write: broken pipe\n\tstorj.io/drpc/drpcstream.(*Stream).rawFlushLocked:352\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:404\n\tstorj.io/common/pb.(*drpcPiecestore_DownloadStream).Send:317\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func4:571\n\tstorj.io/common/rpc/rpctimeout.Run.func1:22"} 
2022-07-26T23:57:51.182Z ERROR piecestore download failed {"Process": "storagenode", "Piece ID": "TBL7OUZINTWLX2OZOELSEDUAUMRD5JFYZS4PEZXNECVLORWHEWMA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET_REPAIR", "error": "write tcp 172.17.0.3:28967->5.161.44.25:39932: write: broken pipe", "errorVerbose": "write tcp 172.17.0.3:28967->5.161.44.25:39932: write: broken pipe\n\tstorj.io/drpc/drpcstream.(*Stream).rawFlushLocked:352\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:404\n\tstorj.io/common/pb.(*drpcPiecestore_DownloadStream).Send:317\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func4:571\n\tstorj.io/common/rpc/rpctimeout.Run.func1:22"} 
2022-07-26T23:59:48.783Z ERROR piecestore download failed {"Process": "storagenode", "Piece ID": "IEZFSCICXWILMY5QR37S3SIHAPIIRYWG4C3466GP2ZUX7K5CQ4YA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET_REPAIR", "error": "write tcp 172.17.0.3:28967->5.161.74.40:54706: write: broken pipe", "errorVerbose": "write tcp 172.17.0.3:28967->5.161.74.40:54706: write: broken pipe\n\tstorj.io/drpc/drpcstream.(*Stream).rawFlushLocked:352\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:404\n\tstorj.io/common/pb.(*drpcPiecestore_DownloadStream).Send:317\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func4:571\n\tstorj.io/common/rpc/rpctimeout.Run.func1:22"} 
2022-07-27T04:18:21.261Z ERROR piecestore download failed {"Process": "storagenode", "Piece ID": "XNIOK2LAQQO6KAEC2UG7GRAVHI2FVM6NMK6HM7QFXAJJH2NANLAA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET_REPAIR", "error": "write tcp 172.17.0.3:28967->5.161.44.25:52488: write: broken pipe", "errorVerbose": "write tcp 172.17.0.3:28967->5.161.44.25:52488: write: broken pipe\n\tstorj.io/drpc/drpcstream.(*Stream).rawWriteLocked:322\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:400\n\tstorj.io/common/pb.(*drpcPiecestore_DownloadStream).Send:317\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func5.1:620\n\tstorj.io/common/rpc/rpctimeout.Run.func1:22"} 
2022-07-27T05:04:55.582Z ERROR piecestore download failed {"Process": "storagenode", "Piece ID": "RWLZI26HS6E4FIKED7ZJAV7R6L6QTSDQ75OU3QSL7BLJ3SZPRVZA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET_REPAIR", "error": "use of closed network connection", "errorVerbose": "use of closed network connection\n\tstorj.io/drpc/drpcstream.(*Stream).rawWriteLocked:322\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:400\n\tstorj.io/common/pb.(*drpcPiecestore_DownloadStream).Send:317\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func5.1:620\n\tstorj.io/common/rpc/rpctimeout.Run.func1:22"}

Can you please post the results of the following?

docker logs storagenode --since "2022-07-20T22:00" | grep -E "GET_AUDIT|GET_REPAIR|PUT_REPAIR"  | grep "failed"

docker logs storagenode --since "2022-07-20T22:00" | grep "FATAL" | grep -v "INFO"

jammerdan · July 27, 2022, 7:58am

Bivvo · July 27, 2022, 8:03am

… sounds like unstable network / internet connection.
… in case the following selections do not provide additional errors:

docker logs storagenode --since "2022-07-20T22:00" | grep -E "GET_AUDIT|GET_REPAIR|PUT_REPAIR"  | grep "failed"

docker logs storagenode --since "2022-07-20T22:00" | grep "FATAL" | grep -v "INFO"

mer44 · July 27, 2022, 9:09am

The first command returned a lot (like 100 since today 00:00) of these:
2022-07-27T07:46:40.419Z ERROR piecestore upload failed {"Process": "storagenode", "Piece ID": "MJZLHBSHMH4OBQVWIK7IKONVTYLLVUPDDMHACE2R4GSLCWQTY7GA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT_REPAIR", "error": "order: grace period passed for order limit", "errorVerbose": "order: grace period passed for order limit\n\tstorj.io/storj/storagenode/orders.(*FileStore).BeginEnqueue:86\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).beginSaveOrder:697\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:319\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:220\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:122\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:112\n\tstorj.io/drpc/drpcctx.(*Tracker).track:52", "Size": 0}

What does it mean exactly?

Ps. The second command returned nothing.

mer44 · July 27, 2022, 9:17am

I had that error before,

and the solution was supposed to be the use of timedatectl with the following result:

System clock synchronized: yes

I already have that sorted out, are there any other possible reasons for this error?

Bivvo · July 27, 2022, 9:23am

Hmm, how’s your CPU / RAM usage?
Could it be the case, that the drive is too slow to answer “in time”?
Just an idea. You should not use SMR at all.

mer44 · July 27, 2022, 9:37am

I know SMR is bad but it’s what I have.
That drive is external and plugged via usb3.0 cable. (I guess it could be the reason)
The database is handled by SSD of the pc.
top command result:
%CPU %MEM TIME+ COMMAND
1.3 16.0 347:45.09 storagenode

Alexey · July 27, 2022, 6:29pm

Only failed GET_REPAIR and GET_AUDIT can affect audit score and suspension score.
The audit score is affected when the piece doesn’t exist, piece exist but corrupted, the node cannot provide a piece within 5 minutes and was not able to do so two more times for the same piece.
The suspension score is affected in other cases.
So, the “broken pipe” during GET_AUDIT or GET_REPAIR could affect suspension score and audit score.

Seems either your network unstable, or ISP, or your router cannot keep up. In combination with SMR it makes things worse.

mer44 · July 29, 2022, 9:19am

for these two types of logs I 99.99% of times I get:

“error”: “write tcp 172.17.0.3:28967->5.161.74.40:60626: write: broken pipe”

Seems either your network unstable, or ISP, or your router cannot keep up. In combination with SMR it makes things worse.

If you were in my place, how would you troubleshoot that?
I can run any linux command on my server.

PS. In case of ISP they give me 600MB/s of simetric connection (fiber).

mer44 · July 29, 2022, 10:06am

I ran

traceroute 5.161.50.62

1 network ip: 0.490 ms 0.414 ms 0.575 ms
2 public ip: 4.267 ms 4.245 ms 4.224 ms
3 other local ip: 2.981 ms 3.135 ms 2.841 ms
4 my ISP: 5.616 ms 12.735 ms 13.047 ms

the rest is unclear to me:

Organization name (I replaced IPs) | results
5 RIPE Network Coordination Centre 13.449 ms 12.446 ms 13.406 ms
6 Level 3 Parent 103.691 ms
7 Level 3 Communications 96.946 ms * 97.254 ms
8 RIPE Network Coordination Centre 146.565 ms 117.239 ms 146.503 ms
9 RIPE Network Coordination Centre 112.418 ms 108.537 ms 108.495 ms
10 RIPE Network Coordination Centre 108.997 ms 109.395 ms *
11 RIPE Network Coordination Centre 102.460 ms 118.370 ms 102.713 ms
12 RIPE Network Coordination Centre 113.324 ms 113.315 ms

Is this good / bad ?

If ran

traceroute google.com

the list is much shorter:
1 network ip: 0.506 ms 0.633 ms 0.728 ms
2 public ip: 4.227 ms 4.199 ms 4.171 ms
3 other local ip: 3.125 ms 3.103 ms 3.278 ms
4 my ISP: 11.912 ms 12.937 ms 6.057 ms
4 my ISP again: 10.915 ms 12.318 ms 14.529 ms
5 Google LLC: 14.449 ms 12.020 ms 11.984 ms
6 Google LLC: 11.489 ms 11.918 ms 11.897 ms
7 Google LLC: 11.126 ms 12.309 ms 12.177 ms

Alexey · July 30, 2022, 11:09am

First I would try to reboot a router and monitor for a while - is the number of errors the same or reduced.
Next - I would check all firewalls to do not block traffic from the storagenode (to any host and any port from any port) and to storagenode (from any host and any port to the TCP+UDP port of storagenode).
If you use a VM, then try do not use the VM and run docker container directly on your host.
Sometimes configuration of the network could be non-standard, see Node could not start since yesterday - #13 by ChristianDresel

Such errors could be also because of some hardware problems too (drop packets), but troubleshoot this type of issues is much harder.

mer44 · July 30, 2022, 4:35pm

I’m back to almost 100% on all satelites, I seem to always get issues in the end of each month.

(I went as low as 80%)

Storagenode (docker) is not running in VM btw but directly on the server.
When the issues come back I will try rebooting the router.
The only port I opened for storj was 28967, are there other ports that need opening?

Knowledge · July 30, 2022, 5:24pm

Do you have both TCP and UDP open?

mer44 · July 31, 2022, 8:55am

Yes both TCP and UDP is opened for port 28967, this is going to be 4th month that I run that storj node I think I would have issues before.

mer44 · July 31, 2022, 9:00am

audits and repairs do a lot of file manipulation or in general increese disk usage, right?
I think that’s when my SMR drive will fail to keep up.

I used

iostat -dx /dev/drive 5

and on the day that there was so many issues I had:

%util 80 -140
f_await 100 - 3000

and now I have

%util between 20 and 40
f_await 100 - 200

Interestingly %wrqm was high then and now.
Does it help? I can compare more columns.