EOF errors since update to 1.107.3

Node Operators troubleshooting
1

getting these types of errors on one node every 10 - 20 minutes:

{L:ERROR,T:2024-07-04T19:05:55Z,N:piecestore,M:upload internal error,Process:storagenode,error:EOF}

any ideas?

2

Please provide the whole error, starting from a timestamp.

4

it’s the whole error (json log format).

Here are the logs short before and after the error:

{"L":"INFO","T":"2024-07-04T19:05:54Z","N":"piecestore","M":"upload started","Process":"storagenode","Piece ID":"7R6ZPVQBUZISREDELMJNMOCBA3OGVHGPNEJV6POOETBSWCHHC3DQ","Satellite ID":"12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs","
Action":"PUT","Remote Address":"172.22.0.3:55740","Available Space":7819678272146}
{"L":"INFO","T":"2024-07-04T19:05:54Z","N":"piecestore","M":"uploaded","Process":"storagenode","Piece ID":"7R6ZPVQBUZISREDELMJNMOCBA3OGVHGPNEJV6POOETBSWCHHC3DQ","Satellite ID":"12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs","Action
":"PUT","Remote Address":"172.22.0.3:55740","Size":5120}
{"L":"INFO","T":"2024-07-04T19:05:54Z","N":"piecestore","M":"download started","Process":"storagenode","Piece ID":"4G7CMEIOMXLIC6YXCGKLVUTVOO6V4VLF5QQG4FCQLX2RIVDWTDZQ","Satellite ID":"12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S"
,"Action":"GET","Offset":0,"Size":13824,"Remote Address":"172.22.0.3:50776"}
{"L":"INFO","T":"2024-07-04T19:05:54Z","N":"piecestore","M":"upload started","Process":"storagenode","Piece ID":"2R7EVGUFBUSHRL2GAICB3IL26C7T2WFB46RFR5IMMBNH2PSIW6IA","Satellite ID":"12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S","
Action":"PUT","Remote Address":"172.22.0.3:48694","Available Space":7819678266514}
{"L":"INFO","T":"2024-07-04T19:05:54Z","N":"piecestore","M":"uploaded","Process":"storagenode","Piece ID":"2R7EVGUFBUSHRL2GAICB3IL26C7T2WFB46RFR5IMMBNH2PSIW6IA","Satellite ID":"12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S","Action
":"PUT","Remote Address":"172.22.0.3:48694","Size":65024}
{"L":"INFO","T":"2024-07-04T19:05:54Z","N":"piecestore","M":"upload started","Process":"storagenode","Piece ID":"X3AXLGU5IXVJJGUJGG7KRO2XYSIQ5OHXXQRPTXZIDC532I75ZKLQ","Satellite ID":"12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S","
Action":"PUT","Remote Address":"172.22.0.3:37368","Available Space":7819678200978}
{"L":"INFO","T":"2024-07-04T19:05:54Z","N":"piecestore","M":"downloaded","Process":"storagenode","Piece ID":"4G7CMEIOMXLIC6YXCGKLVUTVOO6V4VLF5QQG4FCQLX2RIVDWTDZQ","Satellite ID":"12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S","Acti
on":"GET","Offset":0,"Size":13824,"Remote Address":"172.22.0.3:50776"}
{"L":"INFO","T":"2024-07-04T19:05:54Z","N":"piecestore","M":"uploaded","Process":"storagenode","Piece ID":"X3AXLGU5IXVJJGUJGG7KRO2XYSIQ5OHXXQRPTXZIDC532I75ZKLQ","Satellite ID":"12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S","Action
":"PUT","Remote Address":"172.22.0.3:37368","Size":289792}
{"L":"INFO","T":"2024-07-04T19:05:55Z","N":"piecestore","M":"upload started","Process":"storagenode","Piece ID":"Y3VOSO7K5WDCZ3JCPSI5PWD7EGBMUMR2GR2LLZTX7MCMKRQCHRJA","Satellite ID":"12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S","
Action":"PUT","Remote Address":"172.22.0.3:48742","Available Space":7819677910674}
{"L":"ERROR","T":"2024-07-04T19:05:55Z","N":"piecestore","M":"upload internal error","Process":"storagenode","error":"EOF"}
{"L":"INFO","T":"2024-07-04T19:05:55Z","N":"piecestore","M":"upload started","Process":"storagenode","Piece ID":"WO2ASQDHAHSPZQJPWPDEWP7UYAA4MNEZP24MJIRF7N7DMFXV6F3Q","Satellite ID":"12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S","
Action":"PUT","Remote Address":"172.22.0.3:56916","Available Space":7819677910674}
{"L":"INFO","T":"2024-07-04T19:05:55Z","N":"piecestore","M":"uploaded","Process":"storagenode","Piece ID":"WO2ASQDHAHSPZQJPWPDEWP7UYAA4MNEZP24MJIRF7N7DMFXV6F3Q","Satellite ID":"12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S","Action
":"PUT","Remote Address":"172.22.0.3:56916","Size":768}
5

You can ignore this error. We all have them.

2024-07-05T13:14:47Z ERROR piecestore upload failed { “Piece ID”: xxxx “Action”: “PUT”, “Remote Address”: “sss”, “Size”: 917504, “error”: “manager closed: unexpected EOF”, “errorVerbose”: “manager closed: unexpected EOF\n\tgithub.com/jtolio/noiseconn.(*Conn).readMsg:225\n\tgithub.com/jtolio/noiseconn.(*Conn).Read:171\n\tstorj.io/drpc/drpcwire.(*Reader).read:68\n\tstorj.io/drpc/drpcwire.(*Reader).ReadPacketUsing:113\n\tstorj.io/drpc/drpcmanager.(*Manager).manageReader:229”}

2 Likes
6

i think it’s a different error. Log was introduced with this changeset: storagenode/piecestore: improve Upload endpoint logging · storj/storj@fd62c4d · GitHub

7

That commit hasn’t been merged yet.

8

https://github.com/storj/storj/blame/v1.107.3/storagenode/piecestore/endpoint.go#L307

really?

9

If it’s related to PUT/GET, then it’s a usual long tail cancelation, nothing to worry about.

3 Likes