Got alot of "Unexpected EOF" on my node

Node Operators troubleshooting
, ,
1

Hi!

I got alot of unexpected EOF errors like these:

|ERROR|collector|unable to delete piece|{Process: storagenode, Satellite ID: 12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S, Piece ID: LO3JE4ZK42XLD4ZOCMU7IXF3WQPO4EWWBDRYDFJVDMYU5TJUVORA, error: pieces error: context canceled; v0pieceinfodb: context canceled, errorVerbose: pieces error: context canceled; v0pieceinfodb: context canceled\n\tstorj.io/storj/storagenode/pieces.(*Store).DeleteExpired:370\n\tstorj.io/storj/storagenode/pieces.(*Store).Delete:349\n\tstorj.io/storj/storagenode/collector.(*Service).Collect:97\n\tstorj.io/storj/storagenode/collector.(*Service).Run.func1:57\n\tstorj.io/common/sync2.(*Cycle).Run:99\n\tstorj.io/storj/storagenode/collector.(*Service).Run:53\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:51\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78}|

|ERROR|collector|unable to update piece info|{Process: storagenode, Satellite ID: 12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S, Piece ID: HXVLFWVS6XSI5VJ25M6735IHKQIGUKDSSBVFK2MQQ5ELJTP6BUJA, error: pieceexpirationdb: context canceled, errorVerbose: pieceexpirationdb: context canceled\n\tstorj.io/storj/storagenode/storagenodedb.(*pieceExpirationDB).DeleteFailed:99\n\tstorj.io/storj/storagenode/pieces.(*Store).DeleteFailed:618\n\tstorj.io/storj/storagenode/collector.(*Service).Collect:109\n\tstorj.io/storj/storagenode/collector.(*Service).Run.func1:57\n\tstorj.io/common/sync2.(*Cycle).Run:99\n\tstorj.io/storj/storagenode/collector.(*Service).Run:53\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:51\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78}|

there is a less volume of ERRORs like these:

ERROR	piecestore	upload failed	{"Process": "storagenode", "Piece ID": "AOJJU5RHHBSXTK34JX5EFLLL57CMG7I7LMJRGVHJCYPDFB5JPEJA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Remote Address": "172.17.0.1:46232", "Size": 1769472, "error": "manager closed: unexpected EOF", "errorVerbose": "manager closed: unexpected EOF\n\tgithub.com/jtolio/noiseconn.(*Conn).readMsg:225\n\tgithub.com/jtolio/noiseconn.(*Conn).Read:171\n\tstorj.io/drpc/drpcwire.(*Reader).read:68\n\tstorj.io/drpc/drpcwire.(*Reader).ReadPacketUsing:113\n\tstorj.io/drpc/drpcmanager.(*Manager).manageReader:229"}

ERROR	piecestore	download failed	{"Process": "storagenode", "Piece ID": "7QQRY6WAMH5UHZBMJ44IEISWE5GCYYW743LZSJ47ZYTOR4GVGMXQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET", "Offset": 0, "Size": 381952, "Remote Address": "172.17.0.1:41674", "error": "context deadline exceeded", "errorVerbose": "context deadline exceeded\n\tstorj.io/common/rpc/rpcstatus.Wrap:76\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func8:829\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:840\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:302\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:61\n\tstorj.io/common/experiment.(*Handler).HandleRPC:42\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:167\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:109\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:157\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35"}

And this pack I’ve got in Febuary:

ERROR	piecestore	failed to add bandwidth usage	{"process": "storagenode", "error": "bandwidthdb: database is locked", "errorVerbose": "bandwidthdb: database is locked\n\tstorj.io/storj/storagenode/storagenodedb.(*bandwidthDB).Add:60\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).beginSaveOrder.func1:913\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:542\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:243\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:61\n\tstorj.io/common/experiment.(*Handler).HandleRPC:42\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:124\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:114\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35"}

WARN	retain	failed to delete piece	{"process": "storagenode", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Piece ID": "SKGT7W6KVM4JOXQPVL3KF7SSDS4FMRP2URWWKSY5ZN5PA3CP4SKA", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist\n\tstorj.io/storj/storagenode/blobstore/filestore.(*blobStore).Stat:110\n\tstorj.io/storj/storagenode/pieces.(*BlobsUsageCache).pieceSizes:245\n\tstorj.io/storj/storagenode/pieces.(*BlobsUsageCache).Trash:290\n\tstorj.io/storj/storagenode/pieces.(*Store).Trash:404\n\tstorj.io/storj/storagenode/retain.(*Service).trash:364\n\tstorj.io/storj/storagenode/retain.(*Service).retainPieces:341\n\tstorj.io/storj/storagenode/retain.(*Service).Run.func2:221\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75"}

Should I do something with them or it’s normal behaviour?
I’m just stressed up because all my nodes in a 2 months loses significant part of storage (about 1-1,5 TB went in trash).

2

The first one is related to your disk is too slow. The second one (with canceled uploads) is likely related to the same problem too, but usually it’s mean that your node is too slow to accept the upload and the customer canceled it. The third error again related to your slow disk, but now this affect databases also.
I would recommend to stop the node, check the disk for errors and fix them. If this is NTFS, you also need to perform a defragmentation and enable the automatic defragmentation for this disk (it’s enabled by default).
If you use any network filesystem (SMB/CIFS, NFS, etc.), then you need to migrate to iSCSI or run your node directly on your file server.
To help databases to be not locked you need to move them to SSD, if the problem with the slow disk would not be fixed by suggestions above.

2 Likes