ERROR use of closed network connection

Node Operators troubleshooting
1

I am really amazed how whenever I fix, or should I say - hack my way out of issues out of my control and system, some storagenode error, another occurs no matter what. And most probably someone will now say, the problem is on my end again…

Nov 09 06:56:35 hostname storagenode[1263]: 2022-11-09T06:56:35.337Z        ERROR        piecestore        upload failed        {"Process": "storagenode", "Piece ID": "5D4FZ7YDUIR2J6H7JI2JBOJMU5FQYF3LD2EXUF63GRWUAWVS7F4A", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "error": "unexpected EOF", "errorVerbose": "unexpected EOF\n\tstorj.io/common/rpc/rpcstatus.Error:82\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:349\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:220\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:61\n\tstorj.io/common/experiment.(*Handler).HandleRPC:42\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:122\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:112\n\tstorj.io/drpc/drpcctx.(*Tracker).track:52", "Size": 0}
Nov 09 07:00:40 hostname storagenode[1263]: 2022-11-09T07:00:40.945Z        ERROR        piecestore        download failed        {"Process": "storagenode", "Piece ID": "LSX5VFGRS367KMTWF2SWW3UFYO66F74NFENTGR25WI62KS3VCN3A", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "error": "write tcp XXX.XXX.XXX.XXX:28967->5.161.154.96:27702: use of closed network connection", "errorVerbose": "write tcp XXX.XXX.XXX.XXX:28967->5.161.154.96:27702: use of closed network connection\n\tstorj.io/drpc/drpcstream.(*Stream).rawWriteLocked:326\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:404\n\tstorj.io/common/pb.(*drpcPiecestore_DownloadStream).Send:317\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func5.1:628\n\tstorj.io/common/rpc/rpctimeout.Run.func1:22"}
Nov 09 07:01:52 hostname storagenode[1263]: 2022-11-09T07:01:52.295Z        ERROR        piecestore        download failed        {"Process": "storagenode", "Piece ID": "3MDUG6UHGKN6VPY5QVB4G2VCZGZAQYEORPS33IQPHZXUKW55PTPQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET", "error": "write tcp XXX.XXX.XXX.XXX:28967->149.6.140.107:28492: use of closed network connection", "errorVerbose": "write tcp XXX.XXX.XXX.XXX:28967->149.6.140.107:28492: use of closed network connection\n\tstorj.io/drpc/drpcstream.(*Stream).rawWriteLocked:326\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:404\n\tstorj.io/common/pb.(*drpcPiecestore_DownloadStream).Send:317\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func5.1:628\n\tstorj.io/common/rpc/rpctimeout.Run.func1:22"}
Nov 09 07:09:04 hostname storagenode[1263]: 2022-11-09T07:09:04.195Z        ERROR        piecestore        upload failed        {"Process": "storagenode", "Piece ID": "57WQT22PERCSVSOWXB5FJU7GCORJPECNHPI5OJLBRTI77JCDRNTQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "error": "unexpected EOF", "errorVerbose": "unexpected EOF\n\tstorj.io/common/rpc/rpcstatus.Error:82\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:349\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:220\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:61\n\tstorj.io/common/experiment.(*Handler).HandleRPC:42\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:122\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:112\n\tstorj.io/drpc/drpcctx.(*Tracker).track:52", "Size": 0}
Nov 09 07:14:31 hostname storagenode[1263]: 2022-11-09T07:14:31.370Z        ERROR        piecestore        download failed        {"Process": "storagenode", "Piece ID": "E44LQPO647C4NUDLIQHRZ7AD6THSATR65HY7VOAK5NSQSMAZNBJQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET", "error": "write tcp XXX.XXX.XXX.XXX:28967->149.6.140.107:59312: use of closed network connection", "errorVerbose": "write tcp XXX.XXX.XXX.XXX:28967->149.6.140.107:59312: use of closed network connection\n\tstorj.io/drpc/drpcstream.(*Stream).rawWriteLocked:326\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:404\n\tstorj.io/common/pb.(*drpcPiecestore_DownloadStream).Send:317\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func5.1:628\n\tstorj.io/common/rpc/rpctimeout.Run.func1:22"}
Nov 09 07:17:24 hostname storagenode[1263]: 2022-11-09T07:17:24.262Z        ERROR        piecestore        upload failed        {"Process": "storagenode", "Piece ID": "36RFDWNK57K2OSDDACYKDHRGIH2KAVJ4EVEWY67IEVQ326BW5EWQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "error": "unexpected EOF", "errorVerbose": "unexpected EOF\n\tstorj.io/common/rpc/rpcstatus.Error:82\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:349\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:220\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:61\n\tstorj.io/common/experiment.(*Handler).HandleRPC:42\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:122\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:112\n\tstorj.io/drpc/drpcctx.(*Tracker).track:52", "Size": 0}

What is this now and why does it even happen? How could this happen on a super fast machine with 40 CPU cores, tons of disks in fast hardware raid and 1Gbps connections on each network port??

2

firewall dont like your trafic? Provider try to prevent p2p connections?

3

Should not be firewall. It is properly configured.

4

There are two possible reasons:

  1. your node lose the race, some nodes were faster
  2. if you have a lot of such errors with different sources and destinations - it may indicate the network problem, as @Vadim stated - it could be your firewall, router or ISP. But in this case you should also see i/o timeouts and/or “no route to host” errors. It could be also related to the number of file descriptors, if you use Linux (each socket is a file)
5

Keeps showing up. Example…

Nov 28 12:58:30 host storagenode[169618]: 2022-11-28T12:58:30.412Z        ERROR        piecestore        download failed        {"Process": "storagenode", "Piece ID": "L5AJYQ6ROJFIAF624RVI7P5GVRTBOBWZCSJOQJUNO3SUCW3WY3JQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "error": "use of closed network connection", "errorVerbose": "use of closed network connection\n\tstorj.io/drpc/drpcstream.(*Stream).rawWriteLocked:326\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:404\n\tstorj.io/common/pb.(*drpcPiecestore_DownloadStream).Send:317\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func5.1:628\n\tstorj.io/common/rpc/rpctimeout.Run.func1:22"}
6

This is normal. If your node is too slow (to upload or to download) this will be the log entry.

7

My node is not slow. Maybe some could be faster, but I doubt it someone uses SSDs to make their node faster… LOL

8

If you search the forum you will see some users do use SSD - either directly for the storage or in a cache. Also connections are coming from all over the internet so your node can’t win all races, e.g. if you are in US but the request comes from Asia, there will be ‘quicker’ nodes in Asia.

3 Likes
9

In neither of regions, but makes sense and most probably is just that. Problem is I need to know, IF the case is such and whether it demands any attention, hence action on my end. Guess not then…

10

in this logs, there is IP usually, then you can try to trace them and understand that how far are they.

11

And you should stop all malware protections in your router or pi hole etc. Also DDOS protection must be OFF in router. The Firewall in router keep it ON.

12

I’m a little surprised by the responses so far in this topic. As far as I’m aware these errors happen on every node every now and then. It’s a simple result of long tail cancellation interrupting the transfer. Depending on where in the code that interruption happens you usually see download cancelled, but in more rare cases these errors show up. It’s not something to worry about unless it happens a lot. But I don’t think anyone can completely get rid of these errors. Sometimes the geographical distance to other nodes compared to yours simply make it impossible to win all races.

And yes, I do use SSD cached storage. I still see these errors from time to time.

2 Likes