Are these regular download errors?

Node Operators troubleshooting
1

I see different types of download errors and I’d like to know if these types are regular ones (like when a download fails for valid reasons).

I see following on “download failed” log

“Connection reset by peer”
“broken pipe”
“use of closed network connection”

2

What the method you have used to download your files?

3

These error messages are from the storage node log. I understand that downloads fail when other nodes are faster. I am just wondering if these errors indicate that or if they indicate a different problem.

4

Those errors could have a canceled in their description as well. So it’s either long tail cut or the canceled upload by the customer.

5

I don’t see canceled with them:
2020-03-20T02:54:47.757Z ERROR piecestore download failed {“Piece ID”: “2BGJWGHGAW7PA3REMY2J53TDONBU3YNBJS4C5OJ6ZYDWF34A3WVA”, “Satellite ID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”, “Action”: “GET”, “error”: “write tcp 172.17.0.2:28967->34.95.8.124:59940: write: connection reset by peer”, “errorVerbose”: “write tcp 172.17.0.2:28967->34.95.8.124:59940: write: connection reset by peer\n\tstorj.io/drpc/drpcstream.(*Stream).pollWrite:221\n\tstorj.io/drpc/drpcwire.SplitN:29\n\tstorj.io/drpc/drpcstream.(*Stream).RawWrite:276\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:318\n\tstorj.io/common/pb.(*drpcPiecestoreDownloadStream).Send:1079\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload.func5.1:648\n\tstorj.io/common/rpc/rpctimeout.Run.func1:22”}

6

What version are you running on?

7

The version is v0.34.6.

8

I would have expected this one to be a download cancelled on that version. Either way, this doesn’t seem like anything to worry about.

9

There are too many of them to not to worry.
If the log is correct it starts on March 19 and there are 4366 occurrences of failed downloads since then seemingly with one of those errors above.

10

Well that may suggest some connection issues. I’d start by giving everything a reboot. Node hardware, routers, switches, modems. I hope you’re not on wifi, but if so, connect a cable and make sure they’re all plugged in well.

11

I am not sure but I believe it started with 0.34.6.
I guess I will wait for 0.35.5 before I will further investigate.

12

No, I am not on a wifi and everything else is working.
This really kills my download rate:

========== DOWNLOAD ===========

Failed: 1177
Fail Rate: 11.397%
Canceled: 11
Cancel Rate: 0.107%
Successful: 9139
Success Rate: 88.496%

My failed rate is 100 fold the cancel rate. i don’t think it should be this way.

13

When I lookup the error response use of closed network connection then Google points to some Golang specific issues.

I wish some Storj folks would look into this as I am not sure if the issue is server/connection specific on my side or if there is some application issue. Or maybe even Docker related. I have no idea.

Edit: I am not even sure if it is ip related. It seems that certain ips come up many times with download failed error in the log, e.g.
34.95.11.198 -> 94 times
34.83.3.52 -> 9 times
34.95.45.148 -> 107 times

As far as I see the log does not contain ip information for successful downloads. So I cannot tell I there are any successful downloads for these ips at all.

14

How is your HDD connected to the PC?

15

Internal Sata port connection.

16

Then there is only location playing role.
Is your node vetted?

18

Yes the node is vetted.

Here is my actual download stats for the last 40 hours:

Failed:                2186
Fail Rate:             15.664%
Canceled:              14
Cancel Rate:           0.100%
Successful:            11756
Success Rate:          84.236%

First of all it is required to understand what is the difference between failed download and canceled download. I understand it that way that a canceled download is a lost race due to speed.
So the huge number of failed downloads seem to be something entirely different.

What kind of location thing could result in 2186 failed downloads and which are not lost races?
In numbers these are 15% of paid downloads that fail that’s why I need to know why.

Is there a way to get more debug information about the failed downloads?

19

I do have 5% of failed downloads, 16% cancelled.
Not entirely sure what the difference is either.
But looking at my logs, a failed download occurs when the peer connection is reset… Maybe that means that the client aborted the download while cancelled errors mean I have lost the race?

20

Currently I do not understand the meaning of the errors. I am totally fine with them if it is like you say, when the client is not happy and leaves.
But if these errors happen because my node is not working properly then I would like to know what the issue is and solve it.
By the way 15% in my case sounds really really high.

It may not really be comparable but my failed upload rate is 0,046%

21

I’m seeing similar errors appear from time to time.

The difference between cancelled and failed is very simple. Only context cancelled errors are now shown as cancelled. Unfortunately those aren’t the only ones that can occur because of “losing the race”. They are the most common ones though. Now I’m not entirely sure that these errors result from losing the race as well, but that is my estimate right now.