GET operations repeatedly fails just after the begining

Node Operators troubleshooting
1

Hi all. Today I checked my storagenode logs and noticed a very strange behavior: almost all download operations are failing (2 days ago everything was fine).

Audit requests is OK, upload (PUT) operations are fine too, but download requests are failing less than a second after start.

Here is some logs:

Example:

2019-08-26T23:28:42.518Z INFO piecestore download started {“Piece ID”: “YBWI5NEPK7TUYJIAKXESYFRXDDMJLVY3FHP6GPLOUTWJ3ABIGX3Q”, “SatelliteID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “GET_AUDIT”}
2019-08-26T23:28:42.870Z INFO piecestore downloaded {“Piece ID”: “YBWI5NEPK7TUYJIAKXESYFRXDDMJLVY3FHP6GPLOUTWJ3ABIGX3Q”, “SatelliteID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “GET_AUDIT”}

(Audit is OK)

2019-08-26T23:28:52.817Z INFO piecestore download started {“Piece ID”: “DG225BAD6DXM4OB2GRTXD6Z3HHRD42PIX7JCCHZVLZ5RWZJB3MNA”, “SatelliteID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “GET”}
2019-08-26T23:28:52.902Z INFO piecestore download failed {“Piece ID”: “DG225BAD6DXM4OB2GRTXD6Z3HHRD42PIX7JCCHZVLZ5RWZJB3MNA”, “SatelliteID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “GET”, “error”: … (error is “context canceled”, stack trace is omitted)

(download failed just after start)

As you can see, the difference between “download started” and “download failed” events is less than 100 milliseconds, and I have many examples like this in my log - is this OK?

UPD: (maybe this helps): failed GET requests happens for the Sattelites 12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S and 12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs.

GET Requests from satellite 118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW processed without errors.

2

I’m going to take a guess and say that you are somewhere in or near europe. In which case you are simply closer to the satellite for which you don’t see context cancelled errors. (Or technically closer to the customers using that satellite) Others are closer to the other satellites which means you generally don’t win the race against those nodes. It’s not something to worry about, optimize your setup as much as you can by using internal directly connected storage and be happy with the races you do win.

2 Likes