Same piece canceled all the time

Node Operators troubleshooting
1

I usually do some successrate monitoring to detect early problems before it could affect my node.

I just seen that today (I rotate logs every day) I have a total of 622 canceled uploads in 6h (87% of cancel rate), which is a lot, and could mean some kind of slow response or bottleneck on the storage layer of my (still young) node.

However, inspecting the logs, I have seen that almos all canceled uploads (604) are from the same satellite and piece:

2020-11-13T06:48:38.081Z INFO piecestore download canceled {"Piece ID": "FQLFLO3DJJ2QC27ITIQYMU6YSZBHF43GMPEJTTIGAUZNP325CRZA", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET"}

It looks that in the past 6h have been:

  • Requested a total of 677 times
  • Canceled a total of 604 times
  • Downloaded a total of 69 times

I suppose that this behaviour may be normal because it depends on the end user using tardigrade, but sometimes the download gets canceled in less than 500ms from the start, and my node even received few download requests of that same piece (all canceled) in a row:

2020-11-13T06:54:04.479Z        INFO    piecestore      download started        {"Piece ID": "FQLFLO3DJJ2QC27ITIQYMU6YSZBHF43GMPEJTTIGAUZNP325CRZA", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET"}
2020-11-13T06:54:04.992Z        INFO    piecestore      download canceled       {"Piece ID": "FQLFLO3DJJ2QC27ITIQYMU6YSZBHF43GMPEJTTIGAUZNP325CRZA", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET"}
2020-11-13T06:54:11.612Z        INFO    piecestore      download started        {"Piece ID": "FQLFLO3DJJ2QC27ITIQYMU6YSZBHF43GMPEJTTIGAUZNP325CRZA", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET"}
2020-11-13T06:54:12.088Z        INFO    piecestore      download canceled       {"Piece ID": "FQLFLO3DJJ2QC27ITIQYMU6YSZBHF43GMPEJTTIGAUZNP325CRZA", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET"}
2020-11-13T06:54:49.131Z        INFO    piecestore      download started        {"Piece ID": "FQLFLO3DJJ2QC27ITIQYMU6YSZBHF43GMPEJTTIGAUZNP325CRZA", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET"}
2020-11-13T06:54:49.613Z        INFO    piecestore      download canceled       {"Piece ID": "FQLFLO3DJJ2QC27ITIQYMU6YSZBHF43GMPEJTTIGAUZNP325CRZA", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET"}
2020-11-13T06:55:54.094Z        INFO    piecestore      download started        {"Piece ID": "FQLFLO3DJJ2QC27ITIQYMU6YSZBHF43GMPEJTTIGAUZNP325CRZA", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET"}
2020-11-13T06:55:54.570Z        INFO    piecestore      download canceled       {"Piece ID": "FQLFLO3DJJ2QC27ITIQYMU6YSZBHF43GMPEJTTIGAUZNP325CRZA", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET"}
2020-11-13T06:56:02.394Z        INFO    piecestore      download started        {"Piece ID": "FQLFLO3DJJ2QC27ITIQYMU6YSZBHF43GMPEJTTIGAUZNP325CRZA", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET"}
2020-11-13T06:56:02.859Z        INFO    piecestore      download canceled       {"Piece ID": "FQLFLO3DJJ2QC27ITIQYMU6YSZBHF43GMPEJTTIGAUZNP325CRZA", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET"}
2020-11-13T06:56:18.851Z        INFO    piecestore      download started        {"Piece ID": "FQLFLO3DJJ2QC27ITIQYMU6YSZBHF43GMPEJTTIGAUZNP325CRZA", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET"}
2020-11-13T06:56:19.317Z        INFO    piecestore      download canceled       {"Piece ID": "FQLFLO3DJJ2QC27ITIQYMU6YSZBHF43GMPEJTTIGAUZNP325CRZA", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET"}
2020-11-13T06:56:25.709Z        INFO    piecestore      download started        {"Piece ID": "FQLFLO3DJJ2QC27ITIQYMU6YSZBHF43GMPEJTTIGAUZNP325CRZA", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET"}
2020-11-13T06:56:26.204Z        INFO    piecestore      download canceled       {"Piece ID": "FQLFLO3DJJ2QC27ITIQYMU6YSZBHF43GMPEJTTIGAUZNP325CRZA", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET"}

Is there something I should worry about?

1 Like
2

If your node passing audits and do have other uploads/downloads - there is nothing to worry about. The customers’ behavior is not predictable anyway.

2 Likes
3

All audits are fine, so I suppose that 10% of success downloads of the same piece is welcome as an extra egress :joy:

4

Exactly :slight_smile:

5

500ms can be a very long time… i mean in 500 ms if we subtract 200ms which is what i think my usual latency is to the united states from here… and i get like 200-300mbit bandwidth, so any file piece size less than 2.4-3.6MB i would be able to serve easily in 500 ms most likely across most of the world if their internet is up for it… and lets say 1/3 of a sec in europe that would put me at 20MB transferred, with all latency included from request is sent from the client until the data is uploaded.

and really i grossly over estimate the latency, so it could be more… but internet bandwidth isn’t to great when moving out across the globe…

500ms can be a long time, but i optimized to reduce latency and i doubt people can get their latency much lower than this.

so what i’m trying to say is that, in some cases some pieces would exist in places where you might not be able to compete.

1 Like
6

Completely agree.

My real concern was not losing those uploads, it was that the same exact piece was downloaded thousand of time, which is a weird behaviour.

But I suppose it was a real cuatomer behaviour, so they are legit requests :sweat_smile:

1 Like
7

don’t forget one account can have many users and thus use the network to distribute data across the world, which could explain hundreds or even thousands of downloads…

or atleast thats how i understand it, tardigrade and storagenodes is a rather complex topic and i’m not that educated in the nitty gritty of it.

1 Like