Consistent "upload canceled"

Alexey · November 4, 2021, 6:35am

The upload is free, so it is possible. Someone could limit nodes by IPs. They could use only geo-limited nodes for example, but why to start upload at all?

Pentium100 · November 4, 2021, 6:43am

Is the size shown in the log the actual size of the piece or just the amount of data that was uploaded before the upload was canceled?

Alexey · November 4, 2021, 6:52am

Accordingly code it’s an upload size.

github.com

storj/storj/blob/2df41028a3ec9e5026c0b9691367fb6c82da8e99/storagenode/piecestore/endpoint.go#L281

    
      
          			mon.Meter("upload_failure_byte_meter").Mark64(uploadSize)
          			mon.IntVal("upload_failure_size_bytes").Observe(uploadSize)
          			mon.IntVal("upload_failure_duration_ns").Observe(uploadDuration)
          			mon.FloatVal("upload_failure_rate_bytes_per_sec").Observe(uploadRate)
          			endpoint.log.Error("upload failed", zap.Stringer("Piece ID", limit.PieceId), zap.Stringer("Satellite ID", limit.SatelliteId), zap.Stringer("Action", limit.Action), zap.Error(err), zap.Int64("Size", uploadSize))
          		} else if errs2.IsCanceled(err) && !committed {
          			mon.Meter("upload_cancel_byte_meter").Mark64(uploadSize)
          			mon.IntVal("upload_cancel_size_bytes").Observe(uploadSize)
          			mon.IntVal("upload_cancel_duration_ns").Observe(uploadDuration)
          			mon.FloatVal("upload_cancel_rate_bytes_per_sec").Observe(uploadRate)
          			endpoint.log.Info("upload canceled", zap.Stringer("Piece ID", limit.PieceId), zap.Stringer("Satellite ID", limit.SatelliteId), zap.Stringer("Action", limit.Action), zap.Int64("Size", uploadSize))
          		} else {
          			mon.Meter("upload_success_byte_meter").Mark64(uploadSize)
          			mon.IntVal("upload_success_size_bytes").Observe(uploadSize)
          			mon.IntVal("upload_success_duration_ns").Observe(uploadDuration)
          			mon.FloatVal("upload_success_rate_bytes_per_sec").Observe(uploadRate)
          			endpoint.log.Info("uploaded", zap.Stringer("Piece ID", limit.PieceId), zap.Stringer("Satellite ID", limit.SatelliteId), zap.Stringer("Action", limit.Action), zap.Int64("Size", uploadSize))
          		}
          	}()
          
          
	endpoint.log.Info("upload started",

clearing · November 4, 2021, 6:58am

in my observation, when i get an upload canceled (with those sizes) the momentum of my node transactions are disrupted. im not sure if thats really a thing in satellites’ selection of node? but, if that really has an effect, then more chances for their nodes to be selected, which will get the legit transactions and eventually no need for them to do fake transaction when their storage are full of legit file pieces.

Alexey · November 4, 2021, 7:12am

Node selection doesn’t have a such functionality, but there is a short-living cache of successful nodes, so these nodes would be offered first (if they meet the unique limit by subnet for the pieces of the same segment).

clearing · November 4, 2021, 7:28am

thanks for clarifying.

few questions about this, how do we determine the successful nodes that will be included in the cache? will the upload canceled affect that?

Alexey · November 4, 2021, 7:47am

You can also use the search here: Search results for 'node selection' - Storj Community Forum (official) or in the repo GitHub - storj/storj: Ongoing Storj v3 development. Decentralized cloud object storage that is affordable, easy to use, private, and secure.

Pentium100 · November 4, 2021, 5:01pm

So, what’s up with zero-sized uploads (they are all canceled or failed)?

2021-11-04T16:22:07.381Z        INFO    piecestore      upload canceled {"Piece ID": "Z4BPVDTD6A7U6NIDAQ25ZXYW7H4B73VBWBX4BEOZ7PF4PXTILZPA", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT", "Size": 0}

2021-11-04T14:02:49.246Z        ERROR   piecestore      upload failed   {"Piece ID": "6S36XZUTWL2RVZSICA7LRO2HXWZKZIIPT5LO5I2NYIRAIOMD4CIA", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT", "error": "unexpected EOF", "errorVerbose": "unexpected EOF\n\tstorj.io/common/rpc/rpcstatus.Error:82\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:342\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:220\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:104\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:60\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:97\n\tstorj.io/drpc/drpcctx.(*Tracker).track:52", "Size": 0}

A customer may want to upload a zero sized file, but aren’t files below a certain size supposed to be stored on the satellite itself? I think the smallest successful upload on my node is 512B (there are quite a few of those).

Alexey · November 4, 2021, 7:35pm

That’s interesting question. It should not be zero. The zero sized should be stored on the satellite with the metainformation altogether.

sean · November 4, 2021, 9:24pm

Thanks everyone for sharing your observations! I’m starting to look a bit deeper on these cancellations now. One initial observation is they appear to be coming from mostly the same satellite: 12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S (US1), so I will see if I can find anything out on the satellite side.

I’ve checked my own storage node logs and I’m also seeing frequent cancellations of the 163840 piece size, and the 0 size ones every so often, but not as frequent.

BrightSilence · November 4, 2021, 9:25pm

If I understand the function correctly that refers to the pieceWriter.Size which looks like it’s incremented with each chunk written. I think it means this is whatever has been written so far and not the total size of the piece upload. Which also explains why 0 can happen. But please correct me if I misread that.

sean · November 4, 2021, 9:38pm

That sounds correct to me, and this comment would justify that: storj/storagenode/pieces/readwrite.go at 2df41028a3ec9e5026c0b9691367fb6c82da8e99 · storj/storj · GitHub

BrightSilence · November 5, 2021, 12:48pm

Doesn’t it make sense then that cancellations align with the default chunk sizes being used for upload?

Any piece over 163840 bytes would at some point be that exact size, just because that’s what the previous chunk sizes add up to. The final piece size doesn’t align with exact chunk steps, so successfully finished pieces by definition have a lot more varied sizes, while anything cancelled before being finished would by definition align with standard chunks. I’d say that explains why cancellations happen at specific sizes, while successful pieces have more varied sizes.

Code_Breaker · November 5, 2021, 7:24pm

Sounds logical, and makes sense to me. Let me see what I can find on our end and if necessary escalate your observations. I’ll also follow up with Sean and see if he saw anything relative to US1, but I suspect that it isn’t isolated to a single satellite.