The nodes which have updated to this version have started spitting out the following error from time to time. 2023-04-06T09:52:30.270Z ERROR piecestore upload failed {EU1.storj.io, "PUT", "error": "context canceled", "errorVerbose": "context canceled\n\tstorj.io/common/rpc/rpcstatus.Wrap:75\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload.func5:498\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:529\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:243\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:61\n\tstorj.io/common/experiment.(*Handler).HandleRPC:42\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:124\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:114\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35", s2319360}
Please ignore some alterations I made to make the logs more informative/compact. What I think is going on here is that this drpc closed error occurs when the piece is already committed but the error is a typical cancelation error, which now gets classified as failed upload.
What I don’t really know is whether in this case the node gets to keep the piece or it gets cleaned up in garbage collection. Either way, this seems like either a canceled or a successful transfer that wasn’t canceled quick enough to prevent it form being committed.
That’s just one example. But yeah, in the folder for us1. But it happens on other sats too. Not sure what exactly you were looking for, but they are in blobs.
Sure, but will the node get to keep the piece? I think depending on that it should either be logged as a success or cancelation. I think the current error means that it was intended to be a long tail cancelation, but the message just didn’t get to the node on time.
Ps. I didn’t expect it to be anything other than a logging/stats issue. But the PR was specifically to improve that and had this opposite side effect, so I thought I’d point it out.