Piece Deleted During Download

Just a heads up that I have had a handful of pieces be deleted during the request to be downloaded.

|"2020-05-06T13:21:48.588Z|ERROR|piecestore|download failed|{""Piece ID"": ""QHYJ7GMN6WNPME7AEHNXANII2DZUIURDBPQ6NOBSAGVMLNDPQHFA"", ""Satellite ID"": ""118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW"", ""Action"": ""GET"", ""error"": ""file does not exist"", ""errorVerbose"": ""file does not exist\n\tstorj.io/common/pb/pbgrpc.init.0.func3:70\n\tstorj.io/common/rpc/rpcstatus.Wrap:77\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload:566\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Download:471\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:995\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:107\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:66\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:111\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:62\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:99\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51""}"|
|"2020-05-06T13:21:47.582Z|INFO|piecestore|deleted|{""Satellite ID"": ""118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW"", ""Piece ID"": ""QHYJ7GMN6WNPME7AEHNXANII2DZUIURDBPQ6NOBSAGVMLNDPQHFA""}"|
|"2020-05-06T13:21:46.923Z|INFO|piecestore|download started|{""Piece ID"": ""QHYJ7GMN6WNPME7AEHNXANII2DZUIURDBPQ6NOBSAGVMLNDPQHFA"", ""Satellite ID"": ""118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW"", ""Action"": ""GET""}"|

So far I have not found this to be happening with GET_AUDITs. Figured that I would point out this is happening just in case it does at some point impact an audit. If it is just testing a race condition or some other testing though, then just ignore this.

It is allowed, to try to download a piece after it is deleted. Just not succeed (maybe)

That would not be a great thing to have in place as this will result in more confusion for SNOs monitoring their logs. As well, if this type of thing can happen with regular downloads it may also happen with audits, resulting in an audit failure from missing a piece. Overall I think that is pretty counterproductive to have something like this occurring.

Also the order of operations here is important. The download request starts then the file is deleted while the download is occurring. IMO this just results in log pollution if anything, which again is not useful for proactive SNOs.

@Alexey Your insight please.

Imagine from the client end. You start downloading and get your data but one storagenode is slow and takes an hour to start downloading…
You the client might want to delete your data, afterall you already got your data from the fast nodes…and you are happy

Then in that case the storage node would have it’s download cancelled with a context cancelled INFO message. That is a scenario that already occurs and the storage node software accounts for that.

Throwing an ERROR message with “file does not exist” is pretty much one of the worst errors you can throw for a SNO as the cause is more likely to be that they have corrupted or deleted files instead of an out of order operation or improperly handled race condition.

but what if there is a second client downloading … :grin:

Could be that you had lost the race of uploading it and script just deleted it right away after finishing. I tried looking for those errors I couldnt find any on mine.

In a scenario where a file is deleted right after downloading is completed, the download canceled should be sent to the in scope storage nodes before the delete command. Hence an incorrect race condition.

I’m done going to go down this path of hypothetical scenarios. Figured I would provide feedback that SNOs will see ERROR level messages in their logs about files not existing in situations where the files don’t exist on purpose. If that’s not constructive, I’ll leave that up to Storj Labs to provide feedback on.

Most HTTP servers in their default configuration willl log 404 errors in the error log and not the general access log, even though it’s a client-side error (the client requested a document that does not exist).

Just figured I would make this post in the event that there is a better way to handle it for storage node in case less experienced SNOs ran into this same thing but I am realizing after this discussion that it is a non-issue and missing file errors can probably just be ignored.