GET - error - file does not exist, due to expired piece

Node Operators troubleshooting
1

When a piece has expired and removed from the node, it looks like the satellite are still trying to retrieve the price, but fails with “download failed - file does not exist”.

I’m guessing that this will continue to happen until GC has done its job, and stop impacting the download success rate…

th3van.dk

root@server030:/disk102/storj/logs/storagenode# cat  ./old/server023*.log server023*.log | grep "REPGG46VE2AUSVVPQGVEQNJPACQ6432SCLLMQ7LT5E2XS7ETNV6Q"
2021-03-01T07:21:55.239+0100    INFO    piecestore      upload started  {"Piece ID": "REPGG46VE2AUSVVPQGVEQNJPACQ6432SCLLMQ7LT5E2XS7ETNV6Q", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT", "Available Space": 10659411255680}
2021-03-01T07:21:55.372+0100    INFO    piecestore      uploaded        {"Piece ID": "REPGG46VE2AUSVVPQGVEQNJPACQ6432SCLLMQ7LT5E2XS7ETNV6Q", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT", "Size": 466688}
2021-03-01T07:22:02.780+0100    INFO    piecestore      download started        {"Piece ID": "REPGG46VE2AUSVVPQGVEQNJPACQ6432SCLLMQ7LT5E2XS7ETNV6Q", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET"}
2021-03-01T07:22:04.555+0100    INFO    piecestore      downloaded      {"Piece ID": "REPGG46VE2AUSVVPQGVEQNJPACQ6432SCLLMQ7LT5E2XS7ETNV6Q", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET", "Size": 466432}
2021-03-01T07:22:05.608+0100    INFO    piecestore      download started        {"Piece ID": "REPGG46VE2AUSVVPQGVEQNJPACQ6432SCLLMQ7LT5E2XS7ETNV6Q", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET"}
2021-03-01T07:22:07.133+0100    INFO    piecestore      downloaded      {"Piece ID": "REPGG46VE2AUSVVPQGVEQNJPACQ6432SCLLMQ7LT5E2XS7ETNV6Q", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET", "Size": 466432}
2021-03-01T07:22:11.008+0100    INFO    piecestore      download started        {"Piece ID": "REPGG46VE2AUSVVPQGVEQNJPACQ6432SCLLMQ7LT5E2XS7ETNV6Q", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET"}
2021-03-01T07:22:11.810+0100    INFO    piecestore      downloaded      {"Piece ID": "REPGG46VE2AUSVVPQGVEQNJPACQ6432SCLLMQ7LT5E2XS7ETNV6Q", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET", "Size": 466432}
2021-03-01T13:07:45.488+0100    INFO    piecestore      download started        {"Piece ID": "REPGG46VE2AUSVVPQGVEQNJPACQ6432SCLLMQ7LT5E2XS7ETNV6Q", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET"}
2021-03-01T13:07:46.970+0100    INFO    piecestore      downloaded      {"Piece ID": "REPGG46VE2AUSVVPQGVEQNJPACQ6432SCLLMQ7LT5E2XS7ETNV6Q", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET", "Size": 466432}
2021-03-01T13:08:24.319+0100    INFO    piecestore      download started        {"Piece ID": "REPGG46VE2AUSVVPQGVEQNJPACQ6432SCLLMQ7LT5E2XS7ETNV6Q", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET"}
2021-03-01T13:08:25.205+0100    INFO    piecestore      downloaded      {"Piece ID": "REPGG46VE2AUSVVPQGVEQNJPACQ6432SCLLMQ7LT5E2XS7ETNV6Q", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET", "Size": 466432}
2021-03-01T14:37:10.623+0100    INFO    piecestore      download started        {"Piece ID": "REPGG46VE2AUSVVPQGVEQNJPACQ6432SCLLMQ7LT5E2XS7ETNV6Q", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET"}
2021-03-01T14:37:12.195+0100    INFO    piecestore      downloaded      {"Piece ID": "REPGG46VE2AUSVVPQGVEQNJPACQ6432SCLLMQ7LT5E2XS7ETNV6Q", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET", "Size": 466432}
2021-03-01T14:37:26.084+0100    INFO    piecestore      download started        {"Piece ID": "REPGG46VE2AUSVVPQGVEQNJPACQ6432SCLLMQ7LT5E2XS7ETNV6Q", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET"}
2021-03-01T14:37:27.509+0100    INFO    piecestore      downloaded      {"Piece ID": "REPGG46VE2AUSVVPQGVEQNJPACQ6432SCLLMQ7LT5E2XS7ETNV6Q", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET", "Size": 466432}
2021-03-16T07:34:09.372+0100    DEBUG   pieces  deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "REPGG46VE2AUSVVPQGVEQNJPACQ6432SCLLMQ7LT5E2XS7ETNV6Q"}
2021-03-16T07:34:09.372+0100    INFO    collector       delete expired  {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "REPGG46VE2AUSVVPQGVEQNJPACQ6432SCLLMQ7LT5E2XS7ETNV6Q"}
2021-03-18T18:48:43.497+0100    INFO    piecestore      download started        {"Piece ID": "REPGG46VE2AUSVVPQGVEQNJPACQ6432SCLLMQ7LT5E2XS7ETNV6Q", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET"}
2021-03-18T18:48:43.497+0100    ERROR   piecestore      download failed {"Piece ID": "REPGG46VE2AUSVVPQGVEQNJPACQ6432SCLLMQ7LT5E2XS7ETNV6Q", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET", "error": "file does not exist", "errorVerbose": "file does not exist\n\tstorj.io/common/rpc/rpcstatus.Wrap:74\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:506\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:1033\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:29\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:111\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:62\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:99\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51"}
2021-03-18T18:48:46.872+0100    INFO    piecestore      download started        {"Piece ID": "REPGG46VE2AUSVVPQGVEQNJPACQ6432SCLLMQ7LT5E2XS7ETNV6Q", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET"}
2021-03-18T18:48:46.873+0100    ERROR   piecestore      download failed {"Piece ID": "REPGG46VE2AUSVVPQGVEQNJPACQ6432SCLLMQ7LT5E2XS7ETNV6Q", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET", "error": "file does not exist", "errorVerbose": "file does not exist\n\tstorj.io/common/rpc/rpcstatus.Wrap:74\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:506\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:1033\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:29\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:111\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:62\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:99\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51"}
2021-03-18T18:48:52.551+0100    INFO    piecestore      download started        {"Piece ID": "REPGG46VE2AUSVVPQGVEQNJPACQ6432SCLLMQ7LT5E2XS7ETNV6Q", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET"}
2021-03-18T18:48:52.552+0100    ERROR   piecestore      download failed {"Piece ID": "REPGG46VE2AUSVVPQGVEQNJPACQ6432SCLLMQ7LT5E2XS7ETNV6Q", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET", "error": "file does not exist", "errorVerbose": "file does not exist\n\tstorj.io/common/rpc/rpcstatus.Wrap:74\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:506\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:1033\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:29\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:111\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:62\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:99\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51"}
2021-03-18T18:48:53.902+0100    INFO    piecestore      download started        {"Piece ID": "REPGG46VE2AUSVVPQGVEQNJPACQ6432SCLLMQ7LT5E2XS7ETNV6Q", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET"}
2021-03-18T18:48:53.902+0100    ERROR   piecestore      download failed {"Piece ID": "REPGG46VE2AUSVVPQGVEQNJPACQ6432SCLLMQ7LT5E2XS7ETNV6Q", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET", "error": "file does not exist", "errorVerbose": "file does not exist\n\tstorj.io/common/rpc/rpcstatus.Wrap:74\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:506\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:1033\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:29\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:111\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:62\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:99\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51"}
2021-03-18T18:49:07.023+0100    INFO    piecestore      download started        {"Piece ID": "REPGG46VE2AUSVVPQGVEQNJPACQ6432SCLLMQ7LT5E2XS7ETNV6Q", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET"}
2021-03-18T18:49:07.024+0100    ERROR   piecestore      download failed {"Piece ID": "REPGG46VE2AUSVVPQGVEQNJPACQ6432SCLLMQ7LT5E2XS7ETNV6Q", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET", "error": "file does not exist", "errorVerbose": "file does not exist\n\tstorj.io/common/rpc/rpcstatus.Wrap:74\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:506\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:1033\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:29\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:111\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:62\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:99\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51"}
2021-03-18T18:49:07.125+0100    INFO    piecestore      download started        {"Piece ID": "REPGG46VE2AUSVVPQGVEQNJPACQ6432SCLLMQ7LT5E2XS7ETNV6Q", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET"}
2021-03-18T18:49:07.126+0100    ERROR   piecestore      download failed {"Piece ID": "REPGG46VE2AUSVVPQGVEQNJPACQ6432SCLLMQ7LT5E2XS7ETNV6Q", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET", "error": "file does not exist", "errorVerbose": "file does not exist\n\tstorj.io/common/rpc/rpcstatus.Wrap:74\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:506\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:1033\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:29\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:111\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:62\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:99\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51"}
2021-03-18T18:49:07.994+0100    INFO    piecestore      download started        {"Piece ID": "REPGG46VE2AUSVVPQGVEQNJPACQ6432SCLLMQ7LT5E2XS7ETNV6Q", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET"}
2021-03-18T18:49:07.994+0100    ERROR   piecestore      download failed {"Piece ID": "REPGG46VE2AUSVVPQGVEQNJPACQ6432SCLLMQ7LT5E2XS7ETNV6Q", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET", "error": "file does not exist", "errorVerbose": "file does not exist\n\tstorj.io/common/rpc/rpcstatus.Wrap:74\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:506\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:1033\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:29\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:111\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:62\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:99\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51"}
2021-03-18T18:49:14.870+0100    INFO    piecestore      download started        {"Piece ID": "REPGG46VE2AUSVVPQGVEQNJPACQ6432SCLLMQ7LT5E2XS7ETNV6Q", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET"}
2021-03-18T18:49:14.871+0100    ERROR   piecestore      download failed {"Piece ID": "REPGG46VE2AUSVVPQGVEQNJPACQ6432SCLLMQ7LT5E2XS7ETNV6Q", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET", "error": "file does not exist", "errorVerbose": "file does not exist\n\tstorj.io/common/rpc/rpcstatus.Wrap:74\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:506\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:1033\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:29\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:111\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:62\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:99\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51"}
2021-03-18T18:49:22.253+0100    INFO    piecestore      download started        {"Piece ID": "REPGG46VE2AUSVVPQGVEQNJPACQ6432SCLLMQ7LT5E2XS7ETNV6Q", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET"}
2021-03-18T18:49:22.253+0100    ERROR   piecestore      download failed {"Piece ID": "REPGG46VE2AUSVVPQGVEQNJPACQ6432SCLLMQ7LT5E2XS7ETNV6Q", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET", "error": "file does not exist", "errorVerbose": "file does not exist\n\tstorj.io/common/rpc/rpcstatus.Wrap:74\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:506\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:1033\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:29\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:111\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:62\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:99\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51"}
2021-03-18T18:49:32.086+0100    INFO    piecestore      download started        {"Piece ID": "REPGG46VE2AUSVVPQGVEQNJPACQ6432SCLLMQ7LT5E2XS7ETNV6Q", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET"}
2021-03-18T18:49:32.086+0100    ERROR   piecestore      download failed {"Piece ID": "REPGG46VE2AUSVVPQGVEQNJPACQ6432SCLLMQ7LT5E2XS7ETNV6Q", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET", "error": "file does not exist", "errorVerbose": "file does not exist\n\tstorj.io/common/rpc/rpcstatus.Wrap:74\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:506\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:1033\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:29\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:111\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:62\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:99\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51"}
2021-03-18T18:49:34.641+0100    INFO    piecestore      download started        {"Piece ID": "REPGG46VE2AUSVVPQGVEQNJPACQ6432SCLLMQ7LT5E2XS7ETNV6Q", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET"}
2021-03-18T18:49:34.641+0100    ERROR   piecestore      download failed {"Piece ID": "REPGG46VE2AUSVVPQGVEQNJPACQ6432SCLLMQ7LT5E2XS7ETNV6Q", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET", "error": "file does not exist", "errorVerbose": "file does not exist\n\tstorj.io/common/rpc/rpcstatus.Wrap:74\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:506\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:1033\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:29\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:111\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:62\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:99\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51"}
2021-03-18T18:49:36.552+0100    INFO    piecestore      download started        {"Piece ID": "REPGG46VE2AUSVVPQGVEQNJPACQ6432SCLLMQ7LT5E2XS7ETNV6Q", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET"}
2021-03-18T18:49:36.553+0100    ERROR   piecestore      download failed {"Piece ID": "REPGG46VE2AUSVVPQGVEQNJPACQ6432SCLLMQ7LT5E2XS7ETNV6Q", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET", "error": "file does not exist", "errorVerbose": "file does not exist\n\tstorj.io/common/rpc/rpcstatus.Wrap:74\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:506\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:1033\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:29\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:111\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:62\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:99\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51"}
2021-03-18T18:49:38.984+0100    INFO    piecestore      download started        {"Piece ID": "REPGG46VE2AUSVVPQGVEQNJPACQ6432SCLLMQ7LT5E2XS7ETNV6Q", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET"}
2021-03-18T18:49:38.985+0100    ERROR   piecestore      download failed {"Piece ID": "REPGG46VE2AUSVVPQGVEQNJPACQ6432SCLLMQ7LT5E2XS7ETNV6Q", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET", "error": "file does not exist", "errorVerbose": "file does not exist\n\tstorj.io/common/rpc/rpcstatus.Wrap:74\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:506\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:1033\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:29\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:111\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:62\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:99\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51"}
2021-03-18T18:49:47.973+0100    INFO    piecestore      download started        {"Piece ID": "REPGG46VE2AUSVVPQGVEQNJPACQ6432SCLLMQ7LT5E2XS7ETNV6Q", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET"}
2021-03-18T18:49:47.973+0100    ERROR   piecestore      download failed {"Piece ID": "REPGG46VE2AUSVVPQGVEQNJPACQ6432SCLLMQ7LT5E2XS7ETNV6Q", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET", "error": "file does not exist", "errorVerbose": "file does not exist\n\tstorj.io/common/rpc/rpcstatus.Wrap:74\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:506\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:1033\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:29\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:111\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:62\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:99\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51"}
2021-03-18T18:49:50.459+0100    INFO    piecestore      download started        {"Piece ID": "REPGG46VE2AUSVVPQGVEQNJPACQ6432SCLLMQ7LT5E2XS7ETNV6Q", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET"}
2021-03-18T18:49:50.460+0100    ERROR   piecestore      download failed {"Piece ID": "REPGG46VE2AUSVVPQGVEQNJPACQ6432SCLLMQ7LT5E2XS7ETNV6Q", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET", "error": "file does not exist", "errorVerbose": "file does not exist\n\tstorj.io/common/rpc/rpcstatus.Wrap:74\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:506\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:1033\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:29\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:111\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:62\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:99\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51"}
2021-03-18T18:49:51.844+0100    INFO    piecestore      download started        {"Piece ID": "REPGG46VE2AUSVVPQGVEQNJPACQ6432SCLLMQ7LT5E2XS7ETNV6Q", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET"}
2021-03-18T18:49:51.845+0100    ERROR   piecestore      download failed {"Piece ID": "REPGG46VE2AUSVVPQGVEQNJPACQ6432SCLLMQ7LT5E2XS7ETNV6Q", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET", "error": "file does not exist", "errorVerbose": "file does not exist\n\tstorj.io/common/rpc/rpcstatus.Wrap:74\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:506\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:1033\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:29\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:111\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:62\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:99\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51"}
2021-03-18T18:49:52.587+0100    INFO    piecestore      download started        {"Piece ID": "REPGG46VE2AUSVVPQGVEQNJPACQ6432SCLLMQ7LT5E2XS7ETNV6Q", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET"}
2021-03-18T18:49:52.587+0100    ERROR   piecestore      download failed {"Piece ID": "REPGG46VE2AUSVVPQGVEQNJPACQ6432SCLLMQ7LT5E2XS7ETNV6Q", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET", "error": "file does not exist", "errorVerbose": "file does not exist\n\tstorj.io/common/rpc/rpcstatus.Wrap:74\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:506\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:1033\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:29\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:111\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:62\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:99\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51"}
2021-03-18T18:50:03.568+0100    INFO    piecestore      download started        {"Piece ID": "REPGG46VE2AUSVVPQGVEQNJPACQ6432SCLLMQ7LT5E2XS7ETNV6Q", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET"}
2021-03-18T18:50:03.568+0100    ERROR   piecestore      download failed {"Piece ID": "REPGG46VE2AUSVVPQGVEQNJPACQ6432SCLLMQ7LT5E2XS7ETNV6Q", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET", "error": "file does not exist", "errorVerbose": "file does not exist\n\tstorj.io/common/rpc/rpcstatus.Wrap:74\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:506\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:1033\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:29\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:111\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:62\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:99\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51"}
2021-03-18T18:50:06.435+0100    INFO    piecestore      download started        {"Piece ID": "REPGG46VE2AUSVVPQGVEQNJPACQ6432SCLLMQ7LT5E2XS7ETNV6Q", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET"}
2021-03-18T18:50:06.436+0100    ERROR   piecestore      download failed {"Piece ID": "REPGG46VE2AUSVVPQGVEQNJPACQ6432SCLLMQ7LT5E2XS7ETNV6Q", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET", "error": "file does not exist", "errorVerbose": "file does not exist\n\tstorj.io/common/rpc/rpcstatus.Wrap:74\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:506\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:1033\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:29\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:111\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:62\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:99\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51"}
2021-03-18T18:50:13.705+0100    INFO    piecestore      download started        {"Piece ID": "REPGG46VE2AUSVVPQGVEQNJPACQ6432SCLLMQ7LT5E2XS7ETNV6Q", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET"}
2021-03-18T18:50:13.705+0100    ERROR   piecestore      download failed {"Piece ID": "REPGG46VE2AUSVVPQGVEQNJPACQ6432SCLLMQ7LT5E2XS7ETNV6Q", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET", "error": "file does not exist", "errorVerbose": "file does not exist\n\tstorj.io/common/rpc/rpcstatus.Wrap:74\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:506\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:1033\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:29\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:111\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:62\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:99\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51"}
2021-03-18T18:50:16.081+0100    INFO    piecestore      download started        {"Piece ID": "REPGG46VE2AUSVVPQGVEQNJPACQ6432SCLLMQ7LT5E2XS7ETNV6Q", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET"}
2021-03-18T18:50:16.081+0100    ERROR   piecestore      download failed {"Piece ID": "REPGG46VE2AUSVVPQGVEQNJPACQ6432SCLLMQ7LT5E2XS7ETNV6Q", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET", "error": "file does not exist", "errorVerbose": "file does not exist\n\tstorj.io/common/rpc/rpcstatus.Wrap:74\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:506\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:1033\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:29\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:111\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:62\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:99\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51"}
2021-03-18T18:50:18.036+0100    INFO    piecestore      download started        {"Piece ID": "REPGG46VE2AUSVVPQGVEQNJPACQ6432SCLLMQ7LT5E2XS7ETNV6Q", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET"}
2021-03-18T18:50:18.037+0100    ERROR   piecestore      download failed {"Piece ID": "REPGG46VE2AUSVVPQGVEQNJPACQ6432SCLLMQ7LT5E2XS7ETNV6Q", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET", "error": "file does not exist", "errorVerbose": "file does not exist\n\tstorj.io/common/rpc/rpcstatus.Wrap:74\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:506\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:1033\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:29\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:111\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:62\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:99\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51"}
1 Like
2

Hi Th3van, thanks for question.
This is known issue. This happened because we disabled expire delete on Europe west satellite ( we did it because after the migration to new multipart upload code the SQL query has thrown errors to us. We will fix the query and re-enable the expire delete job).
In the meantime storage nodes might see this on theirs end. But don’t worry, there should be no downside. (No penalty)

7 Likes