Failed Audits (v0.21.1)

I see a issue with audit process:

Summary
root@docker03:~/odmin/storj_success_rate# docker logs storagenode 2>&1 | grep GET_AUDIT | grep 'download failed' | awk '{print $8 $10}' | sort | uniq -c | sort -g -r
      1 "GTVC7W57R24Z2XDD6YVGKLFH4WZ5VBOX4FZY65KGGBGYEZAHLOMQ","118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW",
      1 "6OT4Q4BPUMIF4BPM6QQCKX6MWI42OLNYRQW64NEYSCTQ6S6IG5NA","118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW",
      1 "6CXGLKHGKI5ZJCC5WJLGXKQXEGVRSDA65XZ7QVKDDCDEJB6ROE6Q","118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW",
      1 "5DDNLDXBH7KU4ETBSY7AW6PPTCKYZAJJ6IKMCZJ4TOU3GUUKLBYQ","118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW",
root@docker03:~/odmin/storj_success_rate# docker logs storagenode 2>&1 | grep "5DDNLDXBH7KU4ETBSY7AW6PPTCKYZAJJ6IKMCZJ4TOU3GUUKLBYQ"
2019-09-21T14:08:39.049Z        INFO    piecestore      deleted {"Piece ID": "5DDNLDXBH7KU4ETBSY7AW6PPTCKYZAJJ6IKMCZJ4TOU3GUUKLBYQ"}
2019-09-22T18:35:01.936Z        INFO    piecestore      download started        {"Piece ID": "5DDNLDXBH7KU4ETBSY7AW6PPTCKYZAJJ6IKMCZJ4TOU3GUUKLBYQ", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_AUDIT"}
2019-09-22T18:35:01.940Z        INFO    piecestore      download failed {"Piece ID": "5DDNLDXBH7KU4ETBSY7AW6PPTCKYZAJJ6IKMCZJ4TOU3GUUKLBYQ", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_AUDIT", "error": "rpc error: code = NotFound desc = file does not exist"}
root@docker03:~/odmin/storj_success_rate#
root@docker03:~/odmin/storj_success_rate#
root@docker03:~/odmin/storj_success_rate# docker logs storagenode 2>&1 | grep "6CXGLKHGKI5ZJCC5WJLGXKQXEGVRSDA65XZ7QVKDDCDEJB6ROE6Q"
2019-09-21T18:32:24.230Z        INFO    piecestore      deleted {"Piece ID": "6CXGLKHGKI5ZJCC5WJLGXKQXEGVRSDA65XZ7QVKDDCDEJB6ROE6Q"}
2019-09-22T08:06:51.584Z        INFO    piecestore      download started        {"Piece ID": "6CXGLKHGKI5ZJCC5WJLGXKQXEGVRSDA65XZ7QVKDDCDEJB6ROE6Q", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_AUDIT"}
2019-09-22T08:06:51.589Z        INFO    piecestore      download failed {"Piece ID": "6CXGLKHGKI5ZJCC5WJLGXKQXEGVRSDA65XZ7QVKDDCDEJB6ROE6Q", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_AUDIT", "error": "rpc error: code = NotFound desc = file does not exist"}
root@docker03:~/odmin/storj_success_rate#
root@docker03:~/odmin/storj_success_rate#
root@docker03:~/odmin/storj_success_rate# docker logs storagenode 2>&1 | grep "6OT4Q4BPUMIF4BPM6QQCKX6MWI42OLNYRQW64NEYSCTQ6S6IG5NA"
2019-09-21T09:52:27.181Z        INFO    piecestore      deleted {"Piece ID": "6OT4Q4BPUMIF4BPM6QQCKX6MWI42OLNYRQW64NEYSCTQ6S6IG5NA"}
2019-09-21T23:19:25.736Z        INFO    piecestore      download started        {"Piece ID": "6OT4Q4BPUMIF4BPM6QQCKX6MWI42OLNYRQW64NEYSCTQ6S6IG5NA", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_AUDIT"}
2019-09-21T23:19:25.740Z        INFO    piecestore      download failed {"Piece ID": "6OT4Q4BPUMIF4BPM6QQCKX6MWI42OLNYRQW64NEYSCTQ6S6IG5NA", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_AUDIT", "error": "rpc error: code = NotFound desc = file does not exist"}
root@docker03:~/odmin/storj_success_rate#
root@docker03:~/odmin/storj_success_rate#
root@docker03:~/odmin/storj_success_rate# docker logs storagenode 2>&1 | grep "GTVC7W57R24Z2XDD6YVGKLFH4WZ5VBOX4FZY65KGGBGYEZAHLOMQ"
2019-09-21T11:05:01.260Z        INFO    piecestore      deleted {"Piece ID": "GTVC7W57R24Z2XDD6YVGKLFH4WZ5VBOX4FZY65KGGBGYEZAHLOMQ"}
2019-09-21T18:14:49.958Z        INFO    piecestore      download started        {"Piece ID": "GTVC7W57R24Z2XDD6YVGKLFH4WZ5VBOX4FZY65KGGBGYEZAHLOMQ", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_AUDIT"}
2019-09-21T18:14:49.959Z        INFO    piecestore      download failed {"Piece ID": "GTVC7W57R24Z2XDD6YVGKLFH4WZ5VBOX4FZY65KGGBGYEZAHLOMQ", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_AUDIT", "error": "rpc error: code = NotFound desc = file does not exist"}

File was deleted, but satellite ask for download deleted file, this is definitely bug.

1 Like

Check this thread.

Our devs are already aware of this issue, they found the cause and are now implementing a fix.

2 Likes

I have 4 occurrence of these in my logs since my last RM.

2019-09-21T11:05:01.255Z INFO piecestore deleted {“Piece ID”: “AFY3WUEG4QHASE74EXX53Q6XPX3EG4XAIKVEF6LW27OANSXGDCKQ”}
2019-09-21T18:14:50.233Z INFO piecestore download started {“Piece ID”: “AFY3WUEG4QHASE74EXX53Q6XPX3EG4XAIKVEF6LW27OANSXGDCKQ”, “SatelliteID”: “118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW”, “Action”: “GET_AUDIT”}
2019-09-21T18:14:50.234Z INFO piecestore download failed {“Piece ID”: “AFY3WUEG4QHASE74EXX53Q6XPX3EG4XAIKVEF6LW27OANSXGDCKQ”, “SatelliteID”: “118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW”, “Action”: “GET_AUDIT”, “error”: “rpc error: code = NotFound desc = file does not exist”}

I have been watching the web ui since it became active and it has always reported 100% audits until this morning and now its reporting 99.9% and it is concerning to me I don’t like failed audits, I guess the next ? is does the web ui report historical audits for the life of the node or is it based on the logs which now is only 13 hours old since the last update 0.21.3, this node has been online since July 21st that being said is close to 40k successful audits so that tenth of a percent could be a large number so what is next for the node.

It’s a known issue right now and I’m not sure the recent update already included the fix. The reported numbers are for the lifetime of your node as accounted by the satellite. It’s not based on your logs.
Your node will likely be fine, I wouldn’t worry about it too much.

1 Like

thanks I will monitor closely

I’m sure this is just another form of the same issue, but just to be sure I’ll post it.
I’m now also seeing some pieces trying to be repaired after being deleted a while ago.

2019-07-26T05:00:11.512Z  INFO    piecestore      upload started  {"Piece ID": "DANWA6VOOIOXHMBGIBGTZBOSYDIABCRYOQ4QGYP3SFGYWJ5O7QAA", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "PUT"}
2019-07-26T05:00:16.335Z  INFO    piecestore      uploaded        {"Piece ID": "DANWA6VOOIOXHMBGIBGTZBOSYDIABCRYOQ4QGYP3SFGYWJ5O7QAA", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "PUT"}
2019-09-21T22:35:40.689Z  INFO    piecestore      deleted {"Piece ID": "DANWA6VOOIOXHMBGIBGTZBOSYDIABCRYOQ4QGYP3SFGYWJ5O7QAA"}
2019-10-11T16:57:05.867Z        INFO    piecestore      download started        {"Piece ID": "DANWA6VOOIOXHMBGIBGTZBOSYDIABCRYOQ4QGYP3SFGYWJ5O7QAA","SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR"}
2019-10-11T16:57:05.867Z        INFO    piecestore      download failed {"Piece ID": "DANWA6VOOIOXHMBGIBGTZBOSYDIABCRYOQ4QGYP3SFGYWJ5O7QAA", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR", "error": "rpc error: code = NotFound desc = file does not exist"}
2019-10-12T00:59:30.321Z        INFO    piecestore      download started        {"Piece ID": "DANWA6VOOIOXHMBGIBGTZBOSYDIABCRYOQ4QGYP3SFGYWJ5O7QAA","SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR"}
2019-10-12T00:59:30.322Z        INFO    piecestore      download failed {"Piece ID": "DANWA6VOOIOXHMBGIBGTZBOSYDIABCRYOQ4QGYP3SFGYWJ5O7QAA", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR", "error": "rpc error: code = NotFound desc = file does not exist"}
2019-10-12T02:14:22.848Z        INFO    piecestore      download started        {"Piece ID": "DANWA6VOOIOXHMBGIBGTZBOSYDIABCRYOQ4QGYP3SFGYWJ5O7QAA","SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR"}
2019-10-12T02:14:22.849Z        INFO    piecestore      download failed {"Piece ID": "DANWA6VOOIOXHMBGIBGTZBOSYDIABCRYOQ4QGYP3SFGYWJ5O7QAA", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR", "error": "rpc error: code = NotFound desc = file does not exist"}
2019-10-12T02:46:09.099Z        INFO    piecestore      download started        {"Piece ID": "DANWA6VOOIOXHMBGIBGTZBOSYDIABCRYOQ4QGYP3SFGYWJ5O7QAA","SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR"}
2019-10-12T02:46:09.099Z        INFO    piecestore      download failed {"Piece ID": "DANWA6VOOIOXHMBGIBGTZBOSYDIABCRYOQ4QGYP3SFGYWJ5O7QAA", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR", "error": "rpc error: code = NotFound desc = file does not exist"}
2019-10-12T02:58:30.826Z        INFO    piecestore      download started        {"Piece ID": "DANWA6VOOIOXHMBGIBGTZBOSYDIABCRYOQ4QGYP3SFGYWJ5O7QAA","SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR"}
2019-10-12T02:58:30.826Z        INFO    piecestore      download failed {"Piece ID": "DANWA6VOOIOXHMBGIBGTZBOSYDIABCRYOQ4QGYP3SFGYWJ5O7QAA", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR", "error": "rpc error: code = NotFound desc = file does not exist"}
2019-10-12T03:04:02.526Z        INFO    piecestore      download started        {"Piece ID": "DANWA6VOOIOXHMBGIBGTZBOSYDIABCRYOQ4QGYP3SFGYWJ5O7QAA","SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR"}
2019-10-12T03:04:02.527Z        INFO    piecestore      download failed {"Piece ID": "DANWA6VOOIOXHMBGIBGTZBOSYDIABCRYOQ4QGYP3SFGYWJ5O7QAA", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR", "error": "rpc error: code = NotFound desc = file does not exist"}
2019-10-12T03:08:35.778Z        INFO    piecestore      download started        {"Piece ID": "DANWA6VOOIOXHMBGIBGTZBOSYDIABCRYOQ4QGYP3SFGYWJ5O7QAA","SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR"}
2019-10-12T03:08:35.779Z        INFO    piecestore      download failed {"Piece ID": "DANWA6VOOIOXHMBGIBGTZBOSYDIABCRYOQ4QGYP3SFGYWJ5O7QAA", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR", "error": "rpc error: code = NotFound desc = file does not exist"}
2019-10-12T03:14:55.284Z        INFO    piecestore      download started        {"Piece ID": "DANWA6VOOIOXHMBGIBGTZBOSYDIABCRYOQ4QGYP3SFGYWJ5O7QAA","SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR"}
2019-10-12T03:14:55.285Z        INFO    piecestore      download failed {"Piece ID": "DANWA6VOOIOXHMBGIBGTZBOSYDIABCRYOQ4QGYP3SFGYWJ5O7QAA", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR", "error": "rpc error: code = NotFound desc = file does not exist"}
2019-10-12T03:23:52.757Z        INFO    piecestore      download started        {"Piece ID": "DANWA6VOOIOXHMBGIBGTZBOSYDIABCRYOQ4QGYP3SFGYWJ5O7QAA","SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR"}
2019-10-12T03:23:52.757Z        INFO    piecestore      download failed {"Piece ID": "DANWA6VOOIOXHMBGIBGTZBOSYDIABCRYOQ4QGYP3SFGYWJ5O7QAA", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR", "error": "rpc error: code = NotFound desc = file does not exist"}
2019-10-12T03:29:29.821Z        INFO    piecestore      download started        {"Piece ID": "DANWA6VOOIOXHMBGIBGTZBOSYDIABCRYOQ4QGYP3SFGYWJ5O7QAA","SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR"}
2019-10-12T03:29:29.821Z        INFO    piecestore      download failed {"Piece ID": "DANWA6VOOIOXHMBGIBGTZBOSYDIABCRYOQ4QGYP3SFGYWJ5O7QAA", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR", "error": "rpc error: code = NotFound desc = file does not exist"}
2019-10-12T03:37:26.891Z        INFO    piecestore      download started        {"Piece ID": "DANWA6VOOIOXHMBGIBGTZBOSYDIABCRYOQ4QGYP3SFGYWJ5O7QAA","SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR"}
2019-10-12T03:37:26.891Z        INFO    piecestore      download failed {"Piece ID": "DANWA6VOOIOXHMBGIBGTZBOSYDIABCRYOQ4QGYP3SFGYWJ5O7QAA", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR", "error": "rpc error: code = NotFound desc = file does not exist"}
2019-10-12T04:15:42.446Z        INFO    piecestore      download started        {"Piece ID": "DANWA6VOOIOXHMBGIBGTZBOSYDIABCRYOQ4QGYP3SFGYWJ5O7QAA","SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR"}
2019-10-12T04:15:42.447Z        INFO    piecestore      download failed {"Piece ID": "DANWA6VOOIOXHMBGIBGTZBOSYDIABCRYOQ4QGYP3SFGYWJ5O7QAA", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR", "error": "rpc error: code = NotFound desc = file does not exist"}
2019-10-12T05:15:42.567Z        INFO    piecestore      download started        {"Piece ID": "DANWA6VOOIOXHMBGIBGTZBOSYDIABCRYOQ4QGYP3SFGYWJ5O7QAA","SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR"}
2019-10-12T05:15:42.567Z        INFO    piecestore      download failed {"Piece ID": "DANWA6VOOIOXHMBGIBGTZBOSYDIABCRYOQ4QGYP3SFGYWJ5O7QAA", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR", "error": "rpc error: code = NotFound desc = file does not exist"}
2019-10-12T06:15:42.985Z        INFO    piecestore      download started        {"Piece ID": "DANWA6VOOIOXHMBGIBGTZBOSYDIABCRYOQ4QGYP3SFGYWJ5O7QAA","SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR"}
2019-10-12T06:15:42.986Z        INFO    piecestore      download failed {"Piece ID": "DANWA6VOOIOXHMBGIBGTZBOSYDIABCRYOQ4QGYP3SFGYWJ5O7QAA", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR", "error": "rpc error: code = NotFound desc = file does not exist"}
2019-10-13T11:33:16.566Z        INFO    piecestore      download started        {"Piece ID": "DANWA6VOOIOXHMBGIBGTZBOSYDIABCRYOQ4QGYP3SFGYWJ5O7QAA","SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR"}
2019-10-13T11:33:16.567Z        INFO    piecestore      download failed {"Piece ID": "DANWA6VOOIOXHMBGIBGTZBOSYDIABCRYOQ4QGYP3SFGYWJ5O7QAA", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR", "error": "rpc error: code = NotFound desc = file does not exist"}
2019-10-13T12:11:36.667Z        INFO    piecestore      download started        {"Piece ID": "DANWA6VOOIOXHMBGIBGTZBOSYDIABCRYOQ4QGYP3SFGYWJ5O7QAA","SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR"}
2019-10-13T12:11:36.667Z        INFO    piecestore      download failed {"Piece ID": "DANWA6VOOIOXHMBGIBGTZBOSYDIABCRYOQ4QGYP3SFGYWJ5O7QAA", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR", "error": "rpc error: code = NotFound desc = file does not exist"}
2019-10-13T12:16:58.345Z        INFO    piecestore      download started        {"Piece ID": "DANWA6VOOIOXHMBGIBGTZBOSYDIABCRYOQ4QGYP3SFGYWJ5O7QAA","SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR"}
2019-10-13T12:16:58.346Z        INFO    piecestore      download failed {"Piece ID": "DANWA6VOOIOXHMBGIBGTZBOSYDIABCRYOQ4QGYP3SFGYWJ5O7QAA", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR", "error": "rpc error: code = NotFound desc = file does not exist"}
2019-10-13T13:05:27.014Z        INFO    piecestore      download started        {"Piece ID": "DANWA6VOOIOXHMBGIBGTZBOSYDIABCRYOQ4QGYP3SFGYWJ5O7QAA","SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR"}
2019-10-13T13:05:27.015Z        INFO    piecestore      download failed {"Piece ID": "DANWA6VOOIOXHMBGIBGTZBOSYDIABCRYOQ4QGYP3SFGYWJ5O7QAA", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR", "error": "rpc error: code = NotFound desc = file does not exist"}