Storj lost one piece? Repeated "file does not exist" in GET_REPAIR

Started to see these errors on a regular basis in my GET_REPAIR logs (I have set up email alerting on REPAIR traffic) about the B2O25KUSH4TEBUWXNFYETS7YGKQB4H42EUHHFJP2GYHBR4QWM7IQ Piece ID. The first one was today at 2019-11-08T02:55:53.330Z (actually I see one from one month ago at 2019-10-07T04:17:38.778Z about piece IYZ5S2UQRU7ULH7MDWTA33QOPSDKZ5UGD62BUNIGJ7GSZZZKYMGQ but that was only one occurrence).

I don’t think my storage is corrupted on a physical level, the host’s uptime is 64+ days and I didn’t have any ungraceful system shutdowns since I installed storj node.

Here’s a docker logs storagenode 2>&1 | grep "file does not exist":

2019-11-08T02:55:53.330Z        ESC[34mINFOESC[0m       piecestore      download failed {"Piece ID": "B2O25KUSH4TEBUWXNFYETS7YGKQB4H42EUHHFJP2GYHBR4QWM7IQ", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR", "error": "file does not exist"}
2019-11-08T02:55:53.330Z        ESC[31mERRORESC[0m      server  gRPC stream error response      {"error": "file does not exist"}
2019-11-08T03:09:40.744Z        ESC[34mINFOESC[0m       piecestore      download failed {"Piece ID": "B2O25KUSH4TEBUWXNFYETS7YGKQB4H42EUHHFJP2GYHBR4QWM7IQ", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR", "error": "file does not exist"}
2019-11-08T03:09:40.745Z        ESC[31mERRORESC[0m      server  gRPC stream error response      {"error": "file does not exist"}
... skipped a few more attempts ...
2019-11-08T10:22:59.033Z        ESC[34mINFOESC[0m       piecestore      download failed {"Piece ID": "B2O25KUSH4TEBUWXNFYETS7YGKQB4H42EUHHFJP2GYHBR4QWM7IQ", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR", "error": "file does not exist"}
2019-11-08T10:22:59.034Z        ESC[31mERRORESC[0m      server  gRPC stream error response      {"error": "file does not exist"}

Is there anything I can do to help you address this issue?

We currently have a handfull of halve deleted segments. The uplink wanted to delete them, was sending a delete command to your storage node but at the end was unable to delete the pointer from the satellite. We identified a few edge cases like that and are working on solution.

1 Like

it is still happening again and again

2019-08-11T16:52:38.730Z	INFO	piecestore	upload started	{"Piece ID": "O5IAFDLO242VZRCQGKDMZAHLT6KR2QLOWNA43DHKZKDU7UUN6O2A", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "PUT"}
2019-08-11T16:52:52.338Z	INFO	piecestore	uploaded	{"Piece ID": "O5IAFDLO242VZRCQGKDMZAHLT6KR2QLOWNA43DHKZKDU7UUN6O2A", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "PUT"}
2019-09-21T11:01:28.520Z	INFO	piecestore	deleted	{"Piece ID": "O5IAFDLO242VZRCQGKDMZAHLT6KR2QLOWNA43DHKZKDU7UUN6O2A"}
2019-11-16T23:17:21.282Z	INFO	piecestore	download started	{"Piece ID": "O5IAFDLO242VZRCQGKDMZAHLT6KR2QLOWNA43DHKZKDU7UUN6O2A", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR"}
2019-11-16T23:17:21.313Z	INFO	piecestore	download failed	{"Piece ID": "O5IAFDLO242VZRCQGKDMZAHLT6KR2QLOWNA43DHKZKDU7UUN6O2A", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR", "error": "file does not exist"}
2019-11-16T23:27:52.369Z	INFO	piecestore	download started	{"Piece ID": "O5IAFDLO242VZRCQGKDMZAHLT6KR2QLOWNA43DHKZKDU7UUN6O2A", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR"}
2019-11-16T23:27:52.370Z	INFO	piecestore	download failed	{"Piece ID": "O5IAFDLO242VZRCQGKDMZAHLT6KR2QLOWNA43DHKZKDU7UUN6O2A", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR", "error": "file does not exist"}
2019-11-16T23:49:49.715Z	INFO	piecestore	download started	{"Piece ID": "O5IAFDLO242VZRCQGKDMZAHLT6KR2QLOWNA43DHKZKDU7UUN6O2A", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR"}
2019-11-16T23:49:49.845Z	INFO	piecestore	download failed	{"Piece ID": "O5IAFDLO242VZRCQGKDMZAHLT6KR2QLOWNA43DHKZKDU7UUN6O2A", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR", "error": "file does not exist"}
2019-11-17T00:17:17.201Z	INFO	piecestore	download started	{"Piece ID": "O5IAFDLO242VZRCQGKDMZAHLT6KR2QLOWNA43DHKZKDU7UUN6O2A", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR"}
2019-11-17T00:17:17.201Z	INFO	piecestore	download failed	{"Piece ID": "O5IAFDLO242VZRCQGKDMZAHLT6KR2QLOWNA43DHKZKDU7UUN6O2A", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR", "error": "file does not exist"}
2019-11-17T01:17:12.969Z	INFO	piecestore	download started	{"Piece ID": "O5IAFDLO242VZRCQGKDMZAHLT6KR2QLOWNA43DHKZKDU7UUN6O2A", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR"}
2019-11-17T01:17:12.995Z	INFO	piecestore	download failed	{"Piece ID": "O5IAFDLO242VZRCQGKDMZAHLT6KR2QLOWNA43DHKZKDU7UUN6O2A", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR", "error": "file does not exist"}
2019-11-17T02:17:13.182Z	INFO	piecestore	download started	{"Piece ID": "O5IAFDLO242VZRCQGKDMZAHLT6KR2QLOWNA43DHKZKDU7UUN6O2A", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR"}
2019-11-17T02:17:13.182Z	INFO	piecestore	download failed	{"Piece ID": "O5IAFDLO242VZRCQGKDMZAHLT6KR2QLOWNA43DHKZKDU7UUN6O2A", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR", "error": "file does not exist"}
2019-11-17T03:17:12.593Z	INFO	piecestore	download started	{"Piece ID": "O5IAFDLO242VZRCQGKDMZAHLT6KR2QLOWNA43DHKZKDU7UUN6O2A", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR"}
2019-11-17T03:17:12.594Z	INFO	piecestore	download failed	{"Piece ID": "O5IAFDLO242VZRCQGKDMZAHLT6KR2QLOWNA43DHKZKDU7UUN6O2A", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR", "error": "file does not exist"}
2019-11-17T03:27:07.262Z	INFO	piecestore	download started	{"Piece ID": "O5IAFDLO242VZRCQGKDMZAHLT6KR2QLOWNA43DHKZKDU7UUN6O2A", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR"}
2019-11-17T03:27:07.263Z	INFO	piecestore	download failed	{"Piece ID": "O5IAFDLO242VZRCQGKDMZAHLT6KR2QLOWNA43DHKZKDU7UUN6O2A", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR", "error": "file does not exist"}
2019-11-17T04:17:12.012Z	INFO	piecestore	download started	{"Piece ID": "O5IAFDLO242VZRCQGKDMZAHLT6KR2QLOWNA43DHKZKDU7UUN6O2A", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR"}
2019-11-17T04:17:12.012Z	INFO	piecestore	download failed	{"Piece ID": "O5IAFDLO242VZRCQGKDMZAHLT6KR2QLOWNA43DHKZKDU7UUN6O2A", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR", "error": "file does not exist"}
2019-11-17T04:27:07.344Z	INFO	piecestore	download started	{"Piece ID": "O5IAFDLO242VZRCQGKDMZAHLT6KR2QLOWNA43DHKZKDU7UUN6O2A", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR"}
2019-11-17T04:27:07.377Z	INFO	piecestore	download failed	{"Piece ID": "O5IAFDLO242VZRCQGKDMZAHLT6KR2QLOWNA43DHKZKDU7UUN6O2A", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR", "error": "file does not exist"}
2019-11-17T05:17:16.694Z	INFO	piecestore	download started	{"Piece ID": "O5IAFDLO242VZRCQGKDMZAHLT6KR2QLOWNA43DHKZKDU7UUN6O2A", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR"}
2019-11-17T05:17:16.694Z	INFO	piecestore	download failed	{"Piece ID": "O5IAFDLO242VZRCQGKDMZAHLT6KR2QLOWNA43DHKZKDU7UUN6O2A", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR", "error": "file does not exist"}
2019-11-17T05:20:05.866Z	INFO	piecestore	download started	{"Piece ID": "O5IAFDLO242VZRCQGKDMZAHLT6KR2QLOWNA43DHKZKDU7UUN6O2A", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR"}
2019-11-17T05:20:05.866Z	INFO	piecestore	download failed	{"Piece ID": "O5IAFDLO242VZRCQGKDMZAHLT6KR2QLOWNA43DHKZKDU7UUN6O2A", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR", "error": "file does not exist"}
2019-11-17T05:41:11.464Z	INFO	piecestore	download started	{"Piece ID": "O5IAFDLO242VZRCQGKDMZAHLT6KR2QLOWNA43DHKZKDU7UUN6O2A", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR"}
2019-11-17T05:41:11.465Z	INFO	piecestore	download failed	{"Piece ID": "O5IAFDLO242VZRCQGKDMZAHLT6KR2QLOWNA43DHKZKDU7UUN6O2A", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR", "error": "file does not exist"}
2019-11-17T06:17:10.739Z	INFO	piecestore	download started	{"Piece ID": "O5IAFDLO242VZRCQGKDMZAHLT6KR2QLOWNA43DHKZKDU7UUN6O2A", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR"}
2019-11-17T06:17:10.739Z	INFO	piecestore	download failed	{"Piece ID": "O5IAFDLO242VZRCQGKDMZAHLT6KR2QLOWNA43DHKZKDU7UUN6O2A", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR", "error": "file does not exist"}
2019-11-17T07:17:21.494Z	INFO	piecestore	download started	{"Piece ID": "O5IAFDLO242VZRCQGKDMZAHLT6KR2QLOWNA43DHKZKDU7UUN6O2A", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR"}
2019-11-17T07:17:21.495Z	INFO	piecestore	download failed	{"Piece ID": "O5IAFDLO242VZRCQGKDMZAHLT6KR2QLOWNA43DHKZKDU7UUN6O2A", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR", "error": "file does not exist"}
2019-11-17T08:17:12.689Z	INFO	piecestore	download started	{"Piece ID": "O5IAFDLO242VZRCQGKDMZAHLT6KR2QLOWNA43DHKZKDU7UUN6O2A", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR"}
2019-11-17T08:17:12.902Z	INFO	piecestore	download failed	{"Piece ID": "O5IAFDLO242VZRCQGKDMZAHLT6KR2QLOWNA43DHKZKDU7UUN6O2A", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR", "error": "file does not exist"}
2019-11-17T08:19:45.127Z	INFO	piecestore	download started	{"Piece ID": "O5IAFDLO242VZRCQGKDMZAHLT6KR2QLOWNA43DHKZKDU7UUN6O2A", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR"}
2019-11-17T08:19:45.203Z	INFO	piecestore	download failed	{"Piece ID": "O5IAFDLO242VZRCQGKDMZAHLT6KR2QLOWNA43DHKZKDU7UUN6O2A", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR", "error": "file does not exist"}
2019-11-17T09:17:12.746Z	INFO	piecestore	download started	{"Piece ID": "O5IAFDLO242VZRCQGKDMZAHLT6KR2QLOWNA43DHKZKDU7UUN6O2A", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR"}
2019-11-17T09:17:12.746Z	INFO	piecestore	download failed	{"Piece ID": "O5IAFDLO242VZRCQGKDMZAHLT6KR2QLOWNA43DHKZKDU7UUN6O2A", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR", "error": "file does not exist"}
2019-11-17T09:19:03.390Z	INFO	piecestore	download started	{"Piece ID": "O5IAFDLO242VZRCQGKDMZAHLT6KR2QLOWNA43DHKZKDU7UUN6O2A", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR"}
2019-11-17T09:19:03.391Z	INFO	piecestore	download failed	{"Piece ID": "O5IAFDLO242VZRCQGKDMZAHLT6KR2QLOWNA43DHKZKDU7UUN6O2A", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR", "error": "file does not exist"}
2019-11-17T10:17:10.889Z	INFO	piecestore	download started	{"Piece ID": "O5IAFDLO242VZRCQGKDMZAHLT6KR2QLOWNA43DHKZKDU7UUN6O2A", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR"}
2019-11-17T10:17:14.078Z	INFO	piecestore	download failed	{"Piece ID": "O5IAFDLO242VZRCQGKDMZAHLT6KR2QLOWNA43DHKZKDU7UUN6O2A", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR", "error": "file does not exist"}
2019-11-17T11:17:11.311Z	INFO	piecestore	download started	{"Piece ID": "O5IAFDLO242VZRCQGKDMZAHLT6KR2QLOWNA43DHKZKDU7UUN6O2A", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR"}
2019-11-17T11:17:11.350Z	INFO	piecestore	download failed	{"Piece ID": "O5IAFDLO242VZRCQGKDMZAHLT6KR2QLOWNA43DHKZKDU7UUN6O2A", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR", "error": "file does not exist"}
2019-11-17T12:17:10.344Z	INFO	piecestore	download started	{"Piece ID": "O5IAFDLO242VZRCQGKDMZAHLT6KR2QLOWNA43DHKZKDU7UUN6O2A", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR"}
2019-11-17T12:17:13.446Z	INFO	piecestore	download failed	{"Piece ID": "O5IAFDLO242VZRCQGKDMZAHLT6KR2QLOWNA43DHKZKDU7UUN6O2A", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR", "error": "file does not exist"}
2019-11-17T13:17:18.222Z	INFO	piecestore	download started	{"Piece ID": "O5IAFDLO242VZRCQGKDMZAHLT6KR2QLOWNA43DHKZKDU7UUN6O2A", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR"}
2019-11-17T13:17:18.223Z	INFO	piecestore	download failed	{"Piece ID": "O5IAFDLO242VZRCQGKDMZAHLT6KR2QLOWNA43DHKZKDU7UUN6O2A", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR", "error": "file does not exist"}
2019-11-17T14:17:10.694Z	INFO	piecestore	download started	{"Piece ID": "O5IAFDLO242VZRCQGKDMZAHLT6KR2QLOWNA43DHKZKDU7UUN6O2A", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR"}
2019-11-17T14:17:10.744Z	INFO	piecestore	download failed	{"Piece ID": "O5IAFDLO242VZRCQGKDMZAHLT6KR2QLOWNA43DHKZKDU7UUN6O2A", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR", "error": "file does not exist"}
2019-11-17T17:17:15.119Z	INFO	piecestore	download started	{"Piece ID": "O5IAFDLO242VZRCQGKDMZAHLT6KR2QLOWNA43DHKZKDU7UUN6O2A", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR"}
2019-11-17T17:17:15.146Z	INFO	piecestore	download failed	{"Piece ID": "O5IAFDLO242VZRCQGKDMZAHLT6KR2QLOWNA43DHKZKDU7UUN6O2A", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR", "error": "file does not exist"}
2019-11-17T18:17:12.801Z	INFO	piecestore	download started	{"Piece ID": "O5IAFDLO242VZRCQGKDMZAHLT6KR2QLOWNA43DHKZKDU7UUN6O2A", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR"}
2019-11-17T18:17:12.802Z	INFO	piecestore	download failed	{"Piece ID": "O5IAFDLO242VZRCQGKDMZAHLT6KR2QLOWNA43DHKZKDU7UUN6O2A", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR", "error": "file does not exist"}
2019-11-17T18:27:02.680Z	INFO	piecestore	download started	{"Piece ID": "O5IAFDLO242VZRCQGKDMZAHLT6KR2QLOWNA43DHKZKDU7UUN6O2A", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR"}
2019-11-17T18:27:02.680Z	INFO	piecestore	download failed	{"Piece ID": "O5IAFDLO242VZRCQGKDMZAHLT6KR2QLOWNA43DHKZKDU7UUN6O2A", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR", "error": "file does not exist"}
2019-11-18T12:17:11.587Z	INFO	piecestore	download started	{"Piece ID": "O5IAFDLO242VZRCQGKDMZAHLT6KR2QLOWNA43DHKZKDU7UUN6O2A", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR"}
2019-11-18T12:17:11.623Z	INFO	piecestore	download failed	{"Piece ID": "O5IAFDLO242VZRCQGKDMZAHLT6KR2QLOWNA43DHKZKDU7UUN6O2A", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR", "error": "file does not exist"}
2019-11-18T16:53:59.797Z	INFO	piecestore	download started	{"Piece ID": "O5IAFDLO242VZRCQGKDMZAHLT6KR2QLOWNA43DHKZKDU7UUN6O2A", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR"}
2019-11-18T16:53:59.809Z	INFO	piecestore	download failed	{"Piece ID": "O5IAFDLO242VZRCQGKDMZAHLT6KR2QLOWNA43DHKZKDU7UUN6O2A", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR", "error": "file does not exist"}
2019-11-18T17:17:16.199Z	INFO	piecestore	download started	{"Piece ID": "O5IAFDLO242VZRCQGKDMZAHLT6KR2QLOWNA43DHKZKDU7UUN6O2A", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR"}
2019-11-18T17:17:16.199Z	INFO	piecestore	download failed	{"Piece ID": "O5IAFDLO242VZRCQGKDMZAHLT6KR2QLOWNA43DHKZKDU7UUN6O2A", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR", "error": "file does not exist"}
2019-11-18T17:27:03.752Z	INFO	piecestore	download started	{"Piece ID": "O5IAFDLO242VZRCQGKDMZAHLT6KR2QLOWNA43DHKZKDU7UUN6O2A", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR"}
2019-11-18T17:27:03.752Z	INFO	piecestore	download failed	{"Piece ID": "O5IAFDLO242VZRCQGKDMZAHLT6KR2QLOWNA43DHKZKDU7UUN6O2A", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR", "error": "file does not exist"}
2019-11-18T18:17:15.921Z	INFO	piecestore	download started	{"Piece ID": "O5IAFDLO242VZRCQGKDMZAHLT6KR2QLOWNA43DHKZKDU7UUN6O2A", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR"}
2019-11-18T18:17:16.945Z	INFO	piecestore	download failed	{"Piece ID": "O5IAFDLO242VZRCQGKDMZAHLT6KR2QLOWNA43DHKZKDU7UUN6O2A", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR", "error": "file does not exist"}
2019-11-18T18:19:03.007Z	INFO	piecestore	download started	{"Piece ID": "O5IAFDLO242VZRCQGKDMZAHLT6KR2QLOWNA43DHKZKDU7UUN6O2A", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR"}
2019-11-18T18:19:03.008Z	INFO	piecestore	download failed	{"Piece ID": "O5IAFDLO242VZRCQGKDMZAHLT6KR2QLOWNA43DHKZKDU7UUN6O2A", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR", "error": "file does not exist"}
2019-11-18T19:17:10.485Z	INFO	piecestore	download started	{"Piece ID": "O5IAFDLO242VZRCQGKDMZAHLT6KR2QLOWNA43DHKZKDU7UUN6O2A", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR"}
2019-11-18T19:17:10.485Z	INFO	piecestore	download failed	{"Piece ID": "O5IAFDLO242VZRCQGKDMZAHLT6KR2QLOWNA43DHKZKDU7UUN6O2A", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR", "error": "file does not exist"}
2019-11-18T19:19:03.178Z	INFO	piecestore	download started	{"Piece ID": "O5IAFDLO242VZRCQGKDMZAHLT6KR2QLOWNA43DHKZKDU7UUN6O2A", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR"}
2019-11-18T19:19:03.179Z	INFO	piecestore	download failed	{"Piece ID": "O5IAFDLO242VZRCQGKDMZAHLT6KR2QLOWNA43DHKZKDU7UUN6O2A", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR", "error": "file does not exist"}
2019-11-18T20:17:20.369Z	INFO	piecestore	download started	{"Piece ID": "O5IAFDLO242VZRCQGKDMZAHLT6KR2QLOWNA43DHKZKDU7UUN6O2A", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR"}
2019-11-18T20:17:20.369Z	INFO	piecestore	download failed	{"Piece ID": "O5IAFDLO242VZRCQGKDMZAHLT6KR2QLOWNA43DHKZKDU7UUN6O2A", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR", "error": "file does not exist"}
2019-11-18T20:27:02.612Z	INFO	piecestore	download started	{"Piece ID": "O5IAFDLO242VZRCQGKDMZAHLT6KR2QLOWNA43DHKZKDU7UUN6O2A", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR"}
2019-11-18T20:27:02.613Z	INFO	piecestore	download failed	{"Piece ID": "O5IAFDLO242VZRCQGKDMZAHLT6KR2QLOWNA43DHKZKDU7UUN6O2A", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR", "error": "file does not exist"}
2019-11-18T21:17:10.479Z	INFO	piecestore	download started	{"Piece ID": "O5IAFDLO242VZRCQGKDMZAHLT6KR2QLOWNA43DHKZKDU7UUN6O2A", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR"}
2019-11-18T21:17:10.908Z	INFO	piecestore	download failed	{"Piece ID": "O5IAFDLO242VZRCQGKDMZAHLT6KR2QLOWNA43DHKZKDU7UUN6O2A", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR", "error": "file does not exist"}
2019-11-18T21:27:02.797Z	INFO	piecestore	download started	{"Piece ID": "O5IAFDLO242VZRCQGKDMZAHLT6KR2QLOWNA43DHKZKDU7UUN6O2A", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR"}
2019-11-18T21:27:02.797Z	INFO	piecestore	download failed	{"Piece ID": "O5IAFDLO242VZRCQGKDMZAHLT6KR2QLOWNA43DHKZKDU7UUN6O2A", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR", "error": "file does not exist"}
2019-11-18T22:17:11.632Z	INFO	piecestore	download started	{"Piece ID": "O5IAFDLO242VZRCQGKDMZAHLT6KR2QLOWNA43DHKZKDU7UUN6O2A", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR"}
2019-11-18T22:17:11.679Z	INFO	piecestore	download failed	{"Piece ID": "O5IAFDLO242VZRCQGKDMZAHLT6KR2QLOWNA43DHKZKDU7UUN6O2A", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR", "error": "file does not exist"}
2019-11-18T22:19:07.789Z	INFO	piecestore	download started	{"Piece ID": "O5IAFDLO242VZRCQGKDMZAHLT6KR2QLOWNA43DHKZKDU7UUN6O2A", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR"}
2019-11-18T22:19:07.878Z	INFO	piecestore	download failed	{"Piece ID": "O5IAFDLO242VZRCQGKDMZAHLT6KR2QLOWNA43DHKZKDU7UUN6O2A", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR", "error": "file does not exist"}
2019-11-18T23:17:15.217Z	INFO	piecestore	download started	{"Piece ID": "O5IAFDLO242VZRCQGKDMZAHLT6KR2QLOWNA43DHKZKDU7UUN6O2A", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR"}
2019-11-18T23:17:15.260Z	INFO	piecestore	download failed	{"Piece ID": "O5IAFDLO242VZRCQGKDMZAHLT6KR2QLOWNA43DHKZKDU7UUN6O2A", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR", "error": "file does not exist"}
2019-11-19T00:17:12.632Z	INFO	piecestore	download started	{"Piece ID": "O5IAFDLO242VZRCQGKDMZAHLT6KR2QLOWNA43DHKZKDU7UUN6O2A", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR"}
2019-11-19T00:17:12.632Z	INFO	piecestore	download failed	{"Piece ID": "O5IAFDLO242VZRCQGKDMZAHLT6KR2QLOWNA43DHKZKDU7UUN6O2A", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR", "error": "file does not exist"}
2019-11-19T01:17:17.307Z	INFO	piecestore	download started	{"Piece ID": "O5IAFDLO242VZRCQGKDMZAHLT6KR2QLOWNA43DHKZKDU7UUN6O2A", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR"}
2019-11-19T01:17:17.334Z	INFO	piecestore	download failed	{"Piece ID": "O5IAFDLO242VZRCQGKDMZAHLT6KR2QLOWNA43DHKZKDU7UUN6O2A", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR", "error": "file does not exist"}
2019-11-19T02:17:10.945Z	INFO	piecestore	download started	{"Piece ID": "O5IAFDLO242VZRCQGKDMZAHLT6KR2QLOWNA43DHKZKDU7UUN6O2A", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR"}
2019-11-19T02:17:10.974Z	INFO	piecestore	download failed	{"Piece ID": "O5IAFDLO242VZRCQGKDMZAHLT6KR2QLOWNA43DHKZKDU7UUN6O2A", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR", "error": "file does not exist"}
2019-11-19T03:17:11.505Z	INFO	piecestore	download started	{"Piece ID": "O5IAFDLO242VZRCQGKDMZAHLT6KR2QLOWNA43DHKZKDU7UUN6O2A", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR"}
2019-11-19T03:17:11.506Z	INFO	piecestore	download failed	{"Piece ID": "O5IAFDLO242VZRCQGKDMZAHLT6KR2QLOWNA43DHKZKDU7UUN6O2A", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR", "error": "file does not exist"}
2019-11-19T04:17:13.066Z	INFO	piecestore	download started	{"Piece ID": "O5IAFDLO242VZRCQGKDMZAHLT6KR2QLOWNA43DHKZKDU7UUN6O2A", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR"}
2019-11-19T04:17:13.137Z	INFO	piecestore	download failed	{"Piece ID": "O5IAFDLO242VZRCQGKDMZAHLT6KR2QLOWNA43DHKZKDU7UUN6O2A", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR", "error": "file does not exist"}
2019-11-21T00:17:10.839Z	INFO	piecestore	download started	{"Piece ID": "O5IAFDLO242VZRCQGKDMZAHLT6KR2QLOWNA43DHKZKDU7UUN6O2A", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR"}
2019-11-21T00:17:10.856Z	INFO	piecestore	download failed	{"Piece ID": "O5IAFDLO242VZRCQGKDMZAHLT6KR2QLOWNA43DHKZKDU7UUN6O2A", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR", "error": "file does not exist"}
2019-11-21T00:25:58.889Z	INFO	piecestore	download started	{"Piece ID": "O5IAFDLO242VZRCQGKDMZAHLT6KR2QLOWNA43DHKZKDU7UUN6O2A", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR"}
2019-11-21T00:25:58.889Z	INFO	piecestore	download failed	{"Piece ID": "O5IAFDLO242VZRCQGKDMZAHLT6KR2QLOWNA43DHKZKDU7UUN6O2A", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR", "error": "file does not exist"}

I posted to one of our engineers to let the team know. Hopefully someone can comment form the tech team

1 Like

they said there’s no update yet, but that its being actively worked on. they said they are hoping to get it by the next release, if possible.

As soon as there’s any news to tell, we can let everyone know, thanks!

Here is a link to the design document

1 Like

Hello, I also started getting these error messages. Seems satellite 118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW is responsible for all of them:

2019-11-21T00:19:45.325Z        INFO    piecestore      download failed {"Piece ID": "SHNATMYVWQO73ZVFAAEVU6GSUVWUHNBEDZKVAQHFBOQHUPJ4D3XQ", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR", "error": "file does not exist"}
2019-11-21T00:19:45.326Z        ERROR   server  gRPC stream error response      {"error": "file does not exist"}
2019-11-21T02:08:31.580Z        INFO    piecestore      download failed {"Piece ID": "644EYFRPSAPIWIGJIMM7IR77YOD5JJ4NVH3XJ67HWBTP4RJA6LLA", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR", "error": "file does not exist"}
2019-11-21T02:08:31.580Z        ERROR   server  gRPC stream error response      {"error": "file does not exist"}
2019-11-21T02:12:56.065Z        INFO    piecestore      download failed {"Piece ID": "O4LGFWAMCTKDKAQT2CRLZLCIYAPLG7OMHU5CMTPQLDS7G6OLMYDA", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR", "error": "file does not exist"}
2019-11-21T02:12:56.065Z        ERROR   server  gRPC stream error response      {"error": "file does not exist"}
2019-11-21T02:17:14.465Z        INFO    piecestore      download failed {"Piece ID": "SHNATMYVWQO73ZVFAAEVU6GSUVWUHNBEDZKVAQHFBOQHUPJ4D3XQ", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR", "error": "file does not exist"}
2019-11-21T02:17:14.465Z        ERROR   server  gRPC stream error response      {"error": "file does not exist"}
2019-11-21T02:26:04.966Z        INFO    piecestore      download failed {"Piece ID": "SHNATMYVWQO73ZVFAAEVU6GSUVWUHNBEDZKVAQHFBOQHUPJ4D3XQ", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR", "error": "file does not exist"}
2019-11-21T02:26:04.966Z        ERROR   server  gRPC stream error response      {"error": "file does not exist"}
2019-11-21T02:26:54.882Z        INFO    piecestore      download failed {"Piece ID": "O4LGFWAMCTKDKAQT2CRLZLCIYAPLG7OMHU5CMTPQLDS7G6OLMYDA", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR", "error": "file does not exist"}
2019-11-21T02:26:54.882Z        ERROR   server  gRPC stream error response      {"error": "file does not exist"}
2019-11-21T02:28:19.025Z        INFO    piecestore      download failed {"Piece ID": "644EYFRPSAPIWIGJIMM7IR77YOD5JJ4NVH3XJ67HWBTP4RJA6LLA", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR", "error": "file does not exist"}
2019-11-21T02:28:19.025Z        ERROR   server  gRPC stream error response      {"error": "file does not exist"}
2019-11-21T02:44:58.950Z        INFO    piecestore      download failed {"Piece ID": "SHNATMYVWQO73ZVFAAEVU6GSUVWUHNBEDZKVAQHFBOQHUPJ4D3XQ", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR", "error": "file does not exist"}
2019-11-21T02:44:58.950Z        ERROR   server  gRPC stream error response      {"error": "file does not exist"}
2019-11-21T02:47:54.223Z        INFO    piecestore      download failed {"Piece ID": "644EYFRPSAPIWIGJIMM7IR77YOD5JJ4NVH3XJ67HWBTP4RJA6LLA", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR", "error": "file does not exist"}
2019-11-21T02:47:54.223Z        ERROR   server  gRPC stream error response      {"error": "file does not exist"}
2019-11-21T02:48:46.122Z        INFO    piecestore      download failed {"Piece ID": "O4LGFWAMCTKDKAQT2CRLZLCIYAPLG7OMHU5CMTPQLDS7G6OLMYDA", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR", "error": "file does not exist"}
2019-11-21T02:48:46.122Z        ERROR   server  gRPC stream error response      {"error": "file does not exist"}
2019-11-21T03:21:48.297Z        INFO    piecestore      download failed {"Piece ID": "SHNATMYVWQO73ZVFAAEVU6GSUVWUHNBEDZKVAQHFBOQHUPJ4D3XQ", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR", "error": "file does not exist"}
2019-11-21T03:21:48.297Z        ERROR   server  gRPC stream error response      {"error": "file does not exist"}
2019-11-21T04:23:31.515Z        INFO    piecestore      download failed {"Piece ID": "SHNATMYVWQO73ZVFAAEVU6GSUVWUHNBEDZKVAQHFBOQHUPJ4D3XQ", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR", "error": "file does not exist"}
2019-11-21T04:23:31.516Z        ERROR   server  gRPC stream error response      {"error": "file does not exist"}
2019-11-21T04:32:37.461Z        INFO    piecestore      download failed {"Piece ID": "644EYFRPSAPIWIGJIMM7IR77YOD5JJ4NVH3XJ67HWBTP4RJA6LLA", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR", "error": "file does not exist"}
2019-11-21T04:32:37.461Z        ERROR   server  gRPC stream error response      {"error": "file does not exist"}

2019-11-23T07:17:09.957Z INFO piecestore download failed {"Piece ID""JNZHQHP3YJUXAQUB5IGCHAVESULRQOL6SQ5CMX4V5QE77VDUKLUA", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR", "error": "file does not exist"}: 2019-11-23T07:17:09.957Z ERROR server gRPC stream error response {"error""file does not exist"}: 2019-11-23T07:27:03.565Z INFO piecestore download failed {"Piece ID": "JNZHQHP3YJUXAQUB5IGCHAVESULRQOL6SQ5CMX4V5QE77VDUKLUA", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR", "error": "file does not exist"} 2019-11-23T07:27:03.565Z ERROR server gRPC stream error response {"error": "file does not exist"} 2019-11-23T08:17:17.358Z INFO piecestore download failed {"Piece ID": "JNZHQHP3YJUXAQUB5IGCHAVESULRQOL6SQ5CMX4V5QE77VDUKLUA", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR", "error": "file does not exist"} 2019-11-23T08:17:17.358Z ERROR server gRPC stream error response {"error": "file does not exist"} 2019-11-23T09:17:09.455Z INFO piecestore download failed {"Piece ID": "JNZHQHP3YJUXAQUB5IGCHAVESULRQOL6SQ5CMX4V5QE77VDUKLUA", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR", "error": "file does not exist"} 2019-11-23T09:17:09.455Z ERROR server gRPC stream error response {"error": "file does not exist"}

v.0.26.2

Have thoose to, began with last update for me.
Well havnt looked for Them earlier🙂

You had them before but the satellite was not trying to repair them. With the latest update we enabled disqualification. You can’t get disqualified for one of our mistakes. Even if you fail an audit on one of these pieces we do not decrease your reputation. Hoever some storage nodes did delete data and can’t blame this bug for it. They got disqualfied and that will add more segments to the repair queue. This time it was a segment that was stored on your storage node. That is the reason you get these messages now.

The solution for this issue was included in the release but I had not time to execute it. Stay tuned.

4 Likes

Thanks for the update! I’ll check back for these once the fix is fully rolled out. I mostly see it with repair traffic, but also sometimes for audits or deletes that can’t find the file.

0.28.2
2019-12-18T18:16:31.707Z INFO piecestore download failed {"Piece ID": "2IHJSDNN7RKA522LZ3YBHTWY5KVVA2JUFLAT4TI4D25XPBEYP4EQ", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_REPAIR", "error": "file does not exist"}

16 similar errors in 1 hour of work

Zombie segments deleted. The error should be gone.

In a few hours garbage collection should handle the rest.

2 Likes

why is this happening and should I be concern
2019-12-16T23:28:25.800559946Z 2019-12-16T23:28:25.800Z INFO piecestore download failed {“Piece ID”: “KGCQ6VGMNG2LQRRQHI4D4TFKEJPY4JVKLJEL6XAANH6LS34YOGFQ”, “Satellite ID”: “118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW”, “Action”: “GET_REPAIR”, “error”: “file does not exist”}
2019-12-16T23:15:49.406793217Z 2019-12-16T23:15:49.404Z INFO piecestore download failed {“Piece ID”: “BKFWXXC65GEBP4WRSH2PHCWLJYUU34LHWHCMGUTV56FBYZDIBEWA”, “Satellite ID”: “118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW”, “Action”: “GET_REPAIR”, “error”: “file does not exist”}
2019-12-16T23:15:49.431303385Z 2019-12-16T23:15:49.431Z INFO piecestore download failed {“Piece ID”: “WPFDIX6MO6HH6NWHOPO6CEFTJJVRNKZBBFPHPJ7Z4TOVEQO75A6Q”, “Satellite ID”: “118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW”, “Action”: “GET_REPAIR”, “error”: “file does not exist”}
2019-12-16T23:15:49.446533401Z 2019-12-16T23:15:49.446Z INFO piecestore download failed {“Piece ID”: “SFCYKL562L4K6GNIYKAGGIBKR4MDYPFFWAICUZSU3LE5ULXFXNFQ”, “Satellite ID”: “118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW”, “Action”: “GET_REPAIR”, “error”: “file does not exist”}

2019-12-16T23:15:49.562127985Z 2019-12-16T23:15:49.561Z INFO piecestore download failed {“Piece ID”: “Z4FLJ2LBVRGLB6ZZW7OKCSYQFC3LXLXY7H4CZNRER4GQDFM36SGA”, “Satellite ID”: “118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW”, “Action”: “GET_REPAIR”, “error”: “file does not exist”}

2019-12-16T23:10:49.368331012Z 2019-12-16T23:10:49.368Z INFO piecestore download failed {“Piece ID”: “LBNGHGO22UC75KZTDLPO4YXGPIA4LESOOFE3LB22QNMBJYYTLHJA”, “Satellite ID”: “118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW”, “Action”: “GET_REPAIR”, “error”: “file does not exist”}

2019-12-16T23:03:38.499555211Z 2019-12-16T23:03:38.499Z INFO piecestore download failed {“Piece ID”: “CVAQPQPYC5OSKEVLB3TJ4B7GWT4B2H4W54WUE445L4MUCRAV37XQ”, “Satellite ID”: “118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW”, “Action”: “GET_REPAIR”, “error”: “file does not exist”}

2019-12-16T23:03:44.499187545Z 2019-12-16T23:03:44.499Z INFO piecestore download failed {“Piece ID”: “3KJ62DOTFL6I2OU5HBNUITUFJN4FEWGJPOFSWVNOFUVEECMKVRAQ”, “Satellite ID”: “118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW”, “Action”: “GET_REPAIR”, “error”: “file does not exist”}

2019-12-16T22:56:50.898087860Z 2019-12-16T22:56:50.897Z INFO piecestore download failed {“Piece ID”: “VCSUXNZKRTTC2WV2SYOKB6KRKYIDTJFCAWAU3ETD66EPZU22JCCQ”, “Satellite ID”: “118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW”, “Action”: “GET_REPAIR”, “error”: “file does not exist”}

2019-12-16T22:56:57.357092594Z 2019-12-16T22:56:57.356Z INFO piecestore download failed {“Piece ID”: “IYZWGJ2GH4ZCJKJ4J3BVOTY4MKD34TDFND33AENKMPXI54G62FIA”, “Satellite ID”: “118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW”, “Action”: “GET_REPAIR”, “error”: “file does not exist”}

2019-12-16T22:50:49.073832415Z 2019-12-16T22:50:49.073Z INFO piecestore download failed {“Piece ID”: “L3NMZAZTXPEIUNTVCOBEMQ4ARQGT6JP4T3DOO6GRNGXDRMOTC7MQ”, “Satellite ID”: “118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW”, “Action”: “GET_REPAIR”, “error”: “file does not exist”}

2019-12-16T22:45:49.665742124Z 2019-12-16T22:45:49.665Z INFO piecestore download failed {“Piece ID”: “PCO4M5TO3M3OX45BHUOPBMQGGMNV5HSUJULY5FORV5TRRWHOGK6Q”, “Satellite ID”: “118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW”, “Action”: “GET_REPAIR”, “error”: “file does not exist”}

2019-12-16T22:45:50.075320790Z 2019-12-16T22:45:50.075Z INFO piecestore download failed {“Piece ID”: “DPJL2AK7AHODGQH3DT4FMHM3RPKQ3W32DGOCXH5XYHTACQ5SFZUQ”, “Satellite ID”: “118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW”, “Action”: “GET_REPAIR”, “error”: “file does not exist”}

2019-12-16T22:38:26.422184144Z 2019-12-16T22:38:26.421Z INFO piecestore download failed {“Piece ID”: “KGCQ6VGMNG2LQRRQHI4D4TFKEJPY4JVKLJEL6XAANH6LS34YOGFQ”, “Satellite ID”: “118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW”, “Action”: “GET_REPAIR”, “error”: “file does not exist”}

First, why haven’t you created a new post with your issue instead of posting it here ?

Second, you seem to have lost data. You can search for the piece ID in your logs if you have redirected your log to a file.

Sorry, but it not a data loss, that issue persists few releases here. I reported it here but no reaction.

I think we have a gap with feedbacks because it appears a second time when the problem is reported but no any reactions.

You can check you logs with:

docker logs storagenode 2>&1 | grep -c "file does not exist" 
762

and see details with:

docker logs storagenode 2>&1 | grep "file does not exist"

This problem with REPAIR traffic persists on all nodes.

This same reason @Alexey gave applies to REPAIR traffic too. Those missing pieces aren’t yet completely cleaned from the databases.

Also these particular pieces don’t carry a signature which result in losing rep. In short these don’t affect your audit score at all.

@sorry2xs This might apply to your issue too but please don’t club your issue with irrelevant threads. JesusG, you have been a SNO for long now :slight_smile:

2 Likes

I opened a issue on GitHub about the REPAIR traffic problem.
Let’s keep this thread free for other comments :+1:

1 Like

My reaction was here: Storj lost one piece? Repeated "file does not exist" in GET_REPAIR

I am sorry that I don’t have the time to catch all similar posts. I will simply miss some of them.

1 Like

last post here on this subject I have notice that the file does not exists error only happens my docker node the msi windows node does not produce that error thet is my last 2 cents worth.

Don’t worry, it not a problem, I think it also my fail too, I should post the initial issue on Github it proper place for issues.
I just take care of our product launch will go smoothly and not have big issues.