Yes, that’s why I said the node had a silent issue, that I did notice.
Until 06/08 around 1000 it looks fine and the node responds to GET_AUDIT requests with downloaded.
Then it stops responding succesfully and I see multiple audits that have “Download started” but no “Downloaded” for it.
The satellite keeps requesting audits.
Between 1516 and 1800 there is no audits, probably the node was not able to handle anything due to the silent error. It resumed around 1800 then the satellite picked up requesting pieces for audits.
I see start at 1813 requesting the same piece for auditing:
2021-08-06T18:13:31.696Z INFO piecestore download started {"Piece ID": "FIQ3PWFNVTUIGT5NYSSEYKLEZ3IUR3UU3IIHPIOAYLTRWZVTZATA", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "GET_AUDIT"}
2021-08-06T18:46:22.176Z INFO piecestore download started {"Piece ID": "FIQ3PWFNVTUIGT5NYSSEYKLEZ3IUR3UU3IIHPIOAYLTRWZVTZATA", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "GET_AUDIT"}
2021-08-06T18:55:42.670Z INFO piecestore download started {"Piece ID": "FIQ3PWFNVTUIGT5NYSSEYKLEZ3IUR3UU3IIHPIOAYLTRWZVTZATA", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "GET_AUDIT"}
2021-08-06T19:02:44.963Z INFO piecestore download started {"Piece ID": "FIQ3PWFNVTUIGT5NYSSEYKLEZ3IUR3UU3IIHPIOAYLTRWZVTZATA", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "GET_AUDIT"}
2021-08-06T19:06:42.317Z INFO piecestore download started {"Piece ID": "FIQ3PWFNVTUIGT5NYSSEYKLEZ3IUR3UU3IIHPIOAYLTRWZVTZATA", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "GET_AUDIT"}
2021-08-06T19:40:15.145Z INFO piecestore download canceled {"Piece ID": "FIQ3PWFNVTUIGT5NYSSEYKLEZ3IUR3UU3IIHPIOAYLTRWZVTZATA", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "GET_AUDIT"}
2021-08-06T19:42:53.615Z INFO piecestore download canceled {"Piece ID": "FIQ3PWFNVTUIGT5NYSSEYKLEZ3IUR3UU3IIHPIOAYLTRWZVTZATA", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "GET_AUDIT"}
2021-08-06T19:47:26.911Z INFO piecestore download canceled {"Piece ID": "FIQ3PWFNVTUIGT5NYSSEYKLEZ3IUR3UU3IIHPIOAYLTRWZVTZATA", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "GET_AUDIT"}
2021-08-06T19:49:06.452Z INFO piecestore download canceled {"Piece ID": "FIQ3PWFNVTUIGT5NYSSEYKLEZ3IUR3UU3IIHPIOAYLTRWZVTZATA", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "GET_AUDIT"}
2021-08-06T20:02:53.329Z INFO piecestore download started {"Piece ID": "FIQ3PWFNVTUIGT5NYSSEYKLEZ3IUR3UU3IIHPIOAYLTRWZVTZATA", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "GET_AUDIT"}
2021-08-06T20:06:52.313Z INFO piecestore download started {"Piece ID": "FIQ3PWFNVTUIGT5NYSSEYKLEZ3IUR3UU3IIHPIOAYLTRWZVTZATA", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "GET_AUDIT"}
2021-08-06T20:16:46.677Z INFO piecestore download started {"Piece ID": "FIQ3PWFNVTUIGT5NYSSEYKLEZ3IUR3UU3IIHPIOAYLTRWZVTZATA", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "GET_AUDIT"}
2021-08-06T20:17:19.117Z INFO piecestore download started {"Piece ID": "FIQ3PWFNVTUIGT5NYSSEYKLEZ3IUR3UU3IIHPIOAYLTRWZVTZATA", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "GET_AUDIT"}
2021-08-06T20:45:40.431Z INFO piecestore download started {"Piece ID": "FIQ3PWFNVTUIGT5NYSSEYKLEZ3IUR3UU3IIHPIOAYLTRWZVTZATA", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "GET_AUDIT"}
2021-08-06T21:10:01.886Z INFO piecestore download started {"Piece ID": "FIQ3PWFNVTUIGT5NYSSEYKLEZ3IUR3UU3IIHPIOAYLTRWZVTZATA", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "GET_AUDIT"}
2021-08-06T21:10:54.685Z INFO piecestore download started {"Piece ID": "FIQ3PWFNVTUIGT5NYSSEYKLEZ3IUR3UU3IIHPIOAYLTRWZVTZATA", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "GET_AUDIT"}
2021-08-06T21:20:38.667Z INFO piecestore download started {"Piece ID": "FIQ3PWFNVTUIGT5NYSSEYKLEZ3IUR3UU3IIHPIOAYLTRWZVTZATA", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "GET_AUDIT"}
2021-08-06T21:28:12.355Z INFO piecestore download started {"Piece ID": "FIQ3PWFNVTUIGT5NYSSEYKLEZ3IUR3UU3IIHPIOAYLTRWZVTZATA", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "GET_AUDIT"}
2021-08-06T21:38:27.972Z INFO piecestore download started {"Piece ID": "FIQ3PWFNVTUIGT5NYSSEYKLEZ3IUR3UU3IIHPIOAYLTRWZVTZATA", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "GET_AUDIT"}
2021-08-06T22:21:45.268Z INFO piecestore download canceled {"Piece ID": "FIQ3PWFNVTUIGT5NYSSEYKLEZ3IUR3UU3IIHPIOAYLTRWZVTZATA", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "GET_AUDIT"}
2021-08-06T22:27:18.063Z INFO piecestore download canceled {"Piece ID": "FIQ3PWFNVTUIGT5NYSSEYKLEZ3IUR3UU3IIHPIOAYLTRWZVTZATA", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "GET_AUDIT"}
2021-08-06T22:30:46.174Z INFO piecestore download canceled {"Piece ID": "FIQ3PWFNVTUIGT5NYSSEYKLEZ3IUR3UU3IIHPIOAYLTRWZVTZATA", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "GET_AUDIT"}
2021-08-06T22:30:46.676Z ERROR piecestore download failed {"Piece ID": "FIQ3PWFNVTUIGT5NYSSEYKLEZ3IUR3UU3IIHPIOAYLTRWZVTZATA", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "GET_AUDIT", "error": "tls: use of closed connection", "errorVerbose": "tls: use of closed connection\n\tstorj.io/drpc/drpcstream.(*Stream).rawFlushLocked:276\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:318\n\tstorj.io/common/pb.(*drpcPiecestore_DownloadStream).Send:302\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func5.1:608\n\tstorj.io/common/rpc/rpctimeout.Run.func1:22"}
2021-08-06T22:30:46.773Z ERROR piecestore download failed {"Piece ID": "FIQ3PWFNVTUIGT5NYSSEYKLEZ3IUR3UU3IIHPIOAYLTRWZVTZATA", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "GET_AUDIT", "error": "tls: use of closed connection", "errorVerbose": "tls: use of closed connection\n\tstorj.io/drpc/drpcstream.(*Stream).rawFlushLocked:276\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:318\n\tstorj.io/common/pb.(*drpcPiecestore_DownloadStream).Send:302\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func5.1:608\n\tstorj.io/common/rpc/rpctimeout.Run.func1:22"}
2021-08-06T22:30:49.735Z INFO piecestore download canceled {"Piece ID": "FIQ3PWFNVTUIGT5NYSSEYKLEZ3IUR3UU3IIHPIOAYLTRWZVTZATA", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "GET_AUDIT"}
2021-08-06T22:30:50.037Z INFO piecestore download canceled {"Piece ID": "FIQ3PWFNVTUIGT5NYSSEYKLEZ3IUR3UU3IIHPIOAYLTRWZVTZATA", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "GET_AUDIT"}
For Europe-North the log shows a similar pattern.
I think I restarted the node next day but it was too late the node had been disqualified already since then on both satellites. So finally disqualification happened in less than 24 hours from correctly responding to audits to disqualification.