Missing Piece before the wipe? - Critical audit alert

@Alexey, It looks like I spoke too soon. The issue with 1 failed audit repeating has come back and possibly while I was putting node into shutdown mode to remove the node.log entry from to config file (see other thread regarding node.log file not updating, which I wasn’t able to replicate a second time so I removed the config entry).

I had no failed audits prior to this change (saved a copy of the full node.log file prior to removing the setting entry from config).

6 hours later for a quick log check I found 1 failed audit from satellite 118U. I’ve checked the new logs and found no upload entry for this piece id. There is only a failed audit entry.

sudo docker logs storagenode 2>&1 | grep GET_AUDIT | grep "download failed"
2019-09-15T11:10:58.406Z        INFO    piecestore      download failed {"Piece ID": "ZM3HYYURHZWNRSKGL46FGIOZ4GHDSPVHTW3GAEIVSPKJ5F5S7IKA", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_AUDIT", "error": "rpc error: code = NotFound desc = file does not exist"}
 sudo docker logs storagenode 2>&1 | grep "ZM3HYYURHZWNRSKGL46FGIOZ4GHDSPVHTW3GAEIVSPKJ5F5S7IKA"
2019-09-15T11:10:58.405Z        INFO    piecestore      download started        {"Piece ID": "ZM3HYYURHZWNRSKGL46FGIOZ4GHDSPVHTW3GAEIVSPKJ5F5S7IKA", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_AUDIT"}
2019-09-15T11:10:58.406Z        INFO    piecestore      download failed {"Piece ID": "ZM3HYYURHZWNRSKGL46FGIOZ4GHDSPVHTW3GAEIVSPKJ5F5S7IKA", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_AUDIT", "error": "rpc error: code = NotFound desc = file does not exist"}

My audit score was 1 for this satellite, due to this new issue it’s now down to 0.69 :frowning:

I have a feeling the docker stop command may not be gracefully shutting down the node OR; the satellite which never received confirmation that a piece was uploaded to my node thinks it’s uploaded.

Can 1 of the data engineers investigate?