Missing Piece before the wipe? - Critical audit alert

Node Operators troubleshooting
82

I think dev team is working on fix, here is changelog of current release and I did not see any related changes.

1 Like
83

Do satellite version numbers match the storagenode version numbers though? This change wouldn’t be in storagenode, it would be in the satellites, so I wouldn’t expect to see anything in storagenode release notes about it.

84

They sometimes do and sometimes dont.
I think sat,s are ,1 higher then sno,s atm

85

Just click here:

image
image958×781 76.4 KB

And you can see full change log.

Also for sattelite we have addition hot fix.

1 Like
86

Got 2 more failed audits for deleted peaces today:

# docker logs  storagenode 2>&1 | grep -i audit | grep -i fail
2019-10-07T05:08:34.491Z        INFO    piecestore      download failed {"Piece ID": "GBJQEXIVBWX5VT46IBLN2NY4JQH7DIX67YXKF4HY6LIQW5FWFJFA", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_AUDIT", "error": "rpc error: code = NotFound desc = file does not exist"}
2019-10-08T11:44:51.101Z        INFO    piecestore      download failed {"Piece ID": "WIQVOTKJ3FKOFRFM7U44ED5PDURCEMW7HTD2ERL3DDRSQSGO6VPA", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_AUDIT", "error": "rpc error: code = NotFound desc = file does not exist"}
2019-10-08T15:36:28.844Z        INFO    piecestore      download failed {"Piece ID": "SED27QZULXXRLKP5WLS5TATCI7CMX4UN5E4VWXCBNGAVQ2O4XK7A", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_AUDIT", "error": "rpc error: code = NotFound desc = file does not exist"
# grep -i SED27QZULXXRLKP5WLS5TATCI7CMX4UN5E4VWXCBNGAVQ2O4XK7A logs/*
logs/25-09-19.log:2019-09-21T14:56:00.654Z      INFO    piecestore      deleted {"Piece ID": "SED27QZULXXRLKP5WLS5TATCI7CMX4UN5E4VWXCBNGAVQ2O4XK7A"}
# grep -i SED27QZULXXRLKP5WLS5TATCI7CMX4UN5E4VWXCBNGAVQ2O4XK7A logs/*
logs/25-09-19.log:2019-09-21T14:56:00.654Z      INFO    piecestore      deleted {"Piece ID": "SED27QZULXXRLKP5WLS5TATCI7CMX4UN5E4VWXCBNGAVQ2O4XK7A"}

Does anyone know what is the impact of these?
Can these failures be discarded for my SN maybe? It took a lot of effort to keep my node up at all times and it’s disappointing to see my score drop at no fault on my side.

87

Your score should not drop if you didn’t lost those pieces. At least we have a fix which should prevent this.

2 Likes
88

Ok thanks.
I was looking at number of failed audits as total - success and this number did not recover. Now looking at the score it does drop but recovers on the same day so should be fine:

image
image2918×294 20.3 KB

Wondering if this Is a normal score recovery rate or the fix at work?

1 Like
89

You are welcome!
Just check your dashboard for the score: https://documentation.storj.io/setup/cli/dashboard