auditScore on tardigrade decreased without errors

[Bivvo]

The audit score on tardigrade has dropped some minutes ago to 95%.

In parallel, there are NO errors, failed audits nor repair uploads or downloads within the last 24h.

How can that be?

auditScore 95% @ saltlake.tardigrade.io ...

.. audits (r: 0.00%, c: 0.00%, s: 100%)
.. downloads (c: 0.50%, f: 0.26%, s: 99%)
.. uploads (c: 0.23%, f: 0.01%, s: 100%)
.. rep down (c: 0.00%, f: 0.00%, s: 100%)
.. rep up (c: 0.00%, f: 0.00%, s: 100%)

Referencing the monitoring tool used.

From the logs directly:

pi@pi:~ $ LOGMINDATE=$(TZ=UTC date --date="1
440 minutes ago" +'%Y-%m-%dT%H:%M:%S.000Z')
pi@pi:~ $ cat /mnt/WD1003/logs/sn1.log | awk
 -v date="$LOGMINDATE" '$1 > date' | grep -E 'GET_AUD
IT' | grep 'failed' -c
0
pi@pi:~ $ cat /mnt/WD1003/logs/sn1.log | awk
 -v date="$LOGMINDATE" '$1 > date' | grep -E 'GET_REP
AIR' | grep 'failed' -c
0
pi@pi:~ $ cat /mnt/WD1003/logs/sn1.log | awk
 -v date="$LOGMINDATE" '$1 > date' | grep -E 'PUT_REP
AIR' | grep 'failed' -c
0

[Alexey]

You need to measure time between audit request and when your node finished uploading this piece.
See examples:

Got Disqualified from saltlake

$ cat /mnt/x/storagenode3/storagenode.log | grep 1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE | grep -E "GET_AUDIT" | jq -R '. | split("\t") | (.[4] | fromjson) as $body | {SatelliteID: $body."Satellite ID", ($body."Piece ID"): {(.[0]): .[3]}}' | jq -s 'reduce .[] as $item ({}; . * $item)' ... "EY6AAGG33LEZK3O7NIM2S3R5VEPXRBIYCKPWYSV5LDWNMHFYF7EA": { "2021-08-22T22:42:40.601Z": "download started", "2021-08-22T22:50:28.975Z": "download started", "2021-08-22T23:04:59.672Z": "d…

Script to count time for downloads (uploads) pieces

Here’s my version: Just pipe the log content to it and it will find the fastest and slowest times. An example: # cat /storj/node/log/docker.00065.log | grep 12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs | grep GET_REPAIR | php piece_times.php Slowest: satellite: 12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs, piece: DZSAWMEP4KLTX2CNHAJTFSONAH7XLJNUMYI3AL5AYXAPTZLLJNKA, time: 4.440 Fastest: satellite: 12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs, piece: ZSYW6HKR2WZ7GZI7I3MX…

The second case - if your node provided broken pieces, in the storagenode’s log the download will finish successfully without any error, but the audit score will be affected.

[Bivvo]

Thank you, will analyze that.

Isn’t that some kind of a wormhole? Meaning: without ANY errors the node can be disqualified, if there’s no constant monitoring on the scores (besides the logs) and a fast reaction of the SNO, in case it’s being worse (ugh, my monitoring script works pretty well, I’ve recognized).

That’s really a mess, if it’s happening with a huge node and a lot of audit traffic. Because suspension will be skipped and it’s very fast disqualified.

@Alexey

if your node provided broken pieces, in the storagenode’s log the download will finish successfully without any error, but the audit score will be affected.

This can be the case as well from my issues at the beginning of the year with the performance on MacOS, before I’ve switched back to RPi. Crossing fingers these are some minor hiccups over time as this one, IF that was the reason.

[Alexey]

It would be nice if the satellite could provide some feedback (audit results), but it could help to malicious users too to improve their strategy…
However, you can create an issue to add such feedback to the storagenode’s logs.

[Bivvo]

@Alexey

you can create an issue to add such feedback to the storagenode’s logs.

done

[Bivvo]

@Alexey

time between audit request

Checked - that’s not the case: maximum time lag was 0.7 seconds.

@Alexey

if your node provided broken pieces, in the storagenode’s log the download will finish successfully without any error, but the audit score will be affected.

So it must be this one.

Consequently I calm down, because:

@Bivvo

Crossing fingers these are some minor hiccups over time as this one

Anyway, I’ve added the time lag analysis to the issue list of the monitoring script as well. At least I can create a routine which indicates time lags between audit requests, without the need to have it reported in the logs.

[Bivvo]

@Bivvo

Anyway, I’ve added the time lag analysis to the issue list of the monitoring script as well.

@Alexey I’ve added an automatic alert in version 1.9.0 in case time lags between audit downloads started and finished > 3 mins. Users will be notified automatically by mail and/or discord push message in order to quickly react and avoid a disqualification of their nodes. Will add it to the wiki page, too.