No change on Suspension or Audit statistics after 3 days of actual downtime: why?

[jeremyfritzen]

Hi,

My Internet access was down for 3 days. It is back since monday 12pm and noticed that this downtime impacted my 4 nodes in different ways (these 4 nodes are all behind the same IP address, behind the same network).

• 2 of my nodes are still 100% in Audit and Suspension scores (on the GUI Dashboard).
• 1 of my nodes has the Audit score quite impacted (on Europe west satellite, Audit score is 97.03%, which seems consistent with my actual downtime of about ~72 hours). Suspension is not impacted for this node.
• 1 of my node has only the Suspension score only on 1 satellite (Europe West): 99.99995%

So, I have several questions:

• Why are Audit and Suspension socres not impacted for 2 of my nodes? The real downtime was about 72 hours. It is a good news but a little bit surprising that these scores have not been impacted.
• Why is Audit score so much impacted on one of my nodes? I thought it was related to corrupted data (and not related to the downtime) but I’m not sure.

Thanks for your insight!

[littleskunk]

@jeremyfritzen

Why are Audit and Suspension socres not impacted for 2 of my nodes? The real downtime was about 72 hours. It is a good news but a little bit surprising that these scores have not been impacted.

That is simple.

1. If you are offline and the satellite can’t reach out to you with the audit request the online score is impacted.
2. If the satellite was able to reach you but you didn’t deliver the data the satellite will put you into containment mode and retry the same piece 3 more time. If you fail all of them the same way it will decrease your audit score.
3. If you respond with “I don’t have the data” or the data you returned was corrupted the audit score will decrease.
4. If you return an unknown audit error like “clock out of sync” that will decrease the suspension score.

What ever you do you can only decrease one of the scores. Offline doesn’t decrease audit and suspension score.

[jeremyfritzen]

Thanks!
And how can I know my online score? I’m asking because it seems that DQ for downtime will be enabled soon or later.

[Alexey]

http://localhost:14002/api/sno/satellite/1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE
http://localhost:14002/api/sno/satellite/121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6
and so on
Look at onlineScore

[jeremyfritzen]

Thanks.
Online score is 1 on all my nodes. Is that normal after 72h of downtime?
Maybe because DQ for downtime is not enabled yet?

[Alexey]

Or it doesn’t work as expected…
We will see in the next version, it should be available on the dashboard:

@littleskunk

Online Score
The storage node dashboard is now showing the online score for each satellite. On the satellite side we also changed the calculation for the online score. The old release was giving nodes a score of 100% as long as we didn’t had enough audit results. We want to make sure we only suspend nodes if we have enough audits. The new release will now show the real score while still waiting with the suspension until we have enough audits to make that call.

[endurance]

Hmm I had half a day outage since the inet company was digging around the street cabinet:

Two days later I saw the impact on the dashboard. What confuses me is

• Why is asia-east more impacted
• Why does the value not change since more as a week of 100% online already since the downtime

is there a more detailed desciption how those values should work?

[jeremyfritzen]

I think the downtime calculation is described on these 2 posts below.

In a nutshell, what I understand is:

• the downtime score is based on audits (i.e. audits requests sent by the satellites to the node)
• the node may receive a varying number of audits from satellites
• then, for a given node, the score may be different depending on the satellite we are considering.

Please correct me or complete if I’m wrong or missing something.
(would be really nice to have a topic on this on https://support.storj.io/)

Changelog v1.12.3

I have an update for you. The storage node dashboard API contains a new uptime score. By default it is 1 aka 100%. It is not visible on the storage node dashboard. The score gets updated via audits. The currenty default settings are: # The length of time to give suspended SNOs to diagnose and fix issues causing downtime. Afterwards, they will have one tracking period to reach the minimum online score before disqualification # overlay.audit-history.grace-period: 168h0m0s # The point below whi…

Changelog v1.12.3

One update on the downtime tracking topic: If you managed to collect too much downtime you will get suspended first. The satellite gives you 7 days to fix the issue and additional 30 days later the satellite will make a decsision. So lets say we are strict and would suspend for 48 hours downtime and you managed to be offline for 2 days in a row. You get your storage node back online but these 48 hours of downtime will not go away for the next 30 days. You would be suspended for 30 days. Luckily…

[endurance]

Thx and the critical watermark is 93%? Assumption based on 94% showing in yellow.
And it can go up only after 30days again? This is time after incident not a calendar month I assume?

[jeremyfritzen]

I don’t know the thresholds but from what I’ve been told, it should be up back again in a few days if you keep your node up and running.

[kevink]

The uptime score is a moving window of 30 days. So as long as your day with the downtime is within that window, the uptime score won’t go up.

E.g. if the window would be 3 days and you have a downtime on the last day:

1. 100%
2. 100%
3. 50%

So uptime score would be 83%. Now on the next day it would look like this:

1. 100%
2. 50%
3. 100%

So the average is still 83% as long as that day with 50% is within that window. Only once its outside the window the score will go up again.