No change on Suspension or Audit statistics after 3 days of actual downtime: why?

Hi,

My Internet access was down for 3 days. It is back since monday 12pm and noticed that this downtime impacted my 4 nodes in different ways (these 4 nodes are all behind the same IP address, behind the same network).

  • 2 of my nodes are still 100% in Audit and Suspension scores (on the GUI Dashboard).
  • 1 of my nodes has the Audit score quite impacted (on Europe west satellite, Audit score is 97.03%, which seems consistent with my actual downtime of about ~72 hours). Suspension is not impacted for this node.
  • 1 of my node has only the Suspension score only on 1 satellite (Europe West): 99.99995%

So, I have several questions:

  • Why are Audit and Suspension socres not impacted for 2 of my nodes? The real downtime was about 72 hours. It is a good news but a little bit surprising that these scores have not been impacted.
  • Why is Audit score so much impacted on one of my nodes? I thought it was related to corrupted data (and not related to the downtime) but I’m not sure.

Thanks for your insight!

1 Like

That is simple.

  1. If you are offline and the satellite can’t reach out to you with the audit request the online score is impacted.
  2. If the satellite was able to reach you but you didn’t deliver the data the satellite will put you into containment mode and retry the same piece 3 more time. If you fail all of them the same way it will decrease your audit score.
  3. If you respond with “I don’t have the data” or the data you returned was corrupted the audit score will decrease.
  4. If you return an unknown audit error like “clock out of sync” that will decrease the suspension score.

What ever you do you can only decrease one of the scores. Offline doesn’t decrease audit and suspension score.

2 Likes

Thanks!
And how can I know my online score? I’m asking because it seems that DQ for downtime will be enabled soon or later.

1 Like

http://localhost:14002/api/sno/satellite/1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE
http://localhost:14002/api/sno/satellite/121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6
and so on
Look at onlineScore

2 Likes

Thanks.
Online score is 1 on all my nodes. Is that normal after 72h of downtime?
Maybe because DQ for downtime is not enabled yet?

Or it doesn’t work as expected…
We will see in the next version, it should be available on the dashboard:

1 Like

Hmm I had half a day outage since the inet company was digging around the street cabinet:


Two days later I saw the impact on the dashboard. What confuses me is

  • Why is asia-east more impacted
  • Why does the value not change since more as a week of 100% online already since the downtime

is there a more detailed desciption how those values should work?

I think the downtime calculation is described on these 2 posts below.

In a nutshell, what I understand is:

  • the downtime score is based on audits (i.e. audits requests sent by the satellites to the node)
  • the node may receive a varying number of audits from satellites
  • then, for a given node, the score may be different depending on the satellite we are considering.

Please correct me or complete if I’m wrong or missing something. :slight_smile:
(would be really nice to have a topic on this on https://support.storj.io/)


Thx and the critical watermark is 93%? Assumption based on 94% showing in yellow.
And it can go up only after 30days again? This is time after incident not a calendar month I assume?

I don’t know the thresholds but from what I’ve been told, it should be up back again in a few days if you keep your node up and running.

The uptime score is a moving window of 30 days. So as long as your day with the downtime is within that window, the uptime score won’t go up.

E.g. if the window would be 3 days and you have a downtime on the last day:

  1. 100%
  2. 100%
  3. 50%

So uptime score would be 83%. Now on the next day it would look like this:

  1. 100%
  2. 50%
  3. 100%

So the average is still 83% as long as that day with 50% is within that window. Only once its outside the window the score will go up again.

4 Likes