Uptime and suspension

Node Operators FAQ
1

Hi,
just had some fat finger induced downtime of 48h. And uptime went down to 94%. But was kinda surprised that the node was not disqualified, as I had 95% as disqualification limit in mind.
But then I read

I cannot find that anywhere on the SNO documentation. Is this the rule now? Below 60% uptime you get suspended, but not disqualified? No other consequences?

Just for completeness’ sake:

#./check-online-score.sh
{
  "id": "12tRQrMTWUWwzwGh18i7Fqs67kmdhH9t6aToeiwbo5mfS2rUmo",
  "auditHistory": [
    {
      "windowStart": "2021-09-25T00:00:00Z",
      "totalCount": 32,
      "onlineCount": 31
    },
    {
      "windowStart": "2021-09-26T00:00:00Z",
      "totalCount": 13,
      "onlineCount": 9
    },
    {
      "windowStart": "2021-09-26T12:00:00Z",
      "totalCount": 12,
      "onlineCount": 0
    },
    {
      "windowStart": "2021-09-27T00:00:00Z",
      "totalCount": 21,
      "onlineCount": 0
    },
    {
      "windowStart": "2021-09-27T12:00:00Z",
      "totalCount": 17,
      "onlineCount": 0
    },
    {
      "windowStart": "2021-09-28T00:00:00Z",
      "totalCount": 7,
      "onlineCount": 0
    }
  ]
}
{
  "id": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE",
  "auditHistory": [
    {
      "windowStart": "2021-09-26T00:00:00Z",
      "totalCount": 67,
      "onlineCount": 35
    },
    {
      "windowStart": "2021-09-26T12:00:00Z",
      "totalCount": 65,
      "onlineCount": 0
    },
    {
      "windowStart": "2021-09-27T00:00:00Z",
      "totalCount": 58,
      "onlineCount": 0
    },
    {
      "windowStart": "2021-09-27T12:00:00Z",
      "totalCount": 41,
      "onlineCount": 0
    },
    {
      "windowStart": "2021-09-28T00:00:00Z",
      "totalCount": 14,
      "onlineCount": 0
    }
  ]
}
{
  "id": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6",
  "auditHistory": [
    {
      "windowStart": "2021-09-26T00:00:00Z",
      "totalCount": 42,
      "onlineCount": 22
    },
    {
      "windowStart": "2021-09-26T12:00:00Z",
      "totalCount": 49,
      "onlineCount": 0
    },
    {
      "windowStart": "2021-09-27T00:00:00Z",
      "totalCount": 47,
      "onlineCount": 0
    },
    {
      "windowStart": "2021-09-27T12:00:00Z",
      "totalCount": 23,
      "onlineCount": 0
    },
    {
      "windowStart": "2021-09-28T00:00:00Z",
      "totalCount": 9,
      "onlineCount": 1
    }
  ]
}
{
  "id": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S",
  "auditHistory": [
    {
      "windowStart": "2021-09-26T00:00:00Z",
      "totalCount": 16,
      "onlineCount": 9
    },
    {
      "windowStart": "2021-09-26T12:00:00Z",
      "totalCount": 29,
      "onlineCount": 0
    },
    {
      "windowStart": "2021-09-27T00:00:00Z",
      "totalCount": 21,
      "onlineCount": 0
    },
    {
      "windowStart": "2021-09-27T12:00:00Z",
      "totalCount": 14,
      "onlineCount": 0
    },
    {
      "windowStart": "2021-09-28T00:00:00Z",
      "totalCount": 1,
      "onlineCount": 0
    }
  ]
}
{
  "id": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs",
  "auditHistory": [
    {
      "windowStart": "2021-09-06T00:00:00Z",
      "totalCount": 41,
      "onlineCount": 40
    },
    {
      "windowStart": "2021-09-26T00:00:00Z",
      "totalCount": 41,
      "onlineCount": 24
    },
    {
      "windowStart": "2021-09-26T12:00:00Z",
      "totalCount": 43,
      "onlineCount": 0
    },
    {
      "windowStart": "2021-09-27T00:00:00Z",
      "totalCount": 46,
      "onlineCount": 0
    },
    {
      "windowStart": "2021-09-27T12:00:00Z",
      "totalCount": 23,
      "onlineCount": 0
    },
    {
      "windowStart": "2021-09-28T00:00:00Z",
      "totalCount": 10,
      "onlineCount": 0
    }
  ]
}
{
  "id": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB",
  "auditHistory": [
    {
      "windowStart": "2021-08-30T00:00:00Z",
      "totalCount": 72,
      "onlineCount": 71
    },
    {
      "windowStart": "2021-09-26T00:00:00Z",
      "totalCount": 53,
      "onlineCount": 26
    },
    {
      "windowStart": "2021-09-26T12:00:00Z",
      "totalCount": 69,
      "onlineCount": 0
    },
    {
      "windowStart": "2021-09-27T00:00:00Z",
      "totalCount": 51,
      "onlineCount": 0
    },
    {
      "windowStart": "2021-09-27T12:00:00Z",
      "totalCount": 42,
      "onlineCount": 0
    },
    {
      "windowStart": "2021-09-28T00:00:00Z",
      "totalCount": 9,
      "onlineCount": 0
    }
  ]
}
2

If the online score fall below 60%, then your node would be suspended (no ingress traffic) until the online score would recover above 60%.
The disqualification for downtime could happen if your node managed to be suspended again during 30 days review after the last recover or if it was offline more than 30 days.
To fully recover your node need to be online for the next 30 days. Each downtime event extend the recover period to the next 30 days online. The review period would not be extended though, so the node has a high risk to be disqualified if it managed to fall offline again during the review period.

1 Like
3

For a more detailed (and technical) description you can check: storj/storage-node-downtime-tracking-with-audits.md at 40084d51a14069b1105fcae9c3d0d4cd3b78633b · storj/storj · GitHub

It’s implemented with an offline threshold of 60%, a grace period of 7 days and tracking period of 30 days. I believe the window size is either 12 or 24 hours, but I’m not 100% sure on that.