Disqualification due to downtime?

silaxe · December 13, 2021, 1:45pm

Hi all,
I have just moved places and I will probably have to wait a few weeks before I get a good internet connection (the apartment is not yet wired with coax or fiber). In the meantime I am using a 4g modem which does not allow me to do any kind of port forwarding.
I lost track of the consequences due to downtime for node operators. Is there a maximum amount of downtime after which your node is disqualified? What are the risks I incur more generally?
I think that the docs would deserve a section on common questions encountered during node operation, including how uptime/audits work.
Thanks!

SGC · December 13, 2021, 6:45pm

github.com

storj/storj/blob/c2a97aeb143791dd7edd8bea5bb43558a95b57de/docs/blueprints/storage-node-downtime-tracking-with-audits.md

# Storage Node Downtime Tracking With Audits

## Abstract

This document describes a means of tracking storage node downtime with audits and using this information to suspend and disqualify.

## Background

The previous implementation of uptime reputation consisted of a ratio of online audits to offline audits. We encountered a problem where some nodes' reputations would quickly become destroyed over a relatively short period of downtime due to the frequency of auditing any particular node being directly correlated with the number of pieces it holds. To solve this problem we need a system that takes into account not only how many offline audits occur, but _when_ they occur as well.

## Design

The solution proposed here is to use a series of sliding windows to indicate a general timeframe in which offline audits occur. Each window keeps two separate tallies indicating how many offline audits and total audits a particular node received within its timeframe. Once a window is complete, it is scored by calculating the percentage of total audits for which it was offline. We can average these scores over a trailing period of time, called the _tracking period_, to determine an overall "offline score" to be used for suspension and disqualification. By granting each individual window the same weight in the calculation of the overall average, the effect of any particularly unlucky period can be minimized while still allowing us to take the failures into account over a longer period.

Storage node downtime can have a range of causes. For those storage node operators who may have fallen victim to a temporary issue, we want to give them a chance to diagnose and fix it before disqualifying them for good. For this reason, we are introducing suspension as a component of disqualification.

Once a node's offline score has risen above an _offline threshold_, it is _suspended_ and enters a period of review. A suspended node will not receive any new pieces, but can continue to receive download and audit requests for the pieces it currently holds. However, its pieces are considered to be unhealthy. We repair a segment if it contains too many unhealthy pieces, at which point we may transfer the repaired pieces from a suspended node to a more reliable node. If at any point during the review period we find that a node's score has fallen below the offline threshold, it is unsuspended, or _reinstated_, but it remains _under review_. This prevents nodes from alternating between suspension and reinstatement without consequence.

The review period consists of one _grace period_ and one _tracking period_. The _grace period_ is given to fix whatever issue is causing the downtime. After the grace period has expired, any offline audits will fall within the scope of the tracking period, and thus will be used in the node's final evaluation. If at the end of the review period, the node is still suspended, it is disqualified. Otherwise, the node is no longer _under review_.

This file has been truncated. show original

anything less than 30 days downtime is fine, above that i’m unsure but i think there is a point at or above 30 days where a node would be DQ for downtime.
so i would try to get in under the 30 day mark, if at all possible… also keep in mind that your node’s data will degrade over time the longer the node is offline.

usually not to bad but it still happens.

pius1212 · December 29, 2021, 6:16pm

Can you reiterate this? I can’t seem to find a solid source of information, as multiple sources are reporting having not been suspended after 12+ hours of downtime.

Is the 30-day downtime for the lifespan of the node? The topic of downtime punishment is still quite confusing to me…

SGC · December 29, 2021, 6:48pm

downtime or online score is based upon a rolling 30 day avg of uptime, monitored by the audits data gets.

for a node to reach 60% online score it will need to be offline for a full 12 days, equal to 288 hours.
at 60% online score a node is suspended, suspension means it will not get any ingress until the suspension is lifted, which will happen either when the node is above 60% or when it stops being offline (i think its the former, but i could be mistaken)

nodes that has had over 30 days of downtime has been disqualified, tho i’m unsure if that is actually active and wasn’t just a one time clean up event to get rid of offline network nodes that never came back to the network.

it’s very difficult to get DQ due to downtime, if even possible currently…

i don’t think there is kept track of lifetime downtime of nodes, it will all work over the 30 day avg online score…

the online score is a bit more complex than just being the avg, which is describe in the blueprint linked above in my previous post.

the easiest way to think about it is that the online score is the % of time online a node has been over the last 30 days and 12 days offline is the 60% mark, which one should avoid…

lets say your node is offline for 3 days, so that is 1/4 of the 12 days and 1/4 of the 40% which the 12 days represent, so the node would after 3 days of downtime have an online score of 90%.
when the node then comes back online, it will take 27 days until the 3 days of downtime start pass out of the rolling 30 day avg online time.

no idea where you got the 12 hour from

pius1212 · December 29, 2021, 6:54pm

Thanks, this helps plenty!
(btw is there a commend type thing in this forum)

SGC · December 29, 2021, 7:05pm

i don’t think so, but i appreciate the sentiment