I’d like to propose an improvement to the operation of Storj nodes: the ability for node operators to activate a temporary maintenance mode that allows planned downtime without being penalized.
Why is this needed?
Currently, any temporary disconnection of a node (due to system updates, hardware upgrades, power issues, or physical relocation) can reduce uptime scores and even lead to suspension or disqualification if it happens frequently. This is particularly challenging for home-based or semi-professional setups, where occasional maintenance is unavoidable.
What am I proposing?
A system where node operators can notify the satellite in advance that they will temporarily take the node offline for maintenance. This would:
Allow for safe updates, restarts, or hardware replacements without impacting node reputation.
Ensure network stability by allowing satellites to proactively replicate data if necessary.
Encourage more professional node management with predictable downtime.
How it could work
Requesting maintenance mode
The operator runs a CLI command such as:
./storagenode maintenance start --duration=2h
The node sends an authenticated request to the satellite, indicating the estimated duration and optional reason.
Satellite evaluation and approval
The satellite checks:
Node reputation and audit score.
Data redundancy status.
Whether other nodes in the same region are already in maintenance.
If conditions are met, the request is approved.
Protected period
While in maintenance mode:
The node is exempt from uptime and audit penalties.
No new data is assigned to the node.
Optionally, the node could switch to read-only mode before going offline to minimize risk.
Ending maintenance
Once the node is back online, the operator runs:
./storagenode maintenance stop
The satellite resumes normal operations (audits, pings, etc.).
Rules and limitations to prevent abuse
Maintenance time limits (e.g., maximum 24 hours per month).
Only available to nodes with good standing.
If the node fails to return online within the approved time, normal penalties apply.
Benefits for the network
Fewer unexpected or uncontrolled disconnections.
Reduced risk of unjust suspensions due to minor technical issues.
Better uptime planning by responsible operators.
Increased reliability and predictability across the network.
What do you think? Could this be considered in a future protocol update?
If there’s community interest, I’d be happy to help draft a more technical proposal or collaborate with anyone involved in node development.
Right now I think nodes can have somewhere around 60% uptime before they stop receiving data and somewhere around 30% before they are disqualified.
These numbers are not exact but the point is that these boundaries are very generous and 24hr downtime is no big deal. The network is designed to account for this.
Also as an aside I have given you the benefit of the doubt that despite this being your first post on the forum and the post clearly being LLM generated you are posting in good faith and are not just a bot. Others may not be so kind and your words will have more ears if you are more authentic.
A node needs to have a 99.3% uptime. That’s about 5 hours of downtime in a month.
Assuming that an update needs a simple service restart, that’s about 10 seconds of downtime per update.
If a normal, fully functioning node just needs updates, that’s (5x60x10)/10=300 updates per month. Normally it’s just one update per week, so normal operation does not affect the node’s scores in any (significant) way.
Even if the host system needs to be restarted to finish an OS update, that’s about 10 minutes of downtime.
Nodes running on barely functioning hardware, that constantly needs restarting/tinkering to work, are a danger to the network and customer’s data. Any node that drops below 99.3% uptime in a month should be kept suspended until it can climb up back to 99.3% uptime (month=rolling 30 day period).
Any nodes falling below a certain percentage (ie 50%) should be disqualified, no questions asked, no excuses provided. A node that isn’t available half of the time isn’t any good addition to the network, it’s a burden on the network.
Plus we already have 15,000 (and this number is counting upwards with each passing day) more nodes than actually needed, so actually implementing disqualifications based on ToS (that should be applying anyways) should help the rest of the nodes get the data they are supposed to be getting.
I understand the importance of a temporary maintenance mode for Storj nodes, especially for home-based or semi-professional operators. Being able to notify the satellite about planned downtime, for example, for a hard drive replacement, without penalties is fair and helps avoid suspensions due to brief disconnections.
The satellite’s prior evaluation based on reputation and redundancy, along with limits to prevent abuse, ensures network stability.
I believe this feature, combined with automatic backups of the database and credentials, would greatly improve the experience of operating nodes and the resilience of the network.
Not really, no you don’t. The importance you need to understand is that in a system that needs to adhere to an SLA, the “temporary maintenance mode” is the SLA.
The SLA (Service Level Agreement) states that your system needs to be available for 99.3% of the time. I cannot claim a 5 minute fiber splice “fix” is all it took for a dedicated internet line that got excavated by mistake. The SLA applies from the moment the fiber was cut, until the fiber was returned to its normal operating state. The same applies for nodes as well.
Your disk replacement isn’t a “temporary maintenance mode”: It’s removing a critical part of the network (your node’s stored data) for some time. If you messed up and didn’t check the new drive before putting it in, and that drive fails within the first week, your node signaled to the network “hey, don’t reduce my scores, I’m going into maintenance”, then “oops, sorry, all data is lost”. Between those two points, data could have been rebuilt, if the network needed to.
Again, just to make sure you truly:
IN ANY SLA-BACKED SYSTEM, THERE ISN’T ANY TEMPORARY MAINTENANCE MODE.
I understand that in strict SLA-backed systems, availability is non-negotiable. However, Storj is a decentralized network where most node operators are not running enterprise-grade infrastructure. Many of us contribute from home or semi-professional environments, where occasional downtime is simply unavoidable.
The proposed temporary maintenance mode isn’t about avoiding responsibility — it’s about allowing responsible operators to notify the network of planned downtime (like a disk replacement) without being unfairly penalized. It would only be available under clear conditions: good reputation, strict time limits, and penalties if the deadline is not met.
T his isn’t an excuse for failures, but a coordination tool for controlled, necessary maintenance tasks.
You can replace a disk while the node is up and running by rsyncing to the new disk. This is a process that needs to happen once per node’s lifetime: migrate the node from a temporary disk to its actual disk.
Any other scenario is “node’s data is at risk scenario”. Your failed disk puts the entirety of the data at risk. Your corrupted filesystem puts the entirety of the data at risk. Your failing disk controller puts the entirety of the data at risk. Your DSL line that the last splice point is full of water (shorting the line to every other DSL line in your neighborhood) is putting the data at risk.
Mitsos, the idea is to take care of the hardware and the time we all invest in this. Most of us who run nodes at home don’t have spare drives ready, and when one starts failing, we have to buy a new one and wait for it to arrive. During that time, the node has to be offline to avoid possible complications.
This isn’t to excuse failures, but so the network knows it’s maintenance and doesn’t penalize us for something we can’t avoid. It’s about protecting our reputation and the time, money, and effort we invest, as well as the network itself.
OK then, we’ll point the customer to you when they ask why they can’t access their data. I’m assuming your reply will be “We are waiting for a replacement drive to be shipped, we don’t know yet if the data is completely lost because we did the responsible thing and took the failing disk offline to prevent further damage to the data. Hopefully when the new drive gets here, we can piece together your data so you can download it. Would you like a coffee while you wait?”
Please feel free to correct my assumption if it’s wrong in any way.
I completely understand the concern no one wants a customer to lose access to their data. But that’s exactly why I’m suggesting something like a controlled maintenance mode with clear rules and limits.
The idea isn’t to let a node go offline and just “see what happens,” but rather to allow a responsible operator, who detects an imminent hardware failure, to notify the system before shutting down the node. That way, the satellites have time to take preventive action like replicating data that hasn’t yet reached the redundancy threshold.
This isn’t about avoiding consequences when data is lost. it’s about acting before that happens. Right now, if someone does the responsible thing and powers down the node to avoid further damage, they’re penalized just like someone who ignores the issue or abandons the node. That doesn’t seem fair or efficient for anyone.
A system like this would help protect the data, safeguard the customer, and avoid punishing operators who are genuinely trying to do the right thing.
Alright, let’s break this down. As you’ve already seen from other responses, there is plenty of slack built into the SLA’s to perform maintenance any time you want. So what would be the remaining upside to letting satellites know your node is going to be down?
It seems to come down to this part. Now… first of all, data isn’t replicated, it’s erasure encoded. Which is much more reliable, but it’s not a free process. So you would incur additional costs if you expect the satellites to take your downtime into account. Instead, the satellites just ensure enough redundancy to be able to work with nodes that generally follow the SLA. They have to do that anyway. There is also literally no guarantee that your announced downtime will be accurate. So even if the satellite would prepare for 2 hours of your down time. That additional investment still may not have been enough.
So instead the current implementation allows you to do maintenance whenever as long as it never exceeds the allowed downtime in the SLA. No need to announce anything. Redundancy is built in already to account for unexpected downtime.
If regular people (not datacenters with on-call staff) are the ones supposed to run the nodes (at least in the public network), then this is extremely strict. I don’t think any ISP even gives a 99.3% uptime guarantee for a residential line (even for a business line it’s usually worse).
This is beside the maintenance mode, but since the node software does not natively support running in a cluster (other than running the node VM in a cluster) even something like shutting down the server to clean the dust out may take a lot of the “allowed” downtime.
I have two internet connections, multiple UPSs, a generator and two ACs (it probably meets the basic redundancy requirements for a tier 2 datacenter at least as far as the hardware is concerned), but I am not always at home and not always awake and do not have employees to look after my servers when I am not around. 5 hours is unrealistic.
The storj service remains active even if many nodes suddenly go down. A node being offline does not mean the service is unavailable to the customer. The SLA applies to the final service delivered to the customer.
However, I understand your concerns about the constant growth of nodes.
@Pentium100 : You don’t need 5 hours to dust out the servers. You don’t even need 5 hours to disassemble and reassemble the servers. In that one case where you are away and fall below the 99.3% uptime, then that’s where the suspension comes in. This will happen to you once maybe twice in a node’s lifetime.
@agente : No, 99.3% is in the SNO terms, not the end product. Nobody on this planet would choose a storage service with a 99.3% user SLA.
Howver, I do not have employees on-site 24/7 if something goes wrong. Hell, I know one datacenter , where if you need to acess your stuff without prior notice (you know, like an emergency) specially outside working hours, you may have to wait some hours before an employee arrivs and lets you in. Though in that case if something happens in the middle of the night, not many people notice.
Also, my ISP does not give me that good SLA, probably most ISPs do not give residential customers 99% SLA, so it’s one major ISP outage away from bad things.
I understand 99.3% or even better SLA when the provider is a datacenter and the application can be run in a cluster and has backups. Otherwise it’s one blown up motherboard away (if the provider does not have a spare or if it happens outside working hours) from violating the SLA.
Honestly, a lot of corporate clients I know are less strict. They get really angry if the server is down even for a few minutes during working hours, but don’t really care if the server is down for a few hours at night as they are not using it.
I think more normal SLA would be something like 24 hours/month. The network can handle it and 24 hours would be enough to wake up and fix whatever problem there is (if you have enough spare parts).
I don’t think 5 hours of downtime per month is anything unreasonable. There isn’t any requirement in the ToS that says that every computer on this planet must run a storj node. In plain English: there are certain requirements that SNOs must adhere to. If someone can’t adhere to those requirements, the requirements shouldn’t be changed. It simply means that that particular SNO can’t be a part of the network.
I’m running two independent connections (different ISPs, different technology, different physical path). I also have a cellular backup connection. As far as my ISP’s SLA is concerned, I don’t care if one of the links goes down for an entire week. I’m using the extremely loose SLA provided by each connection and playing the probability game: chances are, whatever happens to my connections, there will be at least 1 available at any time. Can they all go down? Of course they can, but there is (a mathematically proven) greater chance of me having a heart attack and not wake up tomorrow.
The exact same logic applies to the 99.3% required uptime from the nodes: A lot of unreliable nodes have a higher chance of hitting an acceptable end-user-SLA. Will some of those nodes be offline? Of course, but this is the whole point of the topic. My nodes will be there to pick up the slack, if the network signals to me that I need to step in. If the network doesn’t know that the nodes are unavailable, then it can’t signal to me to step in. If the network thinks for example that the unavailable nodes are “in maintenance mode” then it considers that data still safe. The network seeing a node being offline should mean that the node has dropped off the network. If it doesn’t come back within a reasonable time, then assume that it is gone and start healing the data.
This isn’t a discussion as to if a home internet connection is reliable enough to run a node, or if the SNO uses redundant routers/switches/infrastructure. Or if the SNO can step in and fix things. This is a discussion of “we have these things available to us, what can we do to bring the end-user-SLA to an acceptable level”. My redundant connections + my monitoring and alerting system is more than enough for me to do my best to adhere to the required node-SLA: If I slip up and not wake up when I get an alert, that’s completely my fault.
Two months ago, we had issues with fuse board, after a month of random fuse blowing, it eventually caught fire. Causing a 14 hour outage before emergency repairs were made.. Six days later, another 8 hour outage when fuse board replaced.
With the several small outages in the weeks prior to the fire, and the two long outages, my nodes fell to 96.2. Still receiving and sending data and repairs. At no point did they get suspended.
Over the last two months, they have slowly climbed back to 100%.
Exactly. You saw something being abnormal (the first fuse blew). Then you did the fast thing and replaced that fuse. The second fuse blew as well. “Hm, I must be drawing slightly more current than the fuse can take. Why not replace the fuse with a bigger fuse?!?! Problem solved!.” This is where the wiring caught fire, since it wasn’t rated for that current draw.
There was an alert, there was a fix. Was the fix the wrong fix? Absolutely. You responded to an abnormal situation to the best of your abilities.
Some SNOs would be running A-B power feeds. It doesn’t matter if the panel takes a month to repair, you still have the B feed to rely on. Can that go down as well? Probably.
This is where the process needs to change, to adhere to the ToS: 99.3% is the required uptime. If a node falls below that (rolling 30 days), then it needs to be suspended until the uptime comes back up to an acceptable level. It’s not a “blacklisting” of the node or any retaliatory measures (as others may have suggested): It’s simply making sure that the node is ok to start receiving data again.
Correct. The nodes have shown that it was something temporary that they recovered from. It didn’t have any significant impact on the data (sure some data got corupted when power went out, a tiny fraction of the total stored data on that node). That node can be part of the network again. It’s not an SMR disk connected with a handmade USB cable to a raspberrypi.
Disclaimer: This post contains assumptions made to prove a point. Any statement is not a personal attack. Reader discretion is advised.
Why is this needed?
What am I proposing?
How it could work
Rules and limitations to prevent abuse
Benefits for the network