Hi there.
What is your above WHEN to start with a new node, when first HDD gets full?
Waiting until it’s completely full? Some months before, because of vetting? Waiting until ingress is significantly dropping, because HDD gets full?
Thx
Hi,
I usually start new node when old have 100GB FREE
1 Like
Then vetting starts from scratch and income increase slows down for months, right?
Personally, I really don’t wait for a node to be full before starting a new one.
I’d rather slow down the first node for some months than missing out on potential ingress 
No sure what you mean by “income increase slows down for months”. In fact, with a second node, income should increase slightly faster during its vetting phase, in theory.
But we’re talking about so small increases that it does not matter.
More info on how the vetting process works:
Anyway, considering vetting a node can take several months, I tend to start a new node when I feel like the previous node is going to be full in the coming 6 months or so, roughly.
The result is usually something like this:
- First node keeps filling up at full-rate while the second node gets vetted. During that time, the first node is not supposed to be (edit) significantly (/edit) impacted by your second node.
- Second node is fully vetted: At this stage, both nodes are going to share ingress, which means that your first node’s filling is going to slow down substantially, your second node is going to fill up at the same rate as your fist one.
- At some point, you first node is going to be full. At that point, your second node is going to have all the ingress for itself, so it’s going to fill up at full-rate.
3 Likes
I can provide some clarifications.
By my last reading of satellite’s source code, the first node will be slightly impacted. For each upload, first the unvetted nodes are selected to take 5% of node slots. Then, all remaining slots are filled by vetted nodes (at least 95%, but if it would happen that there were not enough unvetted nodes, then this number might be larger). If an unvetted node from a /24 block is chosen, the vetted node from the same block will not be selected.
The sum of ingress of both nodes will be higher compared to just having a vetted node. But some traffic that would normally go to the vetted node will instead go to the unvetted node.
Exact ratios depend on the ratio of /24 blocks with vetted and unvetted nodes, I’ve made a simple simulator to play with numbers. I’m recently observing that this ratio is quite skewed towards vetted nodes, though.
I found it a good idea to stop the ingress for the first node at this point, so that all traffic is handled by one node at a time. You can do so by reducing the allocated disk space below the amount already stored. While this action won’t give change your ingress, it may result in less I/O (one set of databases to be updated).
On the other hand, if your /24 block also has someone else’s nodes, keeping both nodes open will give you more ingress, as it will be 2 your nodes vs. 1 other 
1 Like
Why would you want to do that? 
Having 2 concurrent nodes make both disks work less (I’m assuming each node has its own disk, which is Sotrj’s recommendation).
Besides, we still want the first node to completely fill up, right? 
But the sum of I/O of both drives is bigger. The difference is minimal, but I still observe it on my setup.
Sure, at some point. It’s not that you can’t change the allocation back to the full size when the second node fills, or rotate between them, let say, once a month.
I guess, but I really don’t see why that would be a problem
But I mean… Everyone’s mileage may vary
Absolutely. Talking about I/O though, reconfiguring a node hits its hard drive pretty hard with the filewalker… Although I believe it’s configurable now, I’ll have to look that up and see if I shouldn’t reconfigure my nodes… But that’s another story ^^
Anyway, thx @Toyoo for the additional info
Yeah, here’s the thing—somehow my nodes aren’t as impacted as other people’s. I know some operators report that their file walker takes >24h. Right now for my nodes it’s about 8 minutes on average per TB of stored data, which is really not that much.
What do you mean with “slowing the first node down”? Is that something you can do?!
You can kinda slow down a node’s ingress only with an option that is not recommended unless you really need it (storage2.max-concurrent-requests).
See:
Otherwise, the only possible way to restrain network activity of a node is by using external software restricting network flow for instance… which is probably something you shouldn’t do anyway.
What I meant is that if you start a second node too early (I assumed on the same /24 subnet), it will eventually get fully vetted and functional although the first one isn’t full yet.
At this point, both nodes will share ingress, which means that they will both fill up at half speed.
So the first node gets slowed down, even though the total ingress collected by both nodes should globally stay the same (i.e. as if you had only one node).
When the first node gets full, the second node will then receive ingress at full speed.
Side note: Egress isn’t impacted by the number of nodes behind a /24 subnet.
1 Like
That’s really valuable information, which I wished I had a few months ago 
1 Like
Vetting on the most important satellites takes about 6-8 weeks at the moment. This is for the customer facing satellites which generate the most ingress. The other take more than 6 months currently, but also don’t account for much Ingress anymore anyway.
I recommend starting the new node at least 2 months before the previous one fills up. Or 6+ months if you want to maximize profit.
Keep in mind, running an extra disk uses power… You can safe some costs by postponing that. On the other hand, having more than one node spreads risks, making it less frustrating if one node fails. So in the end it’s up to you which you prefer based on that.
The impact of this is exceptionally small. This only applies if your node was selected for the same segment as the unvetted node. So were talking about 0.1%-0.3% or something.
3 Likes
I’ve observed the effect to be bigger than that. I specifically measured this in June when I got a new drive. I’ve set up a new node and counted the number of ingress requests on both nodes every hour, while switching the unvetted node offline periodically. When both nodes were operating at the same time, the unvetted node had around a quarter of upload requests of the vetted node, and this ratio was quite stable over a week. When the unvetted node was disabled, I had small, but visible increase in the number of requests of the vetted nodes; hard to put firm numbers here, but the 95% confidence interval for the increase was around 0.5%-5% comparing 15-minute intervals next to each other. Not something one would recognize from a graph, but the effect is clearly there.
I was both surprised and satisfied observing these numbers. Surprised, because I didn’t expect the effect to be that visible. Satisfied, as the calculator I made almost two years ago was predicting this effect correctly ^^