Multi nodes behind same /24, when one fills will the data to the others increase?

[Craig]

Probably pretty spelled out in the subject, but I’m curious if anyone knows if when you have multiple nodes that are treated as “one” behind the same /24 subnet, if as the individuals fill up does the traffic of new storage shift over to the remaining at the same total rate? In my current situation, I’ve got 3 nodes (2TB/2TB/1TB) and 2 are already full. For simplicity sake lets ignore racing for pieces/vetting status and assume that if my IP got 3 storage requests that they would have been split across the 3 nodes. Now that 2 nodes are full, if 3 requests were to be sent to my IP do the satellites know that two are full and would request all 3 from the 3rd node?

In short, trying to figure out if there are diminishing returns to having multiple smaller nodes (because that’s what I’ve got available for drives) vs. having a larger node. In theory I’m better covered with the multiple smaller drives as the most likely problem of a single drive failure won’t rob me of as much stored data, although I could still have a catastrophic event that could strike all 3 drives simultaneously. But does this spread protection come with a later cost of slower incoming storage requests once nodes start to fill up?

[KernelPanick]

My strategy is to keep my node size as small as my largest single HDD. This way i don’t get stuck on a large storage array.

I’m curious about this as well, but i think the answer is that they all round robin and fill at the same rate. I would wait for someone more official to chime in though before going with that for sure.

[Pac]

@Craig

Now that 2 nodes are full, if 3 requests were to be sent to my IP do the satellites know that two are full and would request all 3 from the 3rd node?

Satellites do know that 2 of your 3 nodes are full indeed. In fact, they know what nodes are full and which ones have available space within a /24 subnet.

So yes, I’m pretty sure if 3 requests were to be sent to your /24 subnet, they would all go to your only node left with available space (assuming they are ingress requests).

@Craig

But does this spread protection come with a later cost of slower incoming storage requests once nodes start to fill up?

I don’t think so. The only problem I can think of when only one node is left with free space amongst many others behind the same /24 subnet is that it may not keep up in case of high network ingress activity, especially if you’re using SMR drives.
The more nodes with available space you have, the less stressed they are as the load gets spread accross all of them.