Updates on Test Data

Ok while you are removing any concurrency limit you might have set I will explain how the node selection actually works.

When a segment gets commited to the database it will contain the nodes that have been fast enough and it will be missing the nodes that got long tail canceld. The satellite calculates a success rate for each node with that.

The node selection takes that success rate. Instead of 110 total nodes it selectes 220 nodes at first and compares them in pairs and pick the one with the higher success rate. So it throws away the slow nodes. We call this the power of 2 node selection. The benefit of this method is that the risk for a stomping herd effect is low. This effect means the selection of a group of nodes that has a high success rate but because the node selection overshoots the goal the nodes gets selected too many times and success rate decreases. At the same time another group of nodes recovers and is now the new vicitm of a node selection with stomping herd effect. Power of 2 choice is build to minimize that risk.

10 Likes