Test results are a step closer to our target and this time we can keep it running for hours without any errors on the other satellites. That is good. Our target gets more and more in reach.
We are working on a different success tracker. Deployment on a Friday is a bad idea so it will have to wait for Monday. There is a good description in the PR what the differences are: https://review.dev.storj.io/c/storj/storj/+/13308
To me it looks like we might have a problem with the number of connections now. If I run a single node I have almost 100% success rate. If I add a second node on the same IP they split the load and still almost 100% success rate. I can continue like that up until 4 nodes. If I add more nodes they start to get long tail canceled. My reading of this is that TCP fast open or connection pooling gives me 100% success rate for a single node but the more nodes I add the higher the chance that a new connection needs to be established and that makes me lose the race. Not much so I could as well just ignore it but it is visible on my grafana dashboard.