Loop stacktrace discovered in 4MB line in storagenode logs that resulted in a crashed node

[Alexey]

@loueradun

timed out after 1m0s while verifying writability of storage directory

yes, the same error as before. Did you try to optimize the filesystem?

@loueradun

It seems like a monitor for failing uploads/downloads which decreases the connections until things settle would be appropriate IMHO, but to be fair, I’m not intricately familiar with how the jobs are executed.

This is already implemented in the node selection:

Updates on Test Data

Ok while you are removing any concurrency limit you might have set I will explain how the node selection actually works. When a segment gets commited to the database it will contain the nodes that have been fast enough and it will be missing the nodes that got long tail canceld. The satellite calculates a success rate for each node with that. The node selection takes that success rate. Instead of 110 total nodes it selectes 220 nodes at first and compares them in pairs and pick the one with th…