Some nodes are running into problems where a high amount of traffic runs into a bottleneck. This bottleneck could be related to disk IO, like SMR disks running into their write limitations, slow and older disks or USB2 disks. This could lead to the node hardware becoming unbearably slow and unresponsive at times. It also has an impact on success rates with nodes failing more transfers than average. This bottleneck could also occur in relation to internet connection speed. In which case it could cause their internet grinding to a halt for all devices on the network.
Currently many nodes use the max-concurrent-requests setting to work around these issues, despite advise to the contrary. They use this setting because it’s the only way for them to be able to solve these problems other than stopping their node altogether. However, this current setting gets applied too late in the process. The node has already been selected for upload and the satellite and uplink count on it that enough of the selected nodes finish that upload to be successful. If those thresholds aren’t reached customers (uplinks) could run into errors and uploads might fail.
Failed to copy: uplink: segment error: ecclient error: successful puts (79) less than success threshold (80) Failed to copy: uplink: segment error: ecclient error: successful puts (29) less than or equal to repair threshold (35)
Limiting transfers based on node selection
Instead of the nodes rejecting transfers they are already selected for, a much better approach would be to not select busy nodes for transfers to begin with. There are several ways to go about this. I’ll outline some options here.
- When a node rejects a transfer, track those rejections and make that node less likely to be selected in the future. This has the big downside that these nodes also get selected less often during quiet times, which isn’t ideal. Additionally the rejection, which is currently part of the exchange between uplink and storagenode, needs to be reported to the satellite. The upside is that you could keep using the current setting in exactly the same way.
- Don’t select nodes when their max-concurrent-requests limit is reached. This solution would be ideal, as it would use the most current data to limit requests. From the SNO point of view the impact would be very similar to what this setting does now. But there would have to be constant communication between the node and satellite to update the number of active requests. This may run into scalability issues.
- Add settings for maximum number of uploads per minute and maximum number of downloads per minute. These settings are then advertised to the satellites. Whenever a node is selected for a transfer, that gets logged in the satellite’s DB. And whenever nodes need to be selected for transfer, nodes that have reached their maximum threshold for the past minute will be excluded from selection. The downside is that this doesn’t exactly limit the number of concurrent requests, but the number of started requests. Additionally, this would be a limit per satellite. If the number of active satellites changes a lot, it might be needed for SNOs to adjust this setting accordingly. The upload rate limit would be the most important one, because you ideally don’t want to limit downloads as that directly impacts how much your node makes. But I would still implement a download limit as well to account for SNOs with slow upload connections. After all, upload speed is often drastically lower than download speed.
@Pentium100: If a node is overloaded (set by SNO with concurrent request limit or CPU iowait limit or unix oad limit) the node contacts each satellite and informs it that it is overloaded. The satellite then reduces the performance coefficient by some amount.
When the load drops below a lower limit and stays there for 10 minutes, the node contacts the satellite and informs it that it is almost idle. The satellite increases the performance coefficient by some amount that is smaller than the decrease.
The coefficient is used when selecting a node. If a node would be selected, a random value between 0 and 1 is generated. If the value is less than the coefficient, some other node is selected instead.
Personally I think option 3 is the most feasible. The max-concurrent-requests setting could then be deprecated after this solution is in place. SNOs would need to be informed about this change. Preferably an email to those who use the old setting. After having given them time to switch over, the old setting can be removed from the code.
Implementing one of these solutions would have the following advantages.
- Implementing the rate limit in node selection prevents nodes from rejecting transfers and prevents transfers from failing because too many nodes rejected it
- Nodes have a supported way to slow Storj down a bit to prevent bottlenecks in their setup
- Increases upload speed by ensuring there is always a long tail that can be cancelled when the success threshold has been reached. (Currently if enough nodes reject the transfer it’s possible that the uplink has to wait for all remaining nodes to finish, without long tail optimization)
- Less overhead, because it might be possible to lower the amount of nodes having to be selected because you don’t have to account for nodes rejecting the transfer.