Much needed in my opinion indeed!
I shared some thought on this in the past, there:
To sum up, I think what the node software could do is something along those lines:
- Monitor how much RAM is taken by buffered pieces waiting to be written down to disk.
- Accept new pieces as long as this buffer does not go beyond a certain threshold.
- This threshold could be set at 32MiB by default for instance, but it should be configurable so we can adjust it depending on how much RAM we can dedicate to our node.
- Tell the satellite it cannot keep up if this buffer go beyond the threshold, and maybe start rejecting incoming requests from clients.
- Wait for the buffer to be empty, and tell the satellite we’re back in business and ready to accept new pieces again.
That would prevent what’s currently happening which is either:
- the RAM consumption going up and up until it runs out and the node gets killed by the OOM killer and then restarted by docker (which is a real problem as it can cause many issues like DB corruption, file pieces corruption, and it restarts the file walker process that takes hours of high disk usage, which is likely to worsen even more the disk performance…).
- or, the need to configure our nodes with the a very low number of maximum concurrent requests, which wasn’t designed to address this issue and unnecessarily and massively throttles down ingress even when disk can keep up. Indeed, typically an SMR drive can receive surges of massive ingress with no problem as they use their CMR cache section for that. They only struggle with long period of massive ingress: only then would they need the ingress to be throttled down.
Of course all of that is probably way more complicated than my bullet point list above to implement in practice, but as @jammerdan said SMR drives are everywhere and probably here to stay.
So I think they’re right: the node software needs to handle this by itself 