Hi, I’d like to find out if it still makes sense these days to place the temp folder on an internal M.2 SSD to increase the chances of success in a “download race,” or if that is no longer necessary or simply pointless.
translate with Google
Hallo ich möchte mich erkundigen ob es heut zu tage noch sinvoll ist den Temp Ordner auf eine interne M2 SSD zu legen um die Erfolgs chancen zu erhöen beim Downloadrennen. Oder ob das nichtmehr notwendig ist oder unsinnig.
To win upload races… the node writes async now anyways… so really the customer is uploading into your memory for your OS to deal with when-it-has-time. Everyone has crazy-fast memory - so winning is more about your Internet connection.
To win download races… IO is so random you’re unlikely to have it in memory (or L2ARC) so you probably always have to go to the HDD like everyone else. However a special-metadata SSD with small-files support will have those small files available fast (the ones that are the most trouble for a HDD to deal with) so that could help. And even just the metadata being on flash could help your node respond a tiny bit faster.
In the end if you have a 100/100 cable Internet connection and a competing node has a 1G/1G fiber connection… there may not be much a SSD can do to help you win 
Ah, okay. I also wanted to support the HDDs with that, since they are connected via USB.
This optimization was brutally murdered by hashstore. Now access to all pieces equally sucks.
Why? Our tests showed the significant improvement over a piecestore backend - the select nodes can handle a much more high pressure with lower specs.
Do you see a degradation for your setup?
Piece store allowed very fast access to very small pieces that could be located on the special device. But evidently it wasn’t scalable on crappy filesystems.
Hashstore does not allow that optimization. It imposes the floor of hdd seek on every, including very small segments, and in return scales better on crappy filesystems.
It trades latency for throughput. Quite a common trade.
It wasn’t needed on ZFS but that’s a different story. I understand storj not wanting to support two backends, etc.
Do I see degradation of performance? Yes. I now see much fewer IOPs to special device, meaning, a lot of pieces that benefited from my massive special device can no longer do so and experience no longer avoidable seek latency.
Does it affect storj customers in any way? Probably not. That added latency is still lower than some home ISP connections. But it’s there.
I see. Historically a hashstore backend was introduced to fight with high IO pressure on Select nodes, especially when there are piece walkers here and there and it’s solved this problem. It also helped weak devices as a side effect, thus gain a significant improvement on throughput not only on Select nodes, but also on the global network, where the majority uses ext4 or NTFS and sometime BTRFS. ZFS was out of scope, since it showed itself not good on the Select nodes, reported by multiple Select providers and our monitoring (they are fall offline too often, for the Select it’s critical due to a lower number of nodes). I do not know details, but I believe these operators just tried to use maximum of their HW, and overloaded it, or didn’t have a time or willing to understand how to configure it properly.
Nah, it’s just that some crappy file systems do not allow caching small and hot chunks of files.
Caching does not improve first access latency.
So, there’s no point in mapping the temp folder to the SSD via docker run to improve latency—specifically regarding the way data is written to the large HDD.
It would likely not work at all, see Disqualified after 3 years - #7 by AndMetal