Copying node data is abysmally slow

bre · June 10, 2022, 10:43pm

littleskunk made an excellent detailed answer here

github.com/storj/storj

Terribly ineffiecient storage; too many tiny files

opened 11:27PM - 01 Jun 22 UTC

closed 10:38PM - 02 Jun 22 UTC

bathrobehero

Bug

I have over 2,5 million files for 512GB of storage (607GB on disk) which is just… absolutely terribly inefficient even on SSD storage. And it's pure pain on HDD. As many professionals are working on this project, it's baffling that file system optimizations were completely ignored. I love the project and been following since 2017 and recently started a node, but I'm closing that down until it's fixed. There's absolutely zero sane reason to store files in such tiny chunks. It's terrible for performance in every respect. The only thing that it makes it suck just a bit less is that files are stored in 10,6k folders instead of a few... And it's a miracle the client is not eating GBs of RAM (yet). It's only eating ~570MB so far but I have no clue how that would scale. For a project this size, an option should have been made long ago that would allow users to decide the size of chunks they wanted to store data (64/128MB/256/512/1024MB/etc.) without killing file access time, MBR bloat and just general performance on NTFS and most other file systems. I was planning to store 50+ TB of storage but I'm absolutely not going to deal with the pain of storing ~250 million files!

Which links to this forum post as well