Currently the “.partial” temporary files are written in a way that usually end up in file fragmentation (block > 2MB generate between 2-10 fragments) and on big nodes with millions of files created / deleted it add ups. On windows for example (sorry the only platform I know well enough) you can pre…

@Alexey @littleskunk i think this should be given priority, if true… i must admit i haven’t been tracking my fragmentation of the data, so can’t really say, but i could easily imagine it as a coding oversight. since the issues it create will only get worse over time and it will take many months if…

@JDA if you want people to be able to vote you should move it to this subcategory. [image]

There are a few config options available. Can we verify if one of them would change the behavior already? filestore.write-buffer-size: 3MB pieces.write-prealloc-size: 3MB I never checked what these options will do. I just noticed that 2. something MB is the biggest piece on Disk. Lets allocate 3MB…

[image] littleskunk: There are a few config options available. Can we verify if one of them would change the behavior already? In my config.yaml I have: filestore.write-buffer-size: 128.0 KiB Wich seen to be the default value and I can’t find pieces.write-prealloc-size is it new? Is there a…

[image] BrightSilence: @JDA if you want people to be able to vote you should move it to this subcategory. Thank you, I have done that, I’m not a forum expert :slight_smile:

not sure i would increase the write buffer, but certainly the pieces.write-prealloc-size. wouldn’t the write buffer in case of massive numbers of small files eat a ton of memory if set that high. another question with the pieces.write-prealloc-size becomes if this will force even a 1kb sized file …

[image] SGC: can a 1kb file grow to a 2.4mb file or whatever they are, or is it static and will never change? 64MB / 29 pieces = 2.2 MB. We want to change the reed solomon settings and also the segment size to make this hole calculation better for storage node operators but that will take tim…

[image] JDA: I can’t find pieces.write-prealloc-size is it new? There is a nice trick. write-prealloc-size = WritePreallocSize in the code. This way you can search for any value in the code and end up here: storj/storagenode/pieces/store.go at 10372afbe423909e9993588a2df710e6e9cf1c6d · storj/…

If I understand correctly the pieces.write-prealloc-size setting is only during the “upload” once the piece is there, the file is resized to the correct size and moved. (For information, the default is 4MB) The implication is that it should not overuse space, but on the reduce fragmentation side it…

[image] JDA: This will lead to further fragmentation :confused: The best practice is to set the prealocated size to the final file size if it’s known. ofc holes in the data is much preferable to one file being in multiple different places, because those holes can be filled with other smaller…

Storj Community Forum (official)

Pre-allocate .partial files to final size to avoid file system fragmentation

Ideas & Suggestions Storage Node feature requests - voting