Write-buffer-size and write-prealloc-size explained

[bre]

This post and thread should get you started until someone is here who can specifically answer

If I set 1 or 2MiB for buffer and 4MiB for prealloc , is there a problem?

Filestore.write-buffer-size

The write-buffer-size sets the in-memory buffer for incoming pieces. You would need to have that much RAM available per upload request. So, you could end up getting out-of-memory errors when you set it too high. But, there’s really no benefit it setting it over 4MiB. At most it would need to store the whole piece, which at the moment is ~4MiB. Setting it to that number is only useful for drives that don’t like small writes (such as SMR drives). This of course will mean that it needs to write al…

Pre-allocate .partial files to final size to avoid file system fragmentation

Currently the “.partial” temporary files are written in a way that usually end up in file fragmentation (block > 2MB generate between 2-10 fragments) and on big nodes with millions of files created / deleted it add ups. On windows for example (sorry the only platform I know well enough) you can pre-alocate a file with 0 on creation. This force the file system to try to allocate the file on a contiguous space and minimise fragmentation API call used: if ( INVALID_HANDLE_VALUE != (handle=Crea…