Write-buffer-size and write-prealloc-size explained

[snorkel]

Can you explain a little bit what each of this lines does, and what affects each one? RAM, HDD, etc.

--filestore.write-buffer-size 4MiB \
--pieces.write-prealloc-size 4MiB \

Is there a dependency between them, between the values? I understand that buffer affects RAM, what about the other parameter?
I have a machine with small RAM and I’m afraid of setting 4MiB. If I set 1 or 2MiB for buffer and 4MiB for prealloc, is there a problem?
Thanks!

[bre]

This post and thread should get you started until someone is here who can specifically answer

If I set 1 or 2MiB for buffer and 4MiB for prealloc , is there a problem?

Filestore.write-buffer-size

The write-buffer-size sets the in-memory buffer for incoming pieces. You would need to have that much RAM available per upload request. So, you could end up getting out-of-memory errors when you set it too high. But, there’s really no benefit it setting it over 4MiB. At most it would need to store the whole piece, which at the moment is ~4MiB. Setting it to that number is only useful for drives that don’t like small writes (such as SMR drives). This of course will mean that it needs to write al…

Pre-allocate .partial files to final size to avoid file system fragmentation

Currently the “.partial” temporary files are written in a way that usually end up in file fragmentation (block > 2MB generate between 2-10 fragments) and on big nodes with millions of files created / deleted it add ups. On windows for example (sorry the only platform I know well enough) you can pre-alocate a file with 0 on creation. This force the file system to try to allocate the file on a contiguous space and minimise fragmentation API call used: if ( INVALID_HANDLE_VALUE != (handle=Crea…

[Vadim]

@snorkel

--pieces.write-prealloc-size 4MiB \

This option will eat space like hell, at one point you will have half disk but you will not be able to write there, because you will be out of 4MB space at one piece. because if file even 1 kb it will reserve 4MB there. and you will not be able to write there any more next 1kb because left space will be less that 4MB

[snorkel]

@Vadim
This is the default setting. I use it in Docker run command, just to be aware of it, but it’s the default setting in the newest config.yaml.
Even if I don’t use it in docker run, it will still be used with 4MiB. Untill Storj team dosen’t change it, I won’t as well, because I don’t understand what value is better and what problems can create.
What value are you using?
From what I read in linked threads, the space is freed after the piece is written on disk, so it will be full of data, not half empty blocks.
The code looks like this:

[Vadim]

I have a question if Filestore.write-buffer-size: 4 MiB but .partial files are stiil accrue in temp folder why? why it not write directly to folder, where it intend to be?

[Alexey]

because there should be only whole pieces.

[Vadim]

but there is whole piece in the ram already, just write it there. it is the same, you write it to temp then copy to new location, but if buffer is 4mb whole piece can be in buffer. If buffer is only 128k then yes you need make it part by part.

[Alexey]

@Vadim

you write it to temp then copy to new location

actually we use move not copy in this case if I’m not mistaken.
Why to implement two mechanisms to write a piece, if one is working for both cases?

[Vadim]

additional IO usage? but yes if it move then it wont take so much resources.

[jammerdan]

Is

@snorkel

--pieces.write-prealloc-size 4MiB \

still a thing with this change:

storagenode/blobstore: remove preallocate size