Pre-allocate .partial files to final size to avoid file system fragmentation

Currently the “.partial” temporary files are written in a way that usually end up in file fragmentation (block > 2MB generate between 2-10 fragments) and on big nodes with millions of files created / deleted it add ups.

On windows for example (sorry the only platform I know well enough) you can pre-alocate a file with 0 on creation. This force the file system to try to allocate the file on a contiguous space and minimise fragmentation

API call used:


if (  INVALID_HANDLE_VALUE != (handle=CreateFile(fileName,GENERIC_WRITE,0,0,CREATE_ALWAYS,FILE_FLAG_SEQUENTIAL_SCAN,NULL) )) {                                               
        //preallocate 2Gb disk file                
        LARGE_INTEGER size;
        size.QuadPart=2048 * 0x10000;
        ::SetFilePointerEx(handle,size,0,FILE_BEGIN);
        ::SetEndOfFile(handle);
        ::SetFilePointer(handle,0,0,FILE_BEGIN);
}

SetFileValidData() can also be used, not sure if better.

With the system tools:
fsutil file createnew filename filesize

Ideally he client shoud be prealocating the file to his final size and not an arbitrary one (4MB default)

1 Like