What about defragmentation for SN…
A defragmentation tool in the background, without much load on the array, would facilitate faster read/write access.
Why? My personal opinion is that…
On conventional HDD and even RAID will be strong fragmentation, during the operation of the node, files are erased and written (after the introduction of the cleaner), large volumes of disks and over time, read / write will take more and more time for HDD mechanics.
As an option, it can be Diskeeper in automatic mode (THIS is not an ADVERTISEMENT)
Who that thinks on this about ?
p.s. I’m a newbie, and gathered my node on my knee with Windows and 1TB HDD I plan to further expand the volume through RAID6
Have you seen the number of files that a node holds for 1TB! I am pretty sure node’s db has indexes to locate the correct piece and retrieve it for processing. De-fragmentation would put unnecessary load on CPU. Storj is meant to be used even on RPi 3.
Maybe I don’t understand indexes ? correct, but I will say as I understand.
If there is an index, it is fast for the database to find the file, there are no problems and the database knows where the file lies, but it must not only be found but also read and send, and given the fact that the file is fragmented, HDD mechanics need to do extra work, and this in turn slows down the response
Conventional defragmentation tools, Yes, create a large load on the disk subsystem at the moment. That’s why I wrote that we need a background tool that does not take on much and does not aggravate the situation with fragmentation.
According to the reviews of the above Defragmenter, I realized that their customers are very happy with how the response values of their systems on highly loaded servers have improved.
At us, in case of acceptance of a network by the world, nodes of operators will be very loaded with input output and if on a disk there is a mess it will slow down everything, it seems to me so…
I’ve seen it too, but this item is easily disabled in settings. The rest of the functions are very useful to us, such as the preventive operation of their logic, as one of.
But I am interested to hear exactly the opinion of the community and engineers on this issue.
Not necessarily this tool, (I cited this Defragmenter as a suitable example in my opinion just) can be another program,
I use zfs (ext4 on a zvol), so I hope that it can allocate space efficiently. I also use SSD cache for writes and reads. If it so happens that my node stores some files that generate a lot of traffic, the cache would help. However, if all files in the node are accessed equally frequently, then the performance will not be great with HDDs no matter what.
Since the node only creates and deletes files (never modifies them), the fragmentation should not be that big as long as there is enough free space.
Precisely the that not only housed, and will and remove, makes fragmented composition over time, if pitch this on gravity.
Although I have now a small volume of deposited, (pass audit on this moment) judging by tests compact disks are filled strongly.
I also think that a lot depends on the file system, but the average node operator, a normal disk with NTFS
Wrose would be if it extended one file. When the node wants to create a new file, the filesystem will find a non-fragmented block to put the file in, as long as there is enough free space. So, each file should not be fragmented that much.
Files themselves will, of course be all over the disk, but they have no relation to each other and there is no way to predict which files will be accessed frequently (to keep them together).
So, I do not think that this will be a big problem.
On an almost full 3.5TB node I had a problem with a lot of “ordersdb error: database is locked” errors and 99% I/O with 4 threads. The orders.db was heavily fragmented (~180000 extends). After defragmenting all databases the I/O is back to normal and so far no more errors.
I meant the database files only. I’m not sure whether defragmenting the whole drive is a good idea. It might be that it adds to the high I/O too. The files in the blob folder are relatively small, so fragmentation would not be a problem I think. Also, once stored they are not read from and written to, so it doesn’t matter really. The database files on the other hand are constantly in use.
I think Windows can’t defragment files which are currently in use. You can analyze the fragmentation with contig and defragment, after stopping the node.
