My pool is made up of three 6-drive raidz2 vdevs (4TB, 6TB, 8TB - I know different capacities are not optimal, but I slowly expended the pool and different drives were the cheapest per TB at different times). The drives are 7200RPM. My node currently has 27.7TB of data.
Recently I had problems with performance during fiewalker operation. There is a thread about it
But the TLDR version is that I had two problems:
- The filewalker runs inside the VM with lower priority, so it is supposed to “give way” to normal node operations. However, to the host, all IO operations from the VM look the same, so the VM uses all available IO, possibly creating problems for other VMs on the host.
- I had passed the zvol with
discard=unmapparameter and mounted the fs withdiscardoption so that when files are deleted from my node, the disk space is freed on the host as well. This seemed to work OK for a while, until the node started doing a lot of deletes (100GB or so) at once. The discard operations saturated the IO resultin in problems inside the VM and on the host. Combine that with the filewalker and it’s fun times.
What I did to solve the problems (or so it looks):
- Write a couple of scripts to 1) put the filewalker process in a cgroup and limit its speed and 2) adjust the limit so that the IO load is around 60%
- Remove
discardfrom the mount options and instead run a script once in a while that runs fstrim over small ranges (10GB) and with the minimum extent length set to 1MB (no point in discardng a single 4K block), though I may change that to 64K. If the particular range had some free space that needed trimming, then wait 5x the time it took fstrim to run before going to the next range.