ZFS speed and optimizations for VM

[Pentium100]

@IsThisOn

To me it looks like you have cache set directsync (word cache is a little bit misleading here, since it is no cache and I also don’t know if naked KVM calls that differently than Proxmox).

<driver name="qemu" type="raw" cache="none" io="native" discard="unmap"/>

Libvirt documentation says that cache="none" opens the zvol as O_DIRECT. I know that this bypasses the cache. I sometimes use oflag=direct with dd when tryng to measure speed or whatever, especially on servers with lots of RAM. Without it, the data gets written very fast to RAM and then dd freezes for a while until everything is written to the disk, even Ctrl+C does not work. It’s very annoying when I am trying to just measure speed or whatever.

@IsThisOn

I would never virtualize a Firewall

It depends. I would not virtualize the main router/firewall, but other ones can work just fine. I work for a cloud service provider and we have a lot of VMs with pfsense or similar acting as routers for other customer VMs.

I dislike complicated things too, but sometimes it is unavoidable. Ideally, I would have separate hardware that I would use just for the node and so on, but I don’t. I’m trying to use the hardware for other things as well.

@IsThisOn

When a node writes data async inside the VM, the data gets stored in TXGs on RAM, fs calls back that if finished, writes it to the pool. No ZIL involved.

The way I understand it is this:
Node writes data to the ext4 filesystem as async. The data gets written to RAM, the fs call returns and the OS starts writing the data to the virtual disk. After some time the OS updates the filesystem metadata and syncs it to the disk. This then causes the data to be written as some form of sync in the host.