Doh.. Looks like hardware failure coming up!

[Alexey]

@SGC

but raw writes and suchs will always be slower due to more iops being written to the drives… also there is a 3 or 4x minimum io amplification if xattr and atime is enabled, which they are by default.

I know all of that. As an argument I must say the btrfs has the same, but it’s at least in 2-3 times faster than zfs in the same level of RAID.
I also tested the RAID10 in all three systems, as it’s the only viable RAID for production databases.
The results have not changed: LVM raid, btrfs, zfs
BUT.
Since LVM raid have no checksum checks, and btrfs is unstable, there is no other choice. Maybe only Ceph.

@SGC

also i stopped using rsync since my nodes are already on zfs so when i move them i used zfs send | zfs recv
which runs about 10 to 12 times faster than rsync on zfs.

the same for LVM snapshots (with extension like GitHub - davidbartonau/lvm-thin-sendrcv: Send and receive incremental / thin LVM snapshots on a live volume. Replication / synchronisation of an LVM volume to a remote server by transmitting only the difference between snapshots of a live / running volume.), btrfs snapshots. So, no win here.

@SGC

what i really like about zfs is that its basically indestructible when maintained

I second that.
But still - it’s slow. You just need to account that into considerations and make changes accordingly.