I do. That’s expected. Im surprised it survived so far into the madness as it did. See below.
Ouch!
So, with raidz2 your array’s IOPs capacity is bareley that of a single HDD. So, the ceiling of 400iops, is actually quite amazing.
Now, when you hit the IOPS celing, transactions start piling up, node tries to cache them, and ultimately is kiled by OOM watcher.
Most importantly, with iscsi you are removing any and all ability of the filesystem to optimize access. iSCSI + raidz2 == fail. It’s never an appropriate combination.
You can do a number of things:
- Stop using iSCSI, use NFS instead if you have to run node on another host. (don’t! see below). ZFS is amazing if you don’t cripple its ability to be a filesytem by hiding everything by opaque block storage protocol. NFS will give filesystem a chance to optimize access and at least cache metadata in ARC. You have to have a lot of free ram. 64GB would be a start.
- Intead of even NFS, run node directly on truenas. There is no reason not to, and many reasons against.
- Rebuild your pool with a number of raidz1 vdevs. You don’t need raidz2. If your other workloads need snazzy iSCSI – you’ll have to use mirrors.
- Add special device to your pool. Or, at leaset, L2ARC of at least 500GB.
- Ensure databses are fully accelerated (set small blocks size to record size for the dataset where databasses live)
- Read this Notes on storage node performance optimization on ZFS