I just had to reboot the server for unrelated reasons, so here are some plots.
Storagenode on startup was consuming 100% cpu for 10 minutes, during which time it was reading from metadata device at ridiculous rate, peaking at 3k IOPS:
This is the activity on one of the HDDS on the array during the same period:
I.e. big fat nothing.
This confirms that the storagenode startup is metadata heavy: moving metadata to a fast device (cache or otherwise) removes any impact of a storagenode cold startup.
Curious behavior of ARC during this time:
This node holds just short of 12TB of data, and startup took just over 10 minutes, on a 2012 era hardware.