from my experience there is a vast difference in migration time between having a node running on a raidz vs having the node turned off.
also i would either resilver or copy the data out from the raidz, doing both is doubling the workload, which more than doubles the time it takes to complete the tasks in parallel instead of sequentially.
so i would shutdown the nodes and resilver because that would most likely make most sense, unless if you want to change something ashift or whatever normally unchangeable in a pool configuration.
if you want to copy the data out, you should be able to use a stop or pause command on the resilvering.
not sure what the command is, but one could always just pull the drive… the correct drive ofc
talked to one of the storjlings about that recently, but never got around to making a feature request for the option to modify the filewalker process, it will run every time you restart a storagenode… then as far as i can tell it won’t run again when done, maybe if a node runs for extreme lengths of time, but i don’t think so… it sounded a lot like it was basically a storagenode boot process.
and since it’s basically what makes sure how much space is taken up by the blobs folder and such… it is kinda important for various reasons, but having it run repeatedly with lots of free space and when trying to troubleshoot or like you migrate… it’s really annoying
you want to free up iops and the more iops you free up, the faster the stuff will go… and it’s not linear, hdd seek time when doing parallel tasks the time required quickly grows almost exponentially.
ofc if you got NCQ a bit of query depth might not be to bad, but haven’t really done any tests on that, its a hdd hardware feature that helps reduce seek time using some kind of predictive algorithm or so.
ofc in your case the l2arc and arc might carry the brunt of the load, but i have little doubt you can most likely complete your migration in 1/5 or less time as compare to running 3 nodes, resilvering and copying data out, if you shut down the nodes and migrate…
but at the very least i would kill the resilvering if you a migrating anyways or go with the resilvering, atleast with only 2 drives the odds of another failure isn’t to high xD
if you are using rsync then maxing out the blocksize might speed things up using the -B parameter
rsync -B 131072
but last time i tried it, i didn’t really see the big difference i thought i got last time… so might not be worth it… seemed to work well the first time i tried it tho… but that may have been because it was a mainly 128k recordsize pool.
and last but not least… what happened?
new drives and new hdd tray bay thingamajig… would have expected that to run for years.