yeah i just noticed that, it’s been a long long time since i’ve actually seen half decent ingress
but now that it jumped to the new internet connection it seems to help a lot… i’m sure the guy i was sharing the old internet with was happy to go down to being alone… might put one node back for the last 10 days of my subscription tho… just shy of 45gb ingress here to across my 3
yeah i’m running hus726060ala i think it is, i’m running 512B sectors… maybe a little bit on purpose… there seemed to be a lot of iops and i figured it might be better for this use case… but i suppose it all started with me being able to buy a stack of 512B drives cheaply and then ofc everything else had to fit around that.
the allocation tables for that tho gets kinda expensive… needs like 27GB if i wanted to use my IoMemory SSD as a swap drive… compared to like 4GB at 4K
so i’m most certainly going to move towards 4k.
my setup is 2x 4 disk raidz1 so very similar iops specially since the disk are very similar.
ofc you would have a half the write iops and lower sequentials speeds, but for storj wouldn’t expect to see much of a difference, duno how high my iowait gets during a scrub… don’t think it’s anywhere near those levels… but i suspect much of the metadata iops is handled by the l2arc.
there seems to be a big difference when it’s off and on, especially if it’s the 2nd scrub, it seems to run a bit quicker and not generate as much iowait.
and then ofc SLOG which will do the most immediate effect due to it not storing the ZIL on the disks.
one change i’m looking into is that i would like the l2arc to be across the different arrays i got, the way i understand it, then it will only support one of the arrays, would be very nice if it would act just in support of the entire ARC instead of pool based…
i went with sync always to push random write iops into a more sequential data stream, which seems to work quite well and it ofc also long term will limit fragmentation on the array.
still not even a year into using zfs, proxmox, pfsense and linux so fumbling around a bit still
tho have been finding my stride again…
zpool iostat -v
capacity operations bandwidth
pool alloc free read write read write
--------------------------------------------- ----- ----- ----- ----- ----- -----
bitlake 17.7T 26.0T 26 169 445K 1.58M
raidz1 17.0T 4.81T 13 78 386K 692K
ata-HGST_HUS726060ALA640_AR31021EH1P62C - - 3 19 96.6K 174K
ata-HGST_HUS726060ALA640_AR11021EH2JDXB - - 3 19 96.1K 172K
ata-HGST_HUS726060ALA640_AR11021EH21JAB - - 3 19 97.1K 174K
ata-HGST_HUS726060ALA640_AR31051EJSAY0J - - 3 19 96.5K 172K
raidz1 689G 21.2T 12 90 58.3K 930K
ata-HGST_HUS726060ALA640_AR11021EH1XAPB - - 3 22 14.6K 234K
ata-HGST_HUS726060ALA640_AR31021EH1RNNC - - 3 22 14.4K 232K
ata-HGST_HUS726060ALA640_AR31021EH1TRKC - - 3 22 14.8K 234K
ata-HGST_HUS726060ALA640_AR31051EJS7UEJ - - 3 22 14.5K 232K
logs - - - - - -
3486798806301186697 0 5.50G 0 0 0 0
--------------------------------------------- ----- ----- ----- ----- ----- -----
opool 4.18T 1.28T 68 53 32.6M 14.9M
mirror 4.18T 1.28T 68 53 32.6M 14.9M
scsi-35000cca2556d51f4 - - 4 34 3.52M 14.4M
scsi-35000cca2556e97a8 - - 63 19 29.1M 507K
--------------------------------------------- ----- ----- ----- ----- ----- -----
qin 190G 5.25T 1 55 5.82K 527K
mirror 95.0G 2.63T 0 27 2.94K 263K
ata-TOSHIBA_DT01ACA300_Z252JW8AS - - 0 13 1.41K 132K
ata-TOSHIBA_DT01ACA300_99QJHASCS - - 0 13 1.52K 132K
mirror 95.0G 2.63T 0 27 2.88K 264K
ata-TOSHIBA_DT01ACA300_99PGNAYCS - - 0 13 1.48K 132K
ata-TOSHIBA_DT01ACA300_531RH5DGS - - 0 13 1.40K 132K
--------------------------------------------- ----- ----- ----- ----- ----- -----
rpool 89.3G 49.7G 1 36 111K 582K
ata-OCZ-AGILITY3_OCZ-B8LCS0WQ7Z7Q89B6-part3 89.3G 49.7G 1 36 111K 582K
--------------------------------------------- ----- ----- ----- ----- ----- -----
running storj on all 3 pools the rpool is the OS SSD
had to do a reboot after i messed up my network configuration trying to get all my new vlan setup, then because i had used some custom open source drivers for my L2ARC / SLOG device and the kernel was updated after reboot, the driver was dropped… haven’t gotten that fixed, yet but it’s nearly at the top of my todo list…
it’s why the logs on the bitlake pool is dead…
one interesting thing about the bitlake pool is that the avg read iops is near equal on both raidz1’s even tho one has about 90% of the storagenode data, seems to indicate that older data is much less active.
but can’t really complain about that since it means i don’t have to worry about it being unbalanced and it should balance out a lot over time.
and opool iops ratios looks a lot like yours because i’ve been scrubbing it a good deal… the one sas drive is on it’s way out and keep throwing errors… might have to pull it one of these days and see if there is anything i can do to try and improve it… maybe a bit of contact cleaner and some insulation from the metal HDD caddies…
they might not be designed for disks with this bulky a design, so i think the test points on the pcb might at times create leak current… or it seemed to fix another drive that was giving me grief…
damn cheap chenbro case, not only was the caddies way to expensive, the case is also kinda janky, should have gotten a super micro or ibm
got some 5 vm’s running but the storage pools are mainly just storj, everything else is running off the OS drive… but need to get that settled soon… running low on space tho it’s partitioned to 60% of capacity, not sure if it’s got any by default… but want to get that mirrored just don’t have a good partner ssd for it…
was thinking of pushing it to the PCIe SSD but after it’s recent downtime, thats not likely to happen…
might do some internal usb 3.0 boot and then have the OS migrate a copies of itself across multiple pools and thus always be able to boot from something…
a mirror solution might just be so much more simple and easy to approach… 
but then i have no redundancy if the onboard sata controller gives out… but it’s not to high on the list, one of those problems that aren’t really a problem presently, so when i come up with a great solution and an excuse to implement it.
ran a
zpool iostat -v 600 to get some current stats
--------------------------------------------- ----- ----- ----- ----- ----- -----
capacity operations bandwidth
pool alloc free read write read write
--------------------------------------------- ----- ----- ----- ----- ----- -----
bitlake 17.7T 26.0T 11 215 468K 2.24M
raidz1 17.0T 4.81T 8 99 446K 981K
ata-HGST_HUS726060ALA640_AR31021EH1P62C - - 2 25 116K 246K
ata-HGST_HUS726060ALA640_AR11021EH2JDXB - - 2 24 113K 244K
ata-HGST_HUS726060ALA640_AR11021EH21JAB - - 2 25 110K 246K
ata-HGST_HUS726060ALA640_AR31051EJSAY0J - - 1 24 107K 244K
raidz1 689G 21.2T 2 116 22.2K 1.28M
ata-HGST_HUS726060ALA640_AR11021EH1XAPB - - 0 29 5.83K 330K
ata-HGST_HUS726060ALA640_AR31021EH1RNNC - - 0 28 5.53K 327K
ata-HGST_HUS726060ALA640_AR31021EH1TRKC - - 0 29 5.64K 329K
ata-HGST_HUS726060ALA640_AR31051EJS7UEJ - - 0 29 5.20K 327K
logs - - - - - -
3486798806301186697 0 5.50G 0 0 0 0
--------------------------------------------- ----- ----- ----- ----- ----- -----
opool 4.18T 1.28T 477 43 278M 1.18M
mirror 4.18T 1.28T 477 43 278M 1.18M
scsi-35000cca2556d51f4 - - 156 21 139M 605K
scsi-35000cca2556e97a8 - - 320 21 139M 603K
--------------------------------------------- ----- ----- ----- ----- ----- -----
qin 190G 5.25T 1 91 16.0K 1.40M
mirror 95.2G 2.63T 0 44 8.77K 707K
ata-TOSHIBA_DT01ACA300_Z252JW8AS - - 0 22 3.63K 354K
ata-TOSHIBA_DT01ACA300_99QJHASCS - - 0 22 5.14K 354K
mirror 95.2G 2.63T 0 47 7.21K 730K
ata-TOSHIBA_DT01ACA300_99PGNAYCS - - 0 23 4.49K 365K
ata-TOSHIBA_DT01ACA300_531RH5DGS - - 0 23 2.72K 365K
--------------------------------------------- ----- ----- ----- ----- ----- -----
rpool 89.3G 49.7G 0 40 13.1K 637K
ata-OCZ-AGILITY3_OCZ-B8LCS0WQ7Z7Q89B6-part3 89.3G 49.7G 0 40 13.1K 637K
--------------------------------------------- ----- ----- ----- ----- ----- -----
opool is back as scrubbing again… and most likely failing again… for the 5 or 6th time, last time after 14+ scrubs it stopped acting up… but since it’s been running a storagenode for 3 months now, the drive has begun to be complaining again.
the SMART also says its dying… so its most likely dying…