I was using a refurbished HP DL360P Gen8, along with some sh HP SAS (EG0900FBVFQ) drives I got from ebay. The drives had about 4 years spin time on them, no wonder they failed.
@Sasha
often people wonāt know that their array has failed and only become aware of it when a drive finally diesā¦ but drives are most often bad long before they die, so people trust their system to keep everything working and then when it breaks its like 1 disk dead and a Ā½ disk bad, or worn down so much that when the rebuild is started that kicks it over the edge.
raid arrays arenāt backups, they arenāt they are mitigation of the 2% or so odds that a data drive failsā¦ mirror / raid1 is a tank, raid5 is the sports car, raid6 is the SUV, and then raid 0 would be the motorcycle.
either of them will work for what they where intended for, but more often than not when they start to break people donāt notice, if you are running a raid, you should have regular checks of all data, which ensures that, on that day it could all be read within a day and put onto a backup diskā¦
one should have spare drives on hand, doesnāt have to be perfectā¦ it could be an oversized drive that fits most raid arrays on handā¦ ofc if you are running a raid 5 and has a spare ā¦ then you should be running raid 6 unless if its due to very special performance considerations, and most often that wonāt really hold up logical deduction.
@fikros
many of my drives are going to break 10 years of spin time this yearā¦
ofc that cannot keep up forever but thus far quite impressedā¦ but they are enterprise sata drives if memory serves
@Alexey
you really hate raid5 donāt you xD
@Krey
well should have checked the blackblaze data before buying themā¦
apperently that particular model excels at 30-40% AFR
An update on my GE progress:
I seems satellite.stefan-benten.de:7777 succeeded. Itās weird, and seems too good to be true, but I got a Completion Receipt and all. I donāt know how this was possible. I can provide nodeid if devs want to debug, just ask.
2 Likes
Yes for the reason. I was a system administrator and have had 22 branches with chunks of central database and all of them are used RAID5. Too much failures. Even with enterprise drives. Even with small enough (less than 1TB). They fail during rebuild because of two drives failures.
So - yes. Please, do not use RAID5 or RAID0, except you like adventures and thrills.
Every time when the branch is failed, the only way to recover was to unload a branchās chunk of database and transfer it to them. Of course we have had backups. But they was useless because of constant flowing of data (like in Storj) between branchās database an central database (unlike Storj).
But result is the same - the backup is out of sync and recover the sync was a pain. So, simple to unload the chunk and send them.
As result I migrated all branches to RAID10. The problem is gone. Disks are keep failing, but no one branch has been broken since migration (more than 5 years).
4 Likes
I know more than enough 
We have a choice, lose everything (DBās+Customer DATA+Escrow) or lose just DBās and make GE.
I think the choice here is obvious, right?
for obvious reasons, data on new models are not there.
Well, backblaze is not a panacea. There are no drives at all that I use
ouch sounds like the wrong job for raid5ā¦
and i kinda think people tend to miss that the drives are acting upā¦ i know iāve done so repeatedly then all of a sudden 16 days after a drive has failedā¦ i notice lolā¦
and if they are all the same type of drives maybe dying of ageā¦ then i could see how they might fail and the system doesnāt even notice because it doesnāt stress the drives until one start to rebuild and then they just choke and roll overā¦
so i will agree that raid5 isnāt safe, but in theory it should be safe enough if correctly maintained and so long as one doesnāt have the capacity to run anything betterā¦ with my current setup i can only go raid 5 or mirrorā¦ and mirror doesnāt improve my speed atleast write and only has 50% data capacityā¦ with raid 5 in a 5 drive array thats 80% data capacity and i can loose a driveā¦ even with a raid 6 type solution i would be back near 50% capacity and i would loose performanceā¦
so then i would be better of doing mirrors because then i can add and remove vdevs more easily with zfs.
so i donāt really see raid 6 as an option on anything less than 3x(8 drive raidz2 / raid6) getting 75% storage capacityā¦ great performance, quick rebuilds of drives anything less well i guess 2x(8) could work, but its performance would start to get kinda weakā¦
Of course RAID5 will save you from the one disk failure. But it will die with a high probability during rebuild, when you replace the drive. Larger disk - large chances: Hardware configuration and receiving mail with token
well you just said you lost
and they where on the 2014 list where i remembered seeing them, so some part of the logic here failsā¦
sure you may not be able to get the latest state of the art harddisk technology, but if you want endurance why would you want toā¦ thats like nasa sending the newest generation ryzen processors up in a satellite they expect to last 40 yearsā¦
they donāt do thatā¦ they use a older technology which has been proven tough, and have been further redeveloped into a hardened processor costing like 50k to millions $ a piece.
ofc thats not option for us mortals, so we get older verified good modelsā¦ and then build in some redundancy and make sure the system is well maintained.ā
you never want new tech for durabilityā¦ it always a gamble long term and you want to know what you are buying if you want it to last.
1 Like
@Alexey: Your posted image is for Disks with 10^14 bit-failure. Nowadays wd red are in that range (not the PRO-Version). But e.g. Segate Ironwolf (not PRO) already has 10^15. That means recovery-rate of 8x4TB not 10% successs, but 80% !
10^15 bit ist 113 TB of guaranteed writing correctness.
What about the idea to run multi-nodes on a raid5? In case you get a recovery bit-error maybe not ALL of your space is currupt and you do not loose ALL nodes at once but only the one node that has put data on the corrupt sector?
Or am I wrong, and bit-errors will lead to a total raid-loss?
So is is raid6 or no raid! ???
That seems way too fast. Only 1 day for GE? Mine took a lot longerā¦ What was your upload speed?
How much data did stefan-benten still store on your node?
I am sure thereās something fishy here. The amount of data transferred is nowhere near the amount stored. Either there are a lot of orphans on my node, or another kind of error occured.
I started GE before the dbs were rebuilt, figured I didnāt need them anymore. Maybe this caused the problem. However, I was under the impression that the sattelite coordinates the exit, from its end.
I guess weāll see when the payout comes.
Hmmā¦ that will for sure be interesting.
There are 2 images in that post. 8x4TB would indeed be around 80% success rate. Iād say thatās still not acceptable, but you may find that reasonable enough. It goes up fast even with those 10^15 disks though.
Excellent!
letās remember that one āboyā to said 
and you did it!
1 Like
long live raid5 xD
iāve been fighting a lot with my corroded backplane this last month or perhaps a bit longerā¦
itās given me a rich number of errors to evaluate the stability of my raidz1 which is basically raid5ā¦
even tho it should be a tad more enduring in some cases, it seems to me very much so that the excessive rebuild times that is seen in some older arrays during rebuilds could actually be due to bad cables, more drives having failed but due to low activity of the array the bad or failing drives have managed to stay under the radarā¦
then when one starts the rebuild drives kill into overdrive a start dying right and leftā¦ or having trouble thus increasing latency and extending a rebuild that should have taken days into the month range.
this ofc would kill off the remaining poor performing drives in the array and thus it failsā¦
so i think there would be a good argument to be made that most raid5 arrays in the 6-7 drive or lower range that fails on rebuilds, essentially has failed long before the rebuild process is started, they are just zombiesā¦ and thus in my views proper maintenance, scrubbing / patrol reads, performance monitoring and alerts should minimize such risks into acceptable levels for a storagenode.
iām not saying people shouldnāt run 8 disk raid6 insteadā¦ iām just saying 4 disk raid 5 is so much cheaper to setup, and if correctly maintained shouldnāt failā¦ ofc one should have a spare for when one drive diesā¦ and then one might as well go raid6ā¦ ofc a hot spare could be a global hot spare across multiple arrays, which is most likely what ill go for when i get my little home brew datacenter a bit more establishedā¦ not sure if ill stick with āraid5āā¦ atm i sort of feel like zfs was made with only running mirrors in mindā¦ which kinda sucksā¦ so painful to reconfigureā¦
It is and it isnāt. ZFS does a lot more to try and restore any data it can and your entire array most likely wonāt fail due to UREās. You may lose some data, but your array will survive unless another disk fails entirely. I have limited experience with ZFS myself, but Iām pretty sure the two shouldnāt be compared when talking about this issue.
well in gross basics, you got 1 drive redundancyā¦ doesnāt have to be more advanced than thatā¦
drop nr 2 or have uncorrectable errors and you got corrupt data even if the entire zfs pool might not die because of a bit of corruption the raid5 would get like 50% corruptedā¦
yes zfs is very different no doubt about thatā¦ but the odds should be closely relatedā¦
i did read something interesting about RAM which kinda rang true with some of the stuff in relation to raid and failures.
very interesting read, one of the things they note is that ram modules that are connected to each other and experience a correctable error in one module has much higher chances of experiencing an uncorrectable error in a different moduleā¦
it seems to suggest that failure is contagious might not be the best word, but something along those linesā¦ stuff fails because its connected to other stuff that is performing poorlyā¦
which i suppose is kinda true, and which could very well be related to the issue with raid5 not being good enoughā¦ i mean the math sort of says raid 5 should be good enough for arrays of limited disks, but i donāt have enough experience with it to validate thatā¦ this study however does suggest that this kind of failure might be a more wide issue.
not restricted to raid, which i find kinda fascinating and sure maybe raid6 is the only way to go in the real worldā¦ if you want the array to surviveā¦
i just found it so interesting that it seems to apply to storage tech that is so different from hddās
if one supposes this dimm issue on to hddās in raid arrays then read errors in one drive could lead to breaking other drives in the array, thus killing a raid5 basicallyā¦ would explain with raid6 is really needed.
i mean how many decades have people not been storing stuff on single harddrivesā¦ and sure they may throw and error here and there, but they are pretty damn reliableā¦ so raid5 should be plenty ā¦ in mathematical terms
ofc its not always reality wants to conform to the mathematical approximations of us humans.
I was saying āOh boyā, as in āIām in over my headā, most certainly not calling you āboyā. Just wanted to make this clear .
Thank you for your advice, it helped a lot, really.
4 Likes
That just means youāre learningā¦ if youāre not confused, youāre doing it wrong.
2 Likes