The fact that you don’t see the analogy doesn’t mean it isn’t there. Let me phrase it in your own words: “Who wants to volunteer to be one of the 36750 deaths (caused by traffic accidents a year in the US)?”
Answer: Nobody. Everyone goes on with their daily lives since the odds are very low to be one of them.
We get your point, but the math just doens’t support it.
Please show the math.
Hi everyone
I had the exact same question, and in my case, my startup is going to have a fleet of dedicated servers, each of which comes with 2 x 8 TB hard drives in RAID1, and I’m hardly going to make use of it. Storj makes sense since my servers would definitely be idle for a while each day, and I can put all this extra space to good use. Now RAID1 is kinda overkill, but that’s the way it is when I get it. I could either:
The reason I’m leaning towards the second option is I think it would take quite a while to get past the 8 TB mark, and as someone rightly pointed out, it’s very common to have drive failures near the start. I’ll have dozens, and eventually hundreds, of dedicated server instances, each with the same configuration, with each of them being a node operator, and I think it’s highly unlikely I would get enough orders to fill up all that space. Do you think this would most likely end up being the case?
I am using RAID6 (actually raidz2) with 6x4TB drives. The total data stores is about 3.2TB, which means reconfiguring this to 1 node per drive would just leave me with lots of empty space and higher risk of data loss.
Since Storj v3 considers all nodes in the same /24 network as one big node, I would not get more data than I do now, so there is no point in doing that.
5 hour downtime limit and no real way to back up the data (compared to, say, running a web or email server) means that I have to do everything that I can to make my setup as reliable as possible. RAID for drives, UPSs, a backup internet connection etc.
I see, makes sense. But all my servers would end up being in the same /64 ipv6 subnet, and I don’t know if they would even be in the same ipv4 subnet, so it could be the case each server is considered its own node by storj, which might make things interesting. But I think keeping the RAID1 is the best way to go.
Also my servers get 100% SLA, so I can easily stay within the 5 hour downtime as long as I have redundancy.
It sounds to me like your use case depends more on what else those servers are used for. I assume you are getting these servers for a reason.
It’s not inconceivable that would would fill up 8TB eventually on a single IPv4 subnet. So, I would still suggest one node per HDD if it’s just for storj. This equation changes if the servers share a subnet though. In that case you’re likely to have a complete overkill of storage anyway, so might as well stick to the redundancy. (assuming there are other purposes that justify the bills for these servers)
has the following case already been considered:
if you run a raid 5 with 8 HDDs then you lose 1/8 of the capacity but have the security of a failure of a HDD. If 2 disks fail again, then the complete raid network is defective and ALL nodes are broken. If you don’t have a raid and all 8 disks are in operation and 2 disks fail, only 1/4 of the nodes are defective.
You have the choice between high risk + little damage or low risk + high damage.
Translated with www.DeepL.com/Translator (free version)
The problem with RAID 5 is the rebuild stress on the drives. If someone is using very large consumer level drives with an unrecoverable bit error rate of 1 in 10^14 bits … then it’s very likely that an error will occur when the array is rebuilt.
This can be over come by using Enterprise level drives which typically have 1 in 10^15 bit error rate… Or by using ZFS.
Some of us are running large servers with already configured drive arrays… otherwise, it’s recommended to use one drive per storage node.
The problem with running one drive per node, is that most consumer level hard drives are not manufactured to run 24/7 and have a limited lifetime data transfer rating. So, any node running a randomly picked consumer level drive is probably not going to make it through a complete escrow period. This may or may not be a problem for a particular SNO… However, it’s good to be aware of the reality that running a node with a single consumer level drive is likely to be a short term node rather than a long term node.
So, if someone has an old drive sitting in a corner somewhere and a RPi … hooking up the drive to the Internet and running a storage node may yield a few dollars in STORJ coin… but it will likely crash out in a few months. If someone has several old drives sitting in corner, running STORJ nodes on them may help extract some value out of the otherwise idle or “useless” junk drives.
However, if one is expecting to get through the escrow period to full payout and beyond, it’s probably better to choose drives wisely… and/or run a ZFS array…
Priority of SNO’s should be to retain their reputation of their node and not gable their withheld amount by not having a robust backup solution which is what you were promoting with 1 HDD per node.
Playing Russian Roulette and creating 3 nodes with 1 HDD each to increase your short term payout at (25%) at the expense of the network if you loose a HDD with no backup and your own reputation doesn’t make sense.
And to me personally is irresponsible SNO behaviour, which is exactly why StorJ had to create a withheld amount to protect the network from bad node operators.
Your scenario for 1 HDD per node should also include a backup to encourage reputable SNO’s. However if you’re going do use a 2nd HDD as backup, then you might as well use RAID1. And later invest in a RAID5 solution (note that this isn’t a backup solution but a convenience of swapping out a bad drive for a new replacement and to minimize downtime if 1 HDD fails). It wont protect you from a 2 HDD fails and a backup HDD for 1 HDD per node should also be checked to verify integrity of the backup.
In addition to this; you also didn’t take into account that storage alone is not where the profit is but the use of that storage via download. So having 1/3rd of storage for redundancy makes sense to increase your payments from 25% to 100% by retaining your reputation for 10+ months and that small 1/3rd of the space isn’t going to fetch you much for cold storage anyway since download of the 2/3rd of the storage will fetch more profit and a lot more at 100%.
Your 3% increase in payment doesn’t make sense at all.
I would love to have a backup in addition to using RAID because RAID is not a backup. However, backups are not possible, but I still use a 6 drive RAID6 because I have no interest in starting over, especially since “starting over” takes about a year.
Correct, RAID is not a backup solution, but a convenience of swapping out HDD’s as I mentioned.
No one wants to loose out 75% of their income in withheld by gambling running a SNO with 1 HDD and no backup. Those that do probably are using cheap rundown hard drives or hardware and don’t care about the withheld amount and it’s all about the short term gain.
A situation where RAID would be even more important, maybe even 3 drive RAID1.
It’s not just about the held amount. I really hate “starting over”, but even discounting that - my node got 10TB of data over a year. It would take another year to get 10TB again, it’s not like I set the new node up, it gets vetted and then gets filled quickly and I just lose double the held amount (the held amount from the failed node and there has to be a new held amount from the new node).
There is a disconnect between Storj recommendations and requirements. The requirements are very strict (I do not know what the new uptime requirement will be, but the old one was really strict, especially considering most SNOs have home internet connections and do not have employees), but the recommendation is basically “well, just do whatever”, except that cannot meet the requirements.
A node is expected to not lose any data and even not permitted to have backups. I think that if I suggested using a single drive for an email server and not to have backups, everyone would just laugh at me, even if that drive was an enterprise-grade SSD.
Priority of SNOs should be to optimize their income. Priority of Storjlabs should be to make sure that SNOs optimizing their income aligns with network needs. Which it does, so SNOs don’t need to worry about that.
I’m not going to argue the rest of your message anymore. Show me a complete calculation of why RAID is more profitable and we can talk. Because all actual calculation people have done have shown the opposite. I’m done arguing pure opinion posts.
And I did take into account income from downloads. Check again. I even went as far to ignore the fact that recently uploaded data gets downloaded much more often than older data and scaled it linearly with data stored. You would realistic lose even less profit by starting over.
You wouldn’t have lost 10TB if you ran a node on each HDD.
Anyway, I’m putting my money where my mouth is, I recently started a node on a separate HDD that I couldn’t use to expand the array in my NAS. As mentioned before, the only reason I’m using that array is because I already had it and the redundancy was needed for other stuff on the array. With this new node I have no need for it. We’ll see how it goes.
It would still take that node a year to get back to where it was.
If I ran a node for each drive, I would have to use at least one SSD for each node (for SLOG) as well. While that would improve performance, I would have to use more SSDs than just using two of them for the big pool.
Also, if I see that I am running out of space, I can relatively quickly buy new drives and expand the pool (hell, if I see that I am running out of space NOW, I could probably find 6x1TB drives and use them to temporarily expand the pool, replacing them with proper bigger drives when I get them). If I used separate nodes, I would have to anticipate running out of space at least a month in advance because it will take a long time for my new node to get vetted.
Why? Many SNOs are running nodes without it successfully and when running multiple nodes you would spread out the load, which probably makes the HDDs perform just fine. If you really want to you could move the db’s to an SSD with the new version. Of course you’d be creating a single point of failure among them, which is not ideal, so I guess you’d need a RAID1 there then. Make em cheap and small, I’d say.
But honestly the HDD’s alone should do just fine.
Ideally I would use a cluster for all dbs, but it looks like that would not happen without me learning some Go…
I remember managing a lot of nodes in v2 - I did not really like that, but I had to do it, because in v2 more nodes meant more traffic.
Yeah, don’t remind me of that debacle. V2 was much more of a resource hog as well. I ran 4 nodes on one array for no reason except that it would get me 4x the traffic and my NAS couldn’t really do more than that. I’m glad that is fixed, at least now the discussion is purely about how to use the space most effectively.
So why did you choose RAID but you’re telling others not too?
With your setup with the NAS and 4 or 5 bays, you don’t need to run RAID right… Why are you not running 1 node per HDD bay???
Let me answer for @BrightSilence: Because the RAID already existed and he uses it for other purposes. And as it happens he has free space in this array. So why not using it?
I, on the other hand, have a RAID1 BTRFS array consisting of 4 hard disks of mixed sizes for my personal stuff, but 5 nodes on 5 separate hard disks.
Thanks @donald.m.motsinger for answering this! 
Though, I’m almost certain you already knew this, since almost every time I mentioned I use an array I gave this context. It’s also not a 4 or 5 bay NAS, that would be extremely wasteful for Storj, it’s a 12 bay that I started out having more than 12TB free space on and could still expand some more for Storj as well as other uses.
Additionally, I started my second node on a separate HDD, since I rotated the smaller ones out of the NAS array. I didn’t buy a new NAS or server and used a new array, because again, that would be wasteful. I’m running an individual node on it. So yes, I am putting my money where my mouth is.
I’ve never claimed that arrays are bad in all situations. I’ll also freely admit that it makes things simpler if it’s a question of managing one node vs 10 nodes. But it’s not the most profitable thing to do if you’re using your HDD’s only for Storj. And it seems you didn’t contradict that (again).