Storage Node with multiple HDs

In my opinion, I agree with @cdhowie software raid is also solid stable for storage node and you can easy did it on any hardware (not only on the same series raid card like hardware raid controllers). Linux md-raid or ZFS (for advanced users) is excellent.

2 Likes

Well either way you can loose everything with Hardware raid or software raid if software crashes you could loose everything. Less likely to loose with hardware raid vs software But with hardware you can just replace with same controller.

There is always the possibility that you’ll lose everything. I’ve only ever had this happen to me when using hardware RAID when the controller damaged multiple disks as it died. A software bug is certainly possible – by 2020 though I’d hope the really nasty ones have been ironed out of md-raid considering how widely it’s used.

With software RAID you don’t have a controller that needs replacing in the first place. :wink: You also don’t need to ensure the controller is compatible with the RAID metadata on disk. md-raid metadata is readable by pretty much any recent Linux distribution, which makes recovery vastly more simple – you’re not dealing with opaque, proprietary data structures.

3 Likes

That is true but hardware raid is much faster then software raid. Ive had pretty good luck with my hardware raids vs software raids like when I used freenas had issues with crashing but probably has improved since I used it last.

This can be true, particularly if the controller has a writeback and/or read cache. A writeback cache can dramatically boost performance, but the cache storage must be redundant or there is still a risk of data loss.

On Linux systems, you can mirror two SSDs and use that as a writeback cache to improve the write performance of RAID storage.

Even still, you’ll usually only notice a difference during very spiky write workloads. In my tests the bottleneck was pretty much always a disk, and the performance differences between hardware and software RAID were negligible (under our workload, anyway).

1 Like

The bottleneck is the limitation of Sata vs PCIE

In the context of Storj nodes using many small disks (as per OP) you aren’t going to saturate either type of link.

No of course not less your using the same raid for other OSs

Likewise. And I think having a healthy debate about it helps others decide where they fall on this spectrum. That’s why I kind of took issue with definitive statements, like saying a certain setup is a must. There are options and SNOs should choose which one they want based on all the information.

There is no need to have any additional hardware for a non-RAID setup of multiple nodes. Nobody said you had to run them on different systems. The exact same hardware that could run a single node on RAID can also run multiple nodes on individual disks.

This should always be the last choice. It has no advantage over individual nodes and a significantly higher failure rate. My list would be individual nodes on single disks, RAID5/6, spanned/striped.

It doesn’t limit it any more than it would limit a single node. You just always get the same amount of data on one IP, no matter how many nodes you run.

This is not relevant. Multiple nodes will perform the same as a single node, since they don’t get more traffic.

3 Likes

(You already know this, @BrightSilence – this is for @eagleye)

One huge caveat to this is that nodes share potential traffic with each other but not the vetted status. So if you bring up ten nodes at once, they are going to take a very long time to vet because they are each sharing the 5% of traffic that unvetted nodes get. Each node must have 100 successful audits to be vetted, which will take longer with each individual node having fewer pieces that can be selected for audit.

This is why the recommended advice is to bring up one node and wait until it is nearly full (~90%) and then start the second node, and so on.

2 Likes

This is one reason why the recent suggested “Best Practices” for running multiple nodes is to wait until the current running node fills up and then start a second one.

I’ve run a few simulations in GNU/Octave for multiple nodes using single drives versus a more reliable RAID node… the calculations result in similar payouts if there is at least one hard drive failure. So, I’d say it’s up to the individual SNO… single drive vs. RAID will probably be a wash under most conditions.

However, my simulations are not yet completely figured out yet… still working out the possible scenarios.

2 Likes