Okay,
Thank You for the time.
Bye, See You!
I see the conversation has kind of concluded. But Iād like to add some info that might be useful anyway.
There is a little bit confusion about how nodes share traffic. The process is actually really simple. When the satellite selects nodes for upload, it first selects a set of 110 unique ip/24 subnets. Then within those subnets it selects a random node. So effectively, whether you run 1 node or 100, you get exactly the same amount of traffic total. Download selection is based on nodes that actually have pieces and since multiple nodes on the same IP can never have more than one piece of a segment because of how upload selection works, itās essentially just dependent on how much data you store. So by extension you see exactly the same amount of download as well.
Running multiple nodes is therefor only useful when you want to share multiple HDDās. But each node does need to be vetted individually. And when traffic is shared among multiple nodes, that vetting traffic is cut in half per node as well, making vetting 2 nodes take 2 times as long. This is why youāre advised to start 1 node at a time.
I used to advise people to start the second node only when the first one is starting to fill up. But Iāve revised that a little since. Starting the second node when the first one is fully vetted on all satellites has the advantage of having a node that could take over traffic if the first one fails, without having to go through vetting. So there is an argument to be made for starting the second node after vetting is done on the first. For anything beyond that I would in fact wait until at least 75% of available space is used.
You mentioned you have 4TB HDDās available. After a year of running my node it has broken 10TB only recently, got close to 11TB but is now dropping down again. Most likely that 8TB for the first 2 nodes will last you a long time. So donāt even worry about starting more until it becomes an issue.
One last note: Because traffic is split among nodes, the load on each individual HDD is halved as well. Because of this you most likely donāt need to worry about SATA II if youāre running multiple nodes. So as long as you have the ports to connect the HDDās, I wouldnāt spend any additional money on controllers.
This goes for other resources as well to a certain extent. There will of course be some overhead for each individual node process, but for the most part resource use per node will drop when you run more than one. Therefor I think your hardware should be able to run 4 nodes. Just donāt go overboard on the number of nodes, as there will be no advantage of doing that. Add one HDD & node at a time and youāll probably have a pretty good idea on whether your hardware can handle an additional node.
Hope that helps!
7 Likes
so that essentially means there is almost no penalty for running multiple nodesā¦
aside from maybe stuff like if data is stored on one and then is requested, then there is only one drive to deal with itā¦
Thatās actually quite good way to set it upā¦ one sure does quickly run into issues with raidā¦
tho i would say if one is running like 4-5 nodes then itās time to move to raid type setupā¦ because there will always be the 0.5% to 2% or worse of annual failure rate on HDD, so with raid one can start to mitigate that, ofc if one has 10 nodes the its like 20% of one failing in a year, and if one is running something like raid 6 on 10 drives then 20% is lost to redundancyā¦ so it kinda ends out around the sameā¦ aside from raid going all sideways when you after a couple of years have dropped 2 drives and or need to expand, rebuildā¦
splitting out across multiple nodes also makes it more easy to deal with the filesystemā¦ it will get pretty massive around the 10-40tb markā¦ that like 5 to 20million filesā¦ xD
1 Like
Obviously at some point between āI have barely a TB to shareā and āI have more storage space than I know what to do withā there is a point where RAID makes sense. But given the amount of data stored on my node right now (10.5TB) I would say the sweet spot of how much storage space to share is probably between 8TB and 16TB.
Given that RAID could be interesting if you have more than 16TB worth of HDD space. But that almost certainly means youāre using HDDās that are large enough that RAID5 is way too dangerous and you would risk a failed RAID array during rebuild. So, youād have to skip to RAID6, like you already rightfully did in your response. So yeah, if you have 6x4TB, maybe RAID6 is an option. But that comes with limitations as well. Want to expand the array? Youāre stuck with using 4TB HDD or replacing them all with large ones. With separate nodes, you can mix sizes all you want.
Of course the upside of using RAID is having to only manage one node, with one dashboard and managing payouts will be a little easier. Everyone would have to make their own decision, but I think separate nodes is more flexible and itās much easier to start small.
yeah your argument for having multiple nodes nearly convinced meā¦ but i am learning so much about zfs and linux, because iām just trying to my node online, and trying to setup my basic systems to keep track of whats going on.
in the future when the network really grows, raid might be advantageous, but if one even have to ask if one should run a storagenode on raid, then the answer is most likely NO!
simply not something thatās easy to start up withā¦ raid does well when you got let say ā¦ well i duno iām no pro lolā¦ but from my experience thus farā¦ i would hate to run raid with less than 24+ drives, a 36bays disk shelf + two serversā¦ anything less and you are just setting yourself up for PAIN
when one runs into the wall of ā¦ my array is to small i want to migrate it to a bigger oneā¦or my server is damaged i need to take it down for repairā¦ or i need to migrate of this raid pool into anotherā¦
its easy to setup raid, but painful to set it up wrong and its made for enterpriseā¦ not small home brew solutionsā¦ even tho they can be practical, better to just buy a bigger drive or setup two nodes in this caseā¦
if i wasnāt so interested in building up a small data center, i would have trashed this project a long time ago just because of the endless trouble iām having dealing with raid lol
zfs is amazing thoā¦
word of advice, if you arenāt willing to work for this, then donāt setup a raidā¦ just setup a few nodes to begin and let them workā¦ sure you can do raid, and it can be goodā¦ but iām just saying it will most likely suckā¦ because you most likely are guessing more than you are planningā¦
besides raid is apparently done in like say raid 6 on 8 drives and then you stripe across 5 vdevs
giving you a bit redundancy, 8 drives with 2 redundant so 25% capacity lost.
fairly low time to rebuild a driveā¦ maybe a few days or a week
and then for performance you stripe across 5 vdevs of that setup.
requiring 40 drives of which 10 is the redundancy and you get the best of everythingā¦ only now you are up in the scale where it pays offā¦ but then you need to worry about lightning strikes, disastersā¦ power consumption, because you most likely cannot turn it offā¦ well raid is nice and all, but to really make use of itā¦ then you are basically building the computer equivalent of a power plant.
and sure there are people that own wind mills, dams and what notā¦ but most people donāt, for good reasonā¦
1 Like