So are you saying that you would not have redundancy on your storage? My system right now with one node and a lot of other things is not getting bogged down. I have an AMD EPYC 7302P 16-Core Processor CPU and SSD caching and planning on upgrading the CPU in the future. I am not even using 25% of my CPU right now.
Why you need upgrade cpu if you dont use over 25% of it?
1 Like
I was running this on a CEPH cluster but are currently migrating away to single drives as running it on CEPH is a financial insanity, unless you can run all the equipment needed for it for free.
To overcome the redundancy problem with single drives you can do one node per drive, so if one fails you still have the other ones. Moving single drives is also much easier if you are moving between locations, with array you can only take the whole array or you have to copy it away.
And this isnāt a CPU issue but IOPS issue as spinning drives can handle only so much small random R/W workload. And as the node gets bigger it is bigger of a problem.
But if you are planning to use the array for something else than Storj, then I would say go with array.
1 Like
I would Upgrade when I need to.
How is that redundancy when you do ā1 node 1 driveā because if you lose one drive on that node that hole 1 node is gone? That data is gone too. Would you as a customer want the place you are storing your items to be in redundant storage?
The redundancy is done by erasure coding. Loosing one node is nothing to the network and the data will be repaired as they are all the time.
Also, the drive failures I had recently were all soft failures, mainly bad sectors, some drives unable to read random sectors etc., so chances the drive will go from one to zero are minimal I think and you will have time to copy to a new drive.
Iām also storing my most precious data on Storj network, so from customer point of view I think about it very highly.
2 Likes
Ok, thank you for the insight. I did not know about the ārepairedā. I thought the data was just split not duplicated.
This becomes very simple if you use storagenode as conceived: to share unused storage from an appliance you already have deployed for other existing needs.
Nobody has a zoo of individual hard drives running separate file systems in production. Thatās not how modern drives are designed to be used, itās just silly. (I in raid used to stand for āinexpensiveā before it was politically corrected to āindependentā. Today all drives are āinexpensiveā aka crap that is is expected to fail. Raid of crap drives provides better reliability and is cheaper than a single over-reliable drive(that does not exist because nobody would buys it because it would be obnoxiously expensive))
Hence, you already have a redundant array before you even consider storj, and hence the question is moot.
Hello @theguy167,
Welcome to the forum!
We do not use duplication, we uses erasure codes, so any 29 pieces of 80 is enough to recover the segment. The file is encrypted, split to segments (64MiB or less), split to stripes, erasure coded and split to pieces, then uploaded to nodes.
See
All nodes behind /24 subnet of public IPs are treated as a one node for ingress (uploads from the customers), and as a separate ones for egress (downloads by the customers), repair egress traffic, audits and uptime checks.
So, if you go with one node per disk, all these nodes will act as a RAID on the network level. If one disk die, you will lose only this part of the common data, not the whole array, as often happen with a usual RAID5 due to bitrot during rebuild after a one drive failure. It also could affect even zfs, if during rebuild you would get a cascade drive failures due to a higher load during scrub.
I would suggest to read this topic: RAID vs No RAID choice
However, if you already have a RAID for other purposes - you can use it for Storj too.
3 Likes
I have tried both setups. Started with raidz2 since I had an existing system prior to Storj. No major issues, ran flawlessly for years and never even saw anything less than 100% audit score.
Then, for several reasons decided to switch to single disks. So I spent quite a bit of time offloaded all my nodes onto separate disks. My reasons were mainly to take advantage of the extra storage space previously being used for redundancy since Storj doesnāt require it, cheaper upgrading as I can upgrade single disks at a time and not have to buy all drives up front upgrading the whole pool, and figured performance benefits would be good as well.
Wellā¦ tried that for a couple months and now Iām going through it all again to move back to pools. Within the first 2 months I lost 2 larger nodes. Drives just died to the point they couldnāt even be mounted. One drive was practically still new, the other maybe 4 years old. Not only that, loosing the drives caused the whole server to crash so I had a little down time sorting out which drives etc before getting it back up. From that time there was a SHARP drop in data coming in, so not only did it cost me like ~15TB but also new data coming in. Besides this, I started noticing very minor drops in audit scores for nodes on otherwise perfectly healthy disks. Lastly, managing 36 independent disks per server is absolutely annoying.
Soā¦ running Storj is simply not profitable with a small setupā¦ itās a hobby. Donāt spend money. If your hoping to scale somehow on your own now or wait till the data just starts flowing in more the normal ways it will still take time and require money to make moneyā¦ which there is no guarantee same as any other business or investment venture. Be smart and donāt blame anybody else except the US gov if things donāt work out.
That said, the ONLY way Storj is legitimately profitable is quite literally if its running as advertisedā¦ on equipment thatās already on all the time (and assuming your time isnāt worth anything to you), and adding any form of redundancy in any way will pretty much leave you in the hole. This also isnāt a sustainable model long term so āwhalesā will inevitably survive. If you do decide to invest (meaning risk) money, I would at this point go with raidā¦ but software raid, not hardware. Make sure what you have can handle it. Know what your doing. Donāt cheap out where it counts, but donāt overspend either. Successful whales are whales because they know their s**t. And rememberā¦ it could all go away tomorrowā¦ and if thereās anyone to blame itās probably the US. lol