I understand how a platform/business can run on such model. Actually it’s a brilliant idea: recovering people’s sunk cost - unused capacity. The quality control is handled at the level of the network, not the nodes. Most of the reliability (or unreliability) of the nodes is already accounted for and priced in. New investment in dedicated hardware just can’t beat that price.
If true, then this is the answer. Thank you for enlightening me.
That’s the idea. We suggest to use only hardware which is online anyway, so all costs already paid, and literally any income is a pure profit.
The quality is controlled on the network level, if some nodes did not qualify - they will be disqualified, data recovered using Reed-Solomon erasure codes to other more reliable nodes.
So investments are not required, however, it’s your decision anyway, just account all costs.
Reliability and performance are also factors. I’m a (potential) customer, and while the price is attractive I’m still using B2 for my main storage.
I uploaded a few GB of test data some moons ago and went to download it now, initially it very nearly maxed out my 850Mb/s internet connection, but the transfer dropped to well below 100Mb/s for a while, spiked back to 300Mb/s for a little bit, and then fell to below 50Mb/s and stayed there.
I ran it a few more times, it is reasonably consistent from run to run, right around 84Mb/s so let’s use that.
Pulling the same block of test data from B2 has a nice vaguely even rate of 500Mb/s across the entire set.
In a backup / disaster recovery scenario, the difference between being able to restore my 43.2TB at 500Mb/s (about a day a bit over a week) vs 84Mb/s (52 days) is the difference between “Okay, that sucked” and hiring a bankruptcy trustee. The monthly cost 172.80/month vs 221/month doesn’t even matter.
Back-of-the-napkin-math if the only consideration was the amount of time I’d be paying employees to sit around waiting, my break-even is to have a disaster every 15 few years per employee. Obviously that isn’t all that matters as most businesses cannot just stop serving customers for 2 months and then bounce back, so the long-term impact would be far more severe.
I’m ignoring transaction costs, download costs, everything else because it is all a rounding error.
Is backups the only use-case for Storj? Probably not. But it is my main use for “dumb” cloud storage so that’s what I’m using as a starting point.
43.2TB isn’t a massive number either, feel free to multiply everything by whatever figure you like, it only makes the problem worse (either the restore time gets astronomical, or the savings drop to the point that it isn’t worth the time it took to do the math).
I like the concept of Storj, and I’m considering moving a replica of my personal backup just to support the concept (and save a few dollars) since restore speed is less relevant.
EDIT: Fixed a math error inline, see comments below for what I dumbed.
To optimize performance, I recommend you read How to Hod Rod File Transfer Performance on Storj DCS - Storj DCS Docs as well as the Hot Rodding blog posts linked at the end of the article.
I’m using rclone, tried --transfers from 8 to 32. I haven’t seen anything else that is worth optimizing for downloads. The files are a real-world mixture, although if I switch to using Storj for real use I’ll look into optimizing around file sizes, concurrency, and similar.
The machine itself is an Intel i9-10900K 3.7GHz 10/20 core, 64GB of RAM, writing to a Samsung 980 Pro SSD (running on gen3, maxes out around 3450MB/s).
This SSD has 113GB SLC and sustains 1500MB/s writes to TLC (when SLC is full), which well exceeds my internet speed.
When pulling from sufficiently fast servers over the internet (e.g., from my Cloudflare Enterprise account with content already cached at the local POP) my internet connection is the limiting factor, and across the LAN I can max out ethernet sustained for 100s of GBs.
Both the variability of the speed and the actual speed are concerns for business use. For personal use I’m more tolerant.
Greetings @thedaveCA my name is Dominick and I’m the author of the hot rodding document above. Your workstation is very capable so we should be able to get some great speeds. When we hit a limit as you noted it should be the ISP.
Ideally, backups are packaged into relatively large archives via something like restic + rclone. For now, let’s not overcomplicate and just focus on moving data fast.
Assuming your backup data is already flowing over we can focus on fast retrieval. Our uplink is ideal for this.
Lots of files <64M
Moving 48 files in parallel
uplink cp --recursive sj://bucket/path /tmp --recursive --transfers=48
A 10G archive
Moving 48 segments in parallel requires the file is (64M*48=3,072M). A 1G file has 16 segments so a parallelism of 16 can be used per 1G.
uplink cp sj://bucket/10G.img /tmp --parallelism 48
A mix of both
Moving 8 files in parallel and up to 8 segments at a time is at peak the same load as 48 parallel or 48 segments. In practice, this won’t be as fast as separating small and large files but its a good start. Scale --transfers up more if more small files and conversely scale down transfers and up parallelism if larger files. If you were moving 512MB files the ideal setting for a target of 48 would be --parallelism 4 --transfers=12.
uplink cp --recursive sj://bucket/path /tmp --recursive --parallelism 8 --transfers=8
For fast retrievals in your environment Uplink CLI is the way. Happy to work with you to demonstrate the best possible RTO. We commonly see speeds to a single endpoint (with a supporting network) exceed and maintain 300MB/s (2,400Mb/s).
Excited to help!
P.S. I can also help with rclone but figured id share the best way first given the strong client compute enviroment.
The correct calculation is:
500Mb/s —> 8 days
vs
84Mb/s —> 48 days
Sorry, can’t help myself correcting people. The Hod-Rod thing aside, your conclusion still holds though.
BTW, really appreciate you sharing real-world statistics.
You are absolutely correct.
I’m digging through my browser history to see what I typed to get that one wrong. A “byte” when I meant “bit” somewhere? I don’t see it, but the evidence is right there.
Thanks for pointing it out, I’ll take being right eventually rather than being wrong forever.
In the real world I’m using restic to backup to a local NAS, and then I rclone that to the cloud(s) that are in favour.
I have small pack sizes with restic which isn’t going to do me any favours, but it was the right decision when it was made. I have way more data, and restic itself has removed bottlenecks between then and now. Oh and I have a lot more RAM on the various machines running restic now, so I’m open to experimenting for the future.
I also have a machine that generates tons of small files with a high turnover rate, in the past this meant every prune was rewriting a huge portion of my repository and it was a struggle to get them uploaded. I am considering splitting it to a separate repository and just not pruning it.
Upload bandwidth was a factor back then too, I was quite limited for a while in the past due to an ISP misrepresenting their service availability and we were committed before we found out. Ugh. I’ve got 125Mb/s upstream now and symmetric 1Gb/s coming in a matter of weeks so aside from it being inefficient I don’t think I need to care about rebuilding and re-uploading packs at this point.
What would you recommend for restic pack sizes as it applies to Storj? Any matching rclone settings specific to that decision? I’ll run through the hotrodding article again too.
My gut says I should stay just under the 64MB segment size to balance restic’s recommendations and avoid having files get split into multiple segments, then tune the --transfers knob? I’m also thinking --order-by size, mixed to balance out the respective bottlenecks of different file sizes? This helps with mixed sets on B2, anyway.
I’ll build a new set of test files to match the recommendation and re-test B2, Storj, possibly Wasabi, and Cloudflare’s R2.
While I’m thinking about it, is there a minimum amount of data I should be using for testing? I don’t want the longtail at the end to have a disproportionate impact on the average, nor rclone’s scanning of files in the start. I’m usually comfortable with about 5 minutes of transfer (whatever volume of data that ends up requiring).
I would like to stay with rclone because my scripts are cloud-agnostic and adding or dropping a cloud is just a line in a config file. In a disaster recovery situation using another tool to download in bulk would be fine, so I’ll compare uplink, but I don’t want to outright replace rclone for daily use.
Let’s assume a clean slate, if this works I’ll take the opportunity to migrate the old snapshots to a new repository to split out the “special” set of small files, get preferred pack sizes and compress all in one shot, then use it to stress-test the 1Gb/s line.
Great post @Dominick !
I’m wondering though, why are these two separate settings? Couldn’t you just have 1 setting to determine the number of transfers and let uplink figure out whether it should transfer segments from one file or multiple files? This would prevent the end user from having to do that balancing game you describe and would also deal more flexibly with varying file sizes in a single cp operation.
I can imagine cases where a user might have a preference, but I suspect it doesn’t matter in the general case.
I think it should be able to be more automatic than that though. An option to limit RAM and have it do as many up-to-64MB segments as it can at the same time? Or a limit to the maximum number of segments at a time (and let it move sequentially through the files as needed to max out the segment count), so a 100GB file might consume all 48 slots, as it gets to the last 64MB segments it’ll leave some free slots that could start transferring the next file.
This assumes it can dynamically assign parallelism and transfers, which may not be possible, but even if not it could hold off on new files (effectively putting some of the --transfers into a waiting state) when the number of segments to be processed at a time is reached, allowing the count to bounce around the target (which should always be better than having numbers consistently too high or consistently too low on a mixed set of files).
I considered suggesting this, but this could mean many thousands of parallel transfers if the files are small, which could cause connection issues. You’d probably need to set both RAM and transfer limits then.
That’s exactly what I was suggesting. I don’t understand why the user now has to set separate limits. Whether the 48 segments are all 1 file, 2 files or 48 files has pretty much the same impact on RAM, CPU and network connection. This really should just be done automatically. Especially since any combination could cause peaks now. Right now if you set both settings to 8, it would fluctuate between 8 and 64 transfers depending on file sizes. Yet if your resources can handle 64 it should always be using 64.
It may not be how it’s built currently, but the uplink determines which transfers to start, so it could definitely be made to work that way. What you described after this sentence I believe is how it already works if you only set the transfers parameter.
I retract my previous reply. Your statement is NOT true. At least, according to Storj white paper page 11. And I quote:
“It should be economically advantageous to be a storage node operator not only by utilizing underused capacity but also by creating new capacity, so that we can grow the network beyond the capacity that currently exists.”
The purpose of Storj is not just for existing spare capacity but also NEW capacity. Node operators should then be incentivized to invest in new hardware. Meaning: the new investment should yield a fair amount of ROI.
As an aside, another quote:
“… it is required that storage node operators have sufficient incentive to maintain reliable and continuous connections to the network.”
Thus my description of preferable node operators stands:
“c. fairly paid decentralized operators who are incentivized to invest for and maintain a certain quality of service, and NOT to fake decentralization”
If Storj stays true to its own white paper, it must makes changes to make sure that node operators are incentivized to make NEW investment.
The new capacity is added with more unused or used for something else online hardware.
For example, the mining rigs, game servers, home servers, even rented servers if they are not utilized in full. Even existing operators can just add another disk, which is just collect the dust on the shelf otherwise.
So, new capacity ≠ buy hardware.
“added with more unused” still falls under “underused” category. I think your interpretation is not what the white paper meant at all when it says “creating new capacity” and “be economically advantageous”.
already paid and own → underused
Also, at the current rate, it takes years to fill 4TB HDD, of which average life is just 3-4 years. You earn a few dollars, if not cents, a months on average. How “economically advantageous” is that?
it’s more than zero anyway. If you did not buy a dedicated hardware, and did not invest in to make it 24/7 online, any income will be a profit.
The whitepaper is said nothing about investments, so creating new capacity literally mean - add free space and bandwidth to the network and be awarded for that, it does not exactly mean to buy a new hardware, you may use used, refurbished or partially used for other puproses.
The fact is, you may run storagenode even on the router: Running node on OpenWRT router?
Wow… none of that is even close to true. It’d take about 8 months to fill up 4TB and most of that is because you lose months to vetting. This month my ingress was almost 0.9TB on a vetted node (not representative of an average month yet, but it is increasing over time currently). So no, it won’t take years.
Modern HDD’s have an annual failure rate of about 1-2% for the first 5 years. It goes up slowly after that, sure. But the average lifespan of an HDD is well over those 5 years. Most of mine last over 10 years and I’m even running 3 that are now over 15 years old. So no, not a lifespan of 3-4 years.
With 4TB filled you earn about $13 a month. Not cents. The first year a lot of time is spent filling up the HDD, so you make a total of about $80, with subsequent years earning you roughly $160 per year. Buying a 4TB HDD costs about $80. So you have earned your costs back in a single year. It is currently economically viable to expand as long as you don’t have additional running costs (hardware already online anyway). Even if the HDD only lasts 5 years, that’s an ROI of 900%.
well us long standing node operators would have been making new investments… we definitely wouldn’t have started off with a 100TB space on storj… probably started with a 4TB or at best 8TB running a node… then as it gets nearly filled, we either find another underused hdd or buy a new one to provide more disk space
i would say we are pretty well incentivized to invest over a long term view…
it won’t take that long to fill a 4TB currently… probably 1.5 years at most (pretty pessimistic estimate actually), on a single IP… ROI on just a brand new HDD will be cleared within 1.5 years… second hand HDD within a year or less… put into consideration of the bad disk etc, 2 years ain’t that bad for a 100% return?
the bigger the initial sizing, the longer the ROI… can’t do big bang with storj, even if you can get firesale HDD from chia farms shutting down… probably easier to do it with crypto mining where assets can be utilized immediately
Yeah, I failed to recognize that. Your 0.9TB/month reconciles with @Knowledge 's comment.
This does not reconcile well with what I gather. Survival rate starts to dive in year 4. See chart from Blackblaze:
But, yeah, individual cases may differ.
So income from usage and egress is approximately 50/50? Perhaps my estimate was overly pessimistic about egress.
Wow… now future looks a lot brighter. Thanks for sharing.
