Node operator can be a business?

Node Operators
, ,
21

This makes sense, but I do not know how to do it.
Right now, the price for customers is $4/TB stored and $7/TB egress. Even if all of that money went to the nodes, the nodes would get something like $1.4 and $2.5 (because 1TB of customer data takes up 2.76TB on the nodes).
This would require large nodes with lots of egress to make even $50/month.

Another part is the fact that nodes cannot be trusted. There is no way to prove or disprove that multiple nodes are running on the same or separate array, that the node operator has backup power or a backup ISP.

Filecoin does it by having large initial requirements. It seems that there is no point in starting a filecoin node without a full rack of modern servers. I guess thatā€™s one way to make sure that only ā€œseriousā€ people do it. It is not very decentralized then.

Having lots of small nodes probably is better for reliability, but it may indeed look bad (ā€œhey, boss, I think we should use this company to store our files, the files get stored on lots of raspberries in lots of homesā€).

AWS and similar have the advantage that you can run VMs on the same provider and probably pay less to access the data (or at least have it faster). There is no way to run even php on Storj (and sure, Storj is only for storage), but at the same time I am a bit struggling to figure out the use cases for it other than backups.

22

tomorrow storj close buisiness and go away. with your data. who are you going to complain to? and if you are not from the usa? to this forum\internet? :))))

23

We appreciate the feedback and keep it coming! Weā€™re going to go into an increasing level of detail on ideas over the next few months. Weā€™ll be hosting bi-weekly twitter spaces and ultimately publishing a economic whitepaper for external feedback in Q1 next year.

If you can join, feel free to bring questions and suggestions. If you canā€™t join but want to ask questions in advance, weā€™ll publish a way to submit questions in advance so we can cover them. All the twitter spaces are recorded.

Weā€™re really seeing an steady increase in demand right now and weā€™re anticipating needing more storage nodes. We want to make sure we have steady growth but not too much growth ahead of demand so that itā€™s economically rewarding to be a storage node. We also need to ensure storage node operation works for both data centers and individual operators. We also need to make some changes to that the unit economics work out long term for Storj as a satellite operator and for future satellite operators.

Thereā€™s another angle on the demand side. Customers have found the geo-fencing feature to be valuable and have requested a variety of different node selection criteria - high speed, SOCII facilities, more granular geo-fence areas, reduced egress or reduced redundancy - and weā€™re looking at how these request fit into the model.

Weā€™re making our process as transparent as possible, but itā€™s clear weā€™re getting traction. Please keep the ideas coming and join us on Twitter Spaces. We may try other formats, but right now this one seems to be working reasonably well.

10 Likes
24

@serger001 yeah wellā€¦ I got carried away.

I like this forum though. People are responsive.

1 Like
25

That could be Storj 2.0. Storage can be decentralized. Virtual machines, why not?

26

That seems a little optimistic looking at previous months. This month so far is looking a little better than average though. But since previous months were around 600-700GB, Iā€™m not yet going to assume this month is representative.

@DD7 Iā€™ve updated the earnings estimator with more recent traffic data. It looks a little better, but because more data stored also means more data having a chance of being deleted, there is still a theoretical limit as to how much data a single /24 IP subnet could store. Which is currently around 80TB. At that point the amount of deletes and ingress will roughly match and there will no longer be a net growth.
Additionally about 25% of new ingress is deleted within the same monthā€¦ so raw ingress numbers are not representative of node growth.

It can, but with current traffic it will take about 5 years to get there.

The network needs scale, not individual nodes. But when that scale is going to be needed, nodes will start filling up faster and faster. Currently every node with available space gets their share and clearly not more scale on nodes is needed at the moment. Itā€™s simple supply and demand determining this requirement. But for the network to work best, itā€™s a lot better to have more nodes than to have larger nodes.

Also keep in mind for economic incentives that you are competing with pesky node operators like me, who use hardware that mostly wasnā€™t purchased for this purpose and devices that are online anyway. I have basically 0 costs. So do you think you can run a business with the associated costs when you are competing against people who donā€™t have to worry about those costs? Yet without any costs, Iā€™ve made about $3000 since march 2019. Iā€™ll take it. But I also would have taken $1000. Itā€™s basically free money anyway.

Furthermore, Storj currently pays more to node operators than they make running the service. This is fine as long as there is a runway and there is still a substantial runway left. But payouts have to come down at some point to make it long term viable.

When this happens, you will see ingress for nodes with remaining free space go up quickly. This situation kind of resolves itself at that point. It will become much more interesting to start new nodes, existing SNOā€™s who can will expand capacity and new node operators will jump on the now much more profitable bandwagon.
Gotta love classic market effects.

None of this is really relevant. Individual nodes donā€™t matter at all for reliability in the network. Nodes are always going to run on low powered hardware, because they can run just fine on it. Anyone spending more than they have to is making a bad financial decision.
What those larger companies need to know is that the reliability of individual nodes is not important. All nodes are kept to certain standards and disqualified if they fall below those standards. Such that the network can ensure file retention to 11 9ā€™s even if the nodes themselves run on raspberry piā€™s or consumer NAS systems or whatever is available. Thatā€™s the whole point of Storj.

On a per storagenode (/24 subnet) level, Iā€™m seeing this increase mostly since this month. Meaning demand this month is outpacing node availability. While that is encouraging, I tend to not get too excited when itā€™s just a single month. More ingress attracts more SNOā€™s and it may level out again. Keep em coming though. Iā€™ll make sure to expand as needed as long as itā€™s profitable. Just this week I expanded my array with a 20TB HDD and started a new node on the 4TB HDD it replaced in addition to a 3TB HDD my dad no longer needed. So far it has always been worth it to expand, even though I donā€™t buy most of the space I add. That 20TB was expensiveā€¦ but paid for with Storj earnings. As long as I can keep doing that, Iā€™ll make sure to have plenty space available.

Yeah, this place is great. Most people love to share their experience and contribute to a better shared knowledge between node operators. Distributed knowledge on top of distributed storage. Gotta love it. :slight_smile:

4 Likes
27

There are several startups, who trying to achieve that. The main problem, that the operator have an unrestricted access to the customerā€™s processes running on their orchestrator.
So you cannot use it for business, where information have a value.

28

We got lucky, I remember lots of traffic of test data and surge payments of 2x or 3x the current rates.
Also, since I usually HODL, I got lucky that the token price went up.

Someone starting a node right now probably would not be as lucky.

But, in general, I agree, Storj was made so that people with less reliable setups than a datacenter (but not by that much) could provide the service and, of course, this means the costs are lower.

2 Likes
29

Storage is simple because it has a simple abstract API: put a file, get a file, list files, delete. VMs are much more complex, Iā€™ve seen attempts to do so on consumer hardware since 2006 and none worked so far. Customers would need to write software so that it can run regardless of architecture, and you canā€™t usually execute a single job in a distributed way like storage can do with parity.

What could probably work is some prepackaged tools that need to work on bigger data blobs. Iā€™d imagine a distributed data structure server similar to what Redis is. Maybe a distributed grep/sed? Maybe a distributed index into binary files, let say, based on some distributed hash table algorithm? Maybe a distributed log, or a locking mechanism based on Paxos or Raft?

Itā€™s still a lot more work than just focusing on storage, but way easier to do than full VMs, safer for node operators, fitting into hardware that storage nodes already use, and still quite usableā€”for some customers more attractive than just VMs.

30

There are distributed computing platforms, like BOINC, but they are only suitable for jobs that are not latency-sensitive and the node operator can access your data.

31

Yeah, 5x even. I had one month when I made $500 with just 4TB stored (donā€™t get too excited, I doubt weā€™ll ever see something like that again).
That said Iā€™m at 45 months in now and with current network behavior by that time you would still have made over $1800. And ingress does seem to be growing, so you can still make some good money.

Yeah, I managed to sell part of it much higher as well and probably more than tripled that $3K, but Iā€™m not counting that. You could easily also lose, so Iā€™m just going by the amounts that were paid out.

32

I understand how a platform/business can run on such model. Actually itā€™s a brilliant idea: recovering peopleā€™s sunk cost - unused capacity. The quality control is handled at the level of the network, not the nodes. Most of the reliability (or unreliability) of the nodes is already accounted for and priced in. New investment in dedicated hardware just canā€™t beat that price.

If true, then this is the answer. Thank you for enlightening me.

2 Likes
33

Thatā€™s the idea. We suggest to use only hardware which is online anyway, so all costs already paid, and literally any income is a pure profit.
The quality is controlled on the network level, if some nodes did not qualify - they will be disqualified, data recovered using Reed-Solomon erasure codes to other more reliable nodes.
So investments are not required, however, itā€™s your decision anyway, just account all costs.

2 Likes
34

Reliability and performance are also factors. Iā€™m a (potential) customer, and while the price is attractive Iā€™m still using B2 for my main storage.

I uploaded a few GB of test data some moons ago and went to download it now, initially it very nearly maxed out my 850Mb/s internet connection, but the transfer dropped to well below 100Mb/s for a while, spiked back to 300Mb/s for a little bit, and then fell to below 50Mb/s and stayed there.

I ran it a few more times, it is reasonably consistent from run to run, right around 84Mb/s so letā€™s use that.

Pulling the same block of test data from B2 has a nice vaguely even rate of 500Mb/s across the entire set.

In a backup / disaster recovery scenario, the difference between being able to restore my 43.2TB at 500Mb/s (about a day a bit over a week) vs 84Mb/s (52 days) is the difference between ā€œOkay, that suckedā€ and hiring a bankruptcy trustee. The monthly cost 172.80/month vs 221/month doesnā€™t even matter.

Back-of-the-napkin-math if the only consideration was the amount of time Iā€™d be paying employees to sit around waiting, my break-even is to have a disaster every 15 few years per employee. Obviously that isnā€™t all that matters as most businesses cannot just stop serving customers for 2 months and then bounce back, so the long-term impact would be far more severe.

Iā€™m ignoring transaction costs, download costs, everything else because it is all a rounding error.

Is backups the only use-case for Storj? Probably not. But it is my main use for ā€œdumbā€ cloud storage so thatā€™s what Iā€™m using as a starting point.

43.2TB isnā€™t a massive number either, feel free to multiply everything by whatever figure you like, it only makes the problem worse (either the restore time gets astronomical, or the savings drop to the point that it isnā€™t worth the time it took to do the math).

I like the concept of Storj, and Iā€™m considering moving a replica of my personal backup just to support the concept (and save a few dollars) since restore speed is less relevant.

EDIT: Fixed a math error inline, see comments below for what I dumbed.

2 Likes
35

To optimize performance, I recommend you read How to Hod Rod File Transfer Performance on Storj DCS - Storj DCS Docs as well as the Hot Rodding blog posts linked at the end of the article.

36

Iā€™m using rclone, tried --transfers from 8 to 32. I havenā€™t seen anything else that is worth optimizing for downloads. The files are a real-world mixture, although if I switch to using Storj for real use Iā€™ll look into optimizing around file sizes, concurrency, and similar.

The machine itself is an Intel i9-10900K 3.7GHz 10/20 core, 64GB of RAM, writing to a Samsung 980 Pro SSD (running on gen3, maxes out around 3450MB/s).

This SSD has 113GB SLC and sustains 1500MB/s writes to TLC (when SLC is full), which well exceeds my internet speed.

When pulling from sufficiently fast servers over the internet (e.g., from my Cloudflare Enterprise account with content already cached at the local POP) my internet connection is the limiting factor, and across the LAN I can max out ethernet sustained for 100s of GBs.

Both the variability of the speed and the actual speed are concerns for business use. For personal use Iā€™m more tolerant.

37

Greetings @thedaveCA my name is Dominick and Iā€™m the author of the hot rodding document above. Your workstation is very capable so we should be able to get some great speeds. When we hit a limit as you noted it should be the ISP.

Ideally, backups are packaged into relatively large archives via something like restic + rclone. For now, letā€™s not overcomplicate and just focus on moving data fast.

Assuming your backup data is already flowing over we can focus on fast retrieval. Our uplink is ideal for this.

Lots of files <64M
Moving 48 files in parallel
uplink cp --recursive sj://bucket/path /tmp --recursive --transfers=48

A 10G archive
Moving 48 segments in parallel requires the file is (64M*48=3,072M). A 1G file has 16 segments so a parallelism of 16 can be used per 1G.
uplink cp sj://bucket/10G.img /tmp --parallelism 48

A mix of both
Moving 8 files in parallel and up to 8 segments at a time is at peak the same load as 48 parallel or 48 segments. In practice, this wonā€™t be as fast as separating small and large files but its a good start. Scale --transfers up more if more small files and conversely scale down transfers and up parallelism if larger files. If you were moving 512MB files the ideal setting for a target of 48 would be --parallelism 4 --transfers=12.
uplink cp --recursive sj://bucket/path /tmp --recursive --parallelism 8 --transfers=8

For fast retrievals in your environment Uplink CLI is the way. Happy to work with you to demonstrate the best possible RTO. We commonly see speeds to a single endpoint (with a supporting network) exceed and maintain 300MB/s (2,400Mb/s).

Excited to help!

P.S. I can also help with rclone but figured id share the best way first given the strong client compute enviroment.

6 Likes
38

The correct calculation is:

500Mb/s ā€”> 8 days
vs
84Mb/s ā€”> 48 days

Sorry, canā€™t help myself correcting people. The Hod-Rod thing aside, your conclusion still holds though.

BTW, really appreciate you sharing real-world statistics.

2 Likes
39

You are absolutely correct.

Iā€™m digging through my browser history to see what I typed to get that one wrong. A ā€œbyteā€ when I meant ā€œbitā€ somewhere? I donā€™t see it, but the evidence is right there.

Thanks for pointing it out, Iā€™ll take being right eventually rather than being wrong forever.

40

In the real world Iā€™m using restic to backup to a local NAS, and then I rclone that to the cloud(s) that are in favour.

I have small pack sizes with restic which isnā€™t going to do me any favours, but it was the right decision when it was made. I have way more data, and restic itself has removed bottlenecks between then and now. Oh and I have a lot more RAM on the various machines running restic now, so Iā€™m open to experimenting for the future.

I also have a machine that generates tons of small files with a high turnover rate, in the past this meant every prune was rewriting a huge portion of my repository and it was a struggle to get them uploaded. I am considering splitting it to a separate repository and just not pruning it.

Upload bandwidth was a factor back then too, I was quite limited for a while in the past due to an ISP misrepresenting their service availability and we were committed before we found out. Ugh. Iā€™ve got 125Mb/s upstream now and symmetric 1Gb/s coming in a matter of weeks so aside from it being inefficient I donā€™t think I need to care about rebuilding and re-uploading packs at this point.

What would you recommend for restic pack sizes as it applies to Storj? Any matching rclone settings specific to that decision? Iā€™ll run through the hotrodding article again too.

My gut says I should stay just under the 64MB segment size to balance resticā€™s recommendations and avoid having files get split into multiple segments, then tune the --transfers knob? Iā€™m also thinking --order-by size, mixed to balance out the respective bottlenecks of different file sizes? This helps with mixed sets on B2, anyway.

Iā€™ll build a new set of test files to match the recommendation and re-test B2, Storj, possibly Wasabi, and Cloudflareā€™s R2.

While Iā€™m thinking about it, is there a minimum amount of data I should be using for testing? I donā€™t want the longtail at the end to have a disproportionate impact on the average, nor rcloneā€™s scanning of files in the start. Iā€™m usually comfortable with about 5 minutes of transfer (whatever volume of data that ends up requiring).

I would like to stay with rclone because my scripts are cloud-agnostic and adding or dropping a cloud is just a line in a config file. In a disaster recovery situation using another tool to download in bulk would be fine, so Iā€™ll compare uplink, but I donā€™t want to outright replace rclone for daily use.

Letā€™s assume a clean slate, if this works Iā€™ll take the opportunity to migrate the old snapshots to a new repository to split out the ā€œspecialā€ set of small files, get preferred pack sizes and compress all in one shot, then use it to stress-test the 1Gb/s line.