Hi all! Newcomer here. I have tons of storage (around 600TB - coming from Chia) and wondering what would be the best strategy to populate storage with data quickly. I have 2x fiber gigabit connections. My total ingress rate across 4 nodes is about 0.13TB/day on average or 10+ years to fill all my storage. Is this a typical rate or could it improve?
Here is more info on my situation:
I already have 2 fully vetted nodes, approaching to 1 month age (1 per ISP). I currently only show 2TB of space for each, not reached even 1TB utilization yet.
2 more nodes (again 1 per ISP), created roughly a week ago, not fully vetted yet, now resulting 1 older and 1 newer node sharing each connection.
Hardware-wise I am considering consolidating to 2x 16bay QNAP chassis, hosting a node each and resulting 250TB redundant storage each.
Nodes are not even making a dent on my internet bandwidth. By far most traffic comes from us1 and eu1 satellites. Ping to us1.storj.io = 79ms ; ping to eu1.storj.io = 21ms - I am UK based.
ps: I now realize that I may have shot myself in a foot adding 2 more nodes and if that so what would be the quickest way to decom them without causing impact to the network?
My expectation is to get some income to pay for electricity and internet bills. If this is unachievable within say 6 months then I probably will just sell the gear. It would be a shame though as I have grown fond of it during previous projects; and this has been a great learning experience so far. It is also nice to be a part of community! Thank you all for your time and advice!
hi mike, thanks for your reply! That is very pessimistic forecast - much worse than my current ingress rate. Would you mind sharing the source of the stats you used to work this out?
Many here have posted 2-4TB/year/IP as their current rate: you can poke around a bit. So if you’re already at 2TB you’re doing great! Nodes share their own special pool of ingress when they’re new (which can mean they actually get more than older vetted nodes: depending on how many other new-nodes they’re sharing with) - so I hope you’ll continue at your current fast rate.
The main way to speed things up is to have more /24 IPs: as each gets its own share of ingress. Good luck!
Most of the data you receive is deleted over time, so your current ingress rate is not equal to stored space used in a few weeks/months.
The source is an avg of my nodes. They gain this amount for each /24 subnet comparing now and 1 year ago.
Also, there have been several threads discussing the growth rate here - most are actually reporting closer to 2TB/year than what I am seeing on my setups.
The new nodes simply won’t pay as much as the existing nodes due to being younger, so it’s better to provide additional storage to your old nodes while the new nodes age, then more some potential storage capacity back to the new nodes down the line.
Also, you might as well spin up multiple new nodes (more than just two) - worst case, they split the traffic, but it gives you more flexibility in the future so that you don’t have to worry about the warmup period.
Best way would be probably to make a Storj account and upload petabytes of data…
To be more serious: My oldest node is 70 months and has 8.25TB of data.
Surely there are better performing nodes and I guess the largest node is somewhere between 16TB and 20TB after multiple years.
The network data alone are showing that your desired goal to store around 500TB within short period of time, which is around 1.5% of the entire current network data is quite unrealistic.
I cannot advise you but you might be better off in terms of cash flow and costs, to sell the big gear take that money and start with a small setup like 1 or 2 20TB drives on a Raspberry or Odroid or something and see how it is going.
I MEAN: welcome and we appreciate your enthusiasm!
The others’ estimates are about right. Few TB a year. Ingress is split per unique IP /24 address.
I used to do chia on a smaller scale back in the day and storj is waaay different and waaay slower.
storj data gets deleted regularly, including TTL auto delete features (imagine security camera footage) so things will be auto deleting after 7 days, 30 days, whatever.
btw I know It’s another project but I’ve farted around with sia coin hosting, and it is similarly a slow fill exercise.
So anyway, you can probably safely sell off a vast amount of hardware. Keep the storage you need for your real uses and maybe two extra drives for storj.
Thank you all for giving me information and advice. I really appreciate it.
@jammerdan link to Storj network data is most revealing - network is way under-utilized which kind of means only the very fastest, most reliable, oldest nodes are reaping reasonable rewards and the rest of us are fighting over digital scraps which is not nice but okay. challenge accepted! Thank you all!
The extreme skyrocketing of the token did not help either, pushing extreme capacity to the network, which in theory would be nice, but since since the plots in stage one was only used to scan for lottery-ticket-numbers, the demand for each disk was extremely low, leading to some very underperforming storage arrays, if they were suddenly hit with serious customer data.
Overall a super interesting project, that did almost everything wrong in my opinion
Most setups the chia doesn’t recoup the power cost now from keeping the system running. You’d need a very large optimized farming rig and very low electricity costs. For my expensive California electricity it’s not even worth the incremental electricity from just having a single hard running.
I also have some leftover plots that I leave on disks and delete as I need more space, but that’s it.
