Discussion on scaling the network - a multi-node SNO tier proposal

[Humble_Fenix]

Hi all,

Let me reopen the issue of multi-node setups that do not strictly adhere to the ToS of Storj. I understand it has been discussed several times but it appears to me that no solution was found and given the high number of nodes behind hosting services the issue of scaling the network is not properly addressed.

I want to be constructive and propose some ideas that could be beneficial for both Storj as a company and SNOs. First let me give a brief background:

I have been running a small node for many years but never really considered it more than a hobby and nice experiment contributing the ideals of decentralisation. After all an 8TB node pays a couple bucks a month. This was reflected on my attitude towards it. I rarely checked on it, it was sometimes down for several days without me even noticing. Running a single node is simply not worth the time to pay serious attention.

Coming from the failing world of Chia farming the idea of converting one of my smaller rigs with 10 HDDs to a ‘Storagenode Rig’ crossed my mind. To overcome the 1 IP limitation I could route each storagenode via a VPS running a VPN server. I believe there are others out there who already do this.

There is no need to argue whether running 10 storagenodes through 10 different VPS/VPNs is in line with ToS (it is clearly not) or whether ToS are up to date (clearly not). My goal is to discuss the trade-offs between decentralisation, performance and incentives and possibly start a discussion that actually leads somewhere.

On the incentives. There are clear incentives to engage in such practices. A 20 TB node will pay around 30-35 USD / month. The costs including the rent for the VPS and electricity will not exceed 6-8 USD. The ROI is pretty decent and if run until the HDD dies it would pay off many times over. I assume I am not the only one who figured this out since more than half of all nodes are associated with an IP address that is identified as ‘hosting’. I doubt those are college students who use VPNs as a workaround for port forwarding issues. Simply put if there is money to be made the market will make sure that all opportunities are exploited.

On performance. If the goal of Storj is to grow both its storage capacity and stored customers` data to a meaningful scale (Exabyte?) then relying on an army of SNOs around the globe with 4-12 TB single nodes looks a bit naïve to me. I understand Storj introduced a program for professional storage providers, however that is not aimed at SNOs, not even Chia farmers with PBs of capacity. Instead it is a path to become one of the many of Cloud storage providers which is completely opposite to the original objectives of Storj. In my view keeping the focus on SNOs with 1 drive per IP/physical location is a mistake from the longer perspective.

There should be a way – a tier for semi-professional SNOs – that allows scaling of SNOs while complying with the ToS and not compromising the level of decentralisation. I believe that, with the majority of nodes being routed through hosting IPs this is the only sensible way forward.

This tier could and should provide incentives to run larger capacities by SNOs on a semi-professional basis. Fees for stored data TB should be lower, even 0 for egress, uptime requirements higher and perhaps a collateral system with slashing for outages could be considered. There are naturally other parameters that could be considered too, e.g. a maximum number of nodes run behind 1 IP, regular bandwidth checks, etc. or KYC the SNO. The goal is to incentivise the semi-professional SNOs to strive for excellence while still earning decent money in a compliant way.

It could even reduce the overall costs for Storj as a company. I, personally, would be happy to receive half of the current earnings if I could do it without violating the ToS and scale to capacity that could justify the time and resources put into the project. To avoid abusing the system nodes with hosting IPs could not participate in this tier.

On decentralisation. It is hard to assess the current state of decentralisation of the network as more than half of nodes are routed through hosting providers. My guestimate is that the majority are in multi-node setups, since the financial incentives to do so are very compelling. Creating a compliant pathway for scaling up SNOs could contribute to higher transparency. At least the satellites would could see which nodes are at the same location and could make sure that the same data is not being sent to these nodes. Furthermore, less amount of data would be routed through hosting providers thus reducing risks and improving the overall performance of the network.

Please correct me if any of my assumptions are incorrect, especially on the number of nodes run through hosting services. I am really interested to hear what the plans for scaling the network are.

Happy to hear your thoughts.

[Knowledge]

Thanks for the well thought out and interesting ideas. There is a lot to think about here. Nicely done.

Speaking for myself, I don’t see the advantage to Storj to do this. They have plenty of capacity right now. If that capacity gets heavily used, SNO’s will add more as drives fill up. We see capacity increasing all the time.

I think what you want is to remove the subnet filter, but I don’t think this will gain you much additional data because you will still be sharing ingress with all the other nodes around the world. The limitation is customer use and total nodes, and not really the filter. The data center side has a limited number of operators which provides them (in theory) with more data.

In my experience, regardless of cost, there are SNO’s that will flip a switch and turn on a thousand nodes simultaneously if the subnet filter were removed. Some person who works at a company with an OC-192 setup will bogart the entire network. The home user trying to add a few extra nodes will probably end up with less data and not more.

[Roxor]

@Humble_Fenix

If the goal of Storj is to grow both its storage capacity and stored customers` data to a meaningful scale (Exabyte?)…

I’ve never heard Storj state raw unused capacity was a goal for its own sake. As you come from Chia, you know that space is out there (as Chia has 1000x the space Storj customers have filled) … if it’s needed and will be paid for. You don’t have to chase it.

@Humble_Fenix

There should be a way – a tier for semi-professional SNOs – that allows scaling of SNOs,

Why? Storj has been running for years… and only recently, barely managed to use 50% of the available space. With over 30PB unused out there… why do you think semi-professionals need an incentive?

There’s more free space than you can shake a stick at! Available capacity has never ever Ever 3vah been an issue. If anything… they should cut SNO payouts again to gain some margin and discourage some of them .

[Humble_Fenix]

Thanks for your thoughts. The premise I am working with is that the majority of the network is routed through hosting service providers. There is a huge incentive to do that and currently that is the only (although not compliant) way to scale for SNOs. This is purely an assumption from my side, please correct me if I am wrong on this. I would really appreciate if someone could confirm or debunk as this is detrimental to the level of decentralisation which is in the very core of the service Storj promises to provide to their customers (even though they might not care where their data is stored as long as it is safe).

However, if this assumption is correct, I believe ignoring it and silently tolerating ToS violation is not the right approach. It reduces transparency since the same data could be sent to the same machine with multiple nodes and compromises decentralisation. It also puts honest SNOs into disadvantage. I agree that the goal is not increasing unused network capacity but to meet demand for capacity with high quality storage space that meets the expectations.

More than half of all nodes has a hosting service IP. If Storj decided to vigorously enforce its ToS that could seriously disrupt the network. I think the only way to get rid of the VPS/VPN routed nodes is provide incentives against this practice. Reducing the payouts to a level that routing a 20 TB node through VPS is not profitable anymore could be effective. However, that would require a price around 0.5 USD/TB stored data and possibly 0 for egress. That would certainly lead to a strong reduction of SNOs and lower incentives for those who stay to care for the quality of their service.

Regarding Chia, yes, there are tens of exabytes of data in Chia and other PoST chains but it would not move over to Storj as single nodes. If some of the Chia folks really want to move over, they will certainly go with the VPN/VPS-multi-node setup. I am not implying that they should move over to Storj.

[s-t-o-r-j-user]

Hi,

They entirely decoupled, I mean the management of the company. This is my current understanding. As for now, I just can’t find any other explanation. Regarding the fundamentals of your posts, I generally agree with your point of view, albeit with a few important exceptions.

Firstly, I am uncertain whether running a node over a VPN violates the Terms of Service (ToS). Some configurations utilizing VPNs are well-documented on this forum, including all details regarding storagenodes and wallets, and the company has not raised any objections.

Secondly, I might hold a different perspective regarding remuneration and return on investment (ROI) then you. It shouldn’t solely cover the costs of HDD and electricity, as commonly believed and promoted here. Other expenses, such as CPU power, network bandwidth, etc., should always be considered, to the extent that they allow for compensation for depreciation and future investments. Future investments are vital to maintaining network performance at an optimal level. In addition, there are many other costs associated and I am never for lowering my very own incentive, thus I allow myself to strongly disagree with you on this point.

Thirdly, with all due respect, despite my appreciation for democracy, I am somewhat skeptical about decentralization. Though I’m uncertain why, it seems to me that it’s only effective in the early stages, gradually becoming more centralized over time. One example pertinent to this forum might be ZK Sync, but there are numerous others.

Fourthly, I really do not think they are ready for a serious scaling. It looks that there are some technical inconsistencies. On top of that, to scale, the business model requires symmetrical balancing of supply and demand side, which is not that easy.

Now, while I appreciate the concept underlying the Storj Network and generally admire the Community, I harbor significant criticism toward the company. Some claims, such as those related to reliability and carbon footprint, appeared misplaced since my very first post on this forum (probably around one year ago). It’s regrettable because each time I engage in this forum, I encounter this perspective. Furthermore, from a business and management standpoint, I believe this is a detrimental situation. It appears that company members, including management, operate within an artificial environment, potentially leading to suboptimal decisions.

Having expressed these concerns, the question still remains open: What is the proposed solution? And if aspiring to become the pro, why not write to them directly? Who knows, maybe they will be accommodating?

Actually, this could be a viable way forward, until they can maintain the free spirit, I can even forgive this Amway and Avon style of marketing, however, if they cannot, I would have to say, I am sorry.

Cheers.

[Toyoo]

@Humble_Fenix

More than half of all nodes has a hosting service IP. If Storj decided to vigorously enforce its ToS that could seriously disrupt the network. I think the only way to get rid of the VPS/VPN routed nodes is provide incentives against this practice.

How do you plan to recognize whether a given node is actually hosted on the hosting service, as opposed to being just routed through it? I used to operate a node directly on Hetzner hardware (a dedicated server with plenty of unused disk space—it was oversized for our purpose), and I believe this is a valid, T&C-compliant way of hosting a node.

[Roxor]

Saying a bunch of nodes appear on ‘hosting service’ IPs… is completely normal. They’re just another service on the Internet. It’s in no way a problem, and Storj has been operating fine with those configs for years. And as @Toyoo noted: knowing an IP tells you nothing about where a node may be running. Even two nodes sharing the same IP, on two sequential ports… can have those two nodes in servers on the opposite sides of the planet.

I think we’re going to be rehashing the rehash of rehashed multi-node discussions over the next few weeks: as Chia’s 100000 node operators are facing an imminent halving of rewards. They’re going to descend on any other paying storage-related project like a plague of locusts: desperate to find a new home for their mountain of HDDs

I can’t wait for the first Chia refugee asking “How long will it take to fill my 2PB of space?”

[Humble_Fenix]

Yes, this would be impossible. It is up to the leaders of Storj whether hosting (not routing) a node in a datacentre is a practice that is in line with the ethos of decentralisation. I don´t think it is profitable to host a node with cloud providers, perhaps it was before they started reducing fees, so I assume the large majority of nodes with hosting IPs are circumventing ToS.

[s-t-o-r-j-user]

It is “profitable” until you are oversubscribed, however, I don’t think it is a viable business model. Just wanted to say that in general, I wish you best with your proposal. Cheers.

[Alexey]

@Humble_Fenix

the issue of scaling the network is not properly addressed.

what issue? Do you see any?
https://storjstats.info/d/storj/storj-network-statistics?orgId=1

@Humble_Fenix

The ROI

Please try to understand: we recommend to use what you have now and what be online anyway with Storj or without. All investments are highly not recommended.

@Humble_Fenix

4-12 TB single nodes looks a bit naïve to me.

This is easily solved by incentive. If there is a demand - SNO will add storage as needed. We have a lot of SNO with unused free space, it’s just not added, because there is no demand. And there are a lot of chia miners…
I think that it will balance itself, no need to introduce any artificial demand.

@Humble_Fenix

a tier for semi-professional SNOs

We have it, it’s called “Storj Select”:

Introducing Storj Select: Revolutionizing Cloud Storage

@Humble_Fenix

However, if this assumption is correct, I believe ignoring it and silently tolerating ToS violation is not the right approach.

I believe, it should be solved by a protocol not a strict ToS. However, we have what we have now. But please do not think that it would not be addressed.

[s-t-o-r-j-user]

tHeY dO,
sO,
tHeY dO hAvE
a HiGhLy tEcHnIcAl fOrUm,
aS wElL,
a sMaLl CaRbOn fOoTpRiNt,
lOw LeTaNcY,
eXtRemElLy rElIaBle
aNd
FuLlY dEcEnTrAlIzEd nEtWoRk aS
tHeYr vErY oWn
hIgHeR aBsTrAcTiOn lAyEr
vIsIt cArD

and

a fully transparent
ToS and
a very raliable technology
and a business model
with a unique competitive advantage

Shell we continue @Alexey?

All in all,
its not that bad,
I am just smiling a bit.
Hope you do not mind.
My last post on the forum.
Will not take more of your time.
Promise! :- )

You really do have.
:- )

[Alexey]

Do you have a problem with any of it?

[s-t-o-r-j-user]

Well, for example, my additional carbon footprint is relatively negligible, albeit slightly higher on the switches. However, in the case of expansion, it would become a very different subject. Moreover, I do not associate myself with your economic entity. Nevertheless, should I be a public face of your undertaking, an investor, a member of the management board, or an employer, I would probably be concerned about all of those things, with an emphasis on the carbon footprint.

[Alexey]

The idea here, that this carbon footprint already exist. If you would use for Storj the existing running HW which will be online anyway, the increasing in footprint would be negligible (close to zero) in a comparison when you buy/run equipment exclusively for Storj or build a datacenter.
Technically even running a node in the existing data center should not noticeable increase a carbon footprint.

[andrew2.hart]

@Knowledge

my experience, regardless of cost, there are SNO’s that will flip a switch and turn on a thousand nodes simultaneously if the subnet filter were removed.

I think this why kyc sno is suggested.

[Alexey]

Yes, but it’s implemented only for Storj Select: Introducing Storj Select: Revolutionizing Cloud Storage
I do not see any advantages to introduce a new tier outside of this program (btw, I do not like it, but it’s my opinion. As GDPR and other limitations from the Governments, they should regulate everything except internet - it will regulate itself, just don’t interfere… offtopic).

[andrew2.hart]

@Roxor

can’t wait for the first Chia refugee asking “How long will it take to fill my 2PB of space?”

We’re already here and offering (edit : offered) 50c per TB. I think storj wants 300,000 operators with a few TB each instead. It makes sense as they would have to track large operators for tax and also it’s better for bandwidth.

So if you want to make significant money there is cheating by vpn or bgp.
You can still earn a reasonable (but small) amount without cheating

[s-t-o-r-j-user]

@Alexey

The idea here, that this carbon footprint already exist. If you would use for Storj the existing running HW which will be online anyway, the increasing in footprint would be negligible (close to zero) in a comparison when you buy/run equipment exclusively for Storj or build a datacenter.
Technically even running a node in the existing data center should not noticeable increase a carbon footprint.

I’m sorry to say this, but it wasn’t my point. It seems you still deeply misunderstand the reason the author initiated this particular thread. And I still sustain that there may be something wrong on the management side if such a situation is occurring at all. I’m sorry again, however, I just simply can’t commit more time for this discussion. I wish you best.

[Alexey]

I’m sorry to say that you completely ignored what I said earlier.

[Pentium100]

@andrew2.hart

I think storj wants 300,000 operators with a few TB each instead.

It seems so, but with rates so small, it becomes pointless to run a node to get something like $2/month.

@Humble_Fenix

Fees for stored data TB should be lower, even 0 for egress, uptime requirements higher and perhaps a collateral system with slashing for outages could be considered.

So, what would be the incentives to do this? If someone uses VPNs to bypass the current restriction, why would he choose to reduce his payouts by becoming semi-pro? Finding out who is running multiple nodes in the same location with VPNs would be difficult, so the ToS condition is difficult to enforce.
I have two uplinks and in theory I could run a second node on the same server and Storj would have hard time proving that the two nodes are on the same server and the same drives. With VPNs I could run even more nodes.
My second uplink is slow (100mbps download 10mbps upload) so I just use it as a backup.