Let’s fork this ToS discussion thread into it’s own topic please. @Alexey
Yes, I know this, agree with this, and understand the reasoning behind this; but if you go solely by the ToS, it’s not obvious at all.
This is what current ToS say on the subject: (Supplier Terms & Conditions, Last updated: May 24, 2024. BTW-- did anyone proofread it? What’s going on with numbering and repeated indexes all over the place? Did a human proof-read it?)
Restrictions
Operate more than one (1) Storage Node behind the same IP address;
This does not forbid running the 28 nodes on a singe server if you have 28 public IP addresses.
Side node
Without derailing this topic into ToS discussion, what is this:
Disconnect a Storage Node or otherwise purposefully render it “offline” for any reason other than Company-required maintenance or Storage Node Software updates;
How is this reasonable let alone enforceable? I will reboot and maintain my server on my schedule, not on STORJ’es.
I’m very disappointed by this long awaited ToS update that addressed absolutely no shortcomings that were discussed on the forum; how can Storj expect operators to take it seriously if storj themselves don’t seem to care enough to at very least proofread the legal agreement…
Suggestion:
Replace
‘[Don’t] Operate more than one (1) Storage Node behind the same IP address’
with
‘[Don’t] Operate more than one (1) Storage Node in the same physical location, unless their public addresses are on the same /24 network’
Could you please explain what distance is required to be not in the same physical location? If it requires to be not in the same room or building I would guess almost all SNOs violating this rule.
I’d argue that an SNO that knows what they are doing (ie serious SNOs) would run redundant hardware (and no Alexey, that doesn’t exclude A-B power and generators) as well as redundant (not to mention different physical path) ISP connections.
Such SNO can (and should) run as many nodes as they can, IMNSHO. At that hosting level, you are basically at the “small datacenter” level, minus the certs.
Sorry, we are not kidding ourselves. But have you ever seen this site? http://www.th3van.dk/ (Sorry @th3van if I quote you as if it were a negative thing, but it was the most striking example to make the issue clear, I appreciate you and I am happy that the network often relies on SNOs like you)
Here there are hundreds of nodes in the same physical location connected to a single server with various jbod arrays in cascade. Furthermore, before they were all on different IPs or almost, now instead they all seem to be on the same IP, perhaps for maintenance to the internal network. So what do we want to talk about? That there are many SNOs that have nodes in the same physical location? One of those is me, I have five physical locations in Italy with as many public IPs with at least 5 or 6 nodes for each IP. I suppose that most older SNOs and/or with a little more economic possibilities than the average have configurations of this kind. Obviously I’m not saying that what I do is correct, but I suppose that many SNOs have more nodes in the same physical location, some 2, some 5 and others even 500 nodes.
I’d expect SNOs with multiple IP addresses at a given physical location to be a minority (of users, perhaps not of storage).
That’s different from running multiple nodes on the same IP where the satellite can ensure not to send segments of a particular file to more than one node based on IP and splits the bandwidth among them so it acts as a single larger node.
The only reason to use multiple IPs is to get more bandwidth than the network would have otherwise opted to send to your physical location.
Oh, not over again about that malwritten ToS only @Alexey and some other people seem to know what’s in the spirit of the ToS although not being written there… Kept upright, with many bogus explanations that never will survive a trial.
Which I remember to be very counterproductive, especially since STORJ seems to like the ambiguity and no one can actually explain the meaning and real authority of the ToS anymore
Ah, you mean the same /24 and VPN discussions we’ve seen every few weeks for years now?
But… but… maybe someone will have some fresh insights this time! A nuanced perspective! A unique position that finally gets the entire community on the same page!
no, it does not. And I like your suggestion, however we also need to extend it to /64 for IPv6 (maybe we would have to use it if the situation with IPv6 would change). However, I would prefer it to be implemented in the protocol instead (or in addition).
What is will become broken in the current ToS, if someone would run multiple nodes using multiple IPs on the same server:
5. Restrictions. You will operate the Storage Node in strict accordance with the terms of this Agreement and in no other manner. Without limiting the generality of the foregoing, you will not:
...
5.1.7. Manipulate or alter the default behavior of the Storage Network to artificially increase or decrease the value of any reputation factor of any Storage Node;
...
5.1.12. Manipulate or alter the default behavior of the Storage Network to artificially increase or decrease the value of any reputation factor of any Storage Node;
It’s even mentioned twice
Because running multiple nodes on the same physical server with multiple IPs will change the default network behavior to do not store more than a one piece of the same segment in the same physical location, because it’s what the /24 rule for.
They should be independent as much as possible. I.e. different power supplies and/ or a redundant one, independent internet channels from different ISPs, a redundant hardware, include server and disks.
The good example is to run multiple nodes in a datacenter as a high availability cluster. Basically it could be a one server, but if it’s durable (to some extent of course), it can be considered as an acceptable compromise, but must follow some standards for the high availability, including redundancy for everything, hardware, disks, power supply and the internet connection, etc.
exactly. I would also add a technical specialists into the list to service that 24/7.
This is a DC with high availability and redundancy.
So did I, ‘don’t tell, don’t ask’-policy.
Because the ‘spirit of Storj’ is just too ambiguous if you have to resolve for example CG-NAT issues and a VPN provider delivering you multiple IP-adresses. Too often just seen “it’s against the ToS”-answers, when it wasn’t backed up by any reasonable explanation of the ToS. For example:
Explaining VPN as changing the node software or functioning of the Storj network in one situation, but condoning it in case of CG-NAT.
The issue of this topic of multiple IPs for one physical location, but in case of the already mentioned data centers or CG-NAT evading is all of a sudden no problem.
Running multiple nodes behind one IP, which was forbidden by the ToS, but explicitly condoned by the forum including FAQs how to do so.
Stating one disk and at least one CPU core should be assigned for any node, but accepting RAIDs with multiple nodes.
99.3+% uptime requirement, but the node software is showing you to be in safe conditions in case of 98+% uptime (green) and the satellite doesn’t turn you down unless you are 60-% online.
Telling each time that everything should be managed as much software-wise as possible, in the meantime demanding you to use graceful exit in case of cessation
…
I remark myself to get yawning when talking about the ToS, since it’s actually not relevant anymore because of the many discrepancies which often is also being agreed not to be enforceable in any way.
Indeed, I see now while reading it. Let’s not talk about changes of terms and the necessity of an update notice, especially in case you’re holding the other party to follow it (Update Notice for Changes in Legal Agreements - TermsFeed just a shirt explanation about the ESIGN act).
You are correct, there are multiple cases, which cannot be covered by ToS literally, so we discuss some of them here on the forum, and it would be nice to have collected all exclusions and why they could be allowed, but I have no idea how to do it right now.
We just want to have all pieces stored statistically independent (or at least redundant, even if it’s not required when you start the one node), then the customers’ data will be in safe.
Again, Alexey, we got over it many times now. But:
There is no definition of the storage network in the ToS leave alone it’s default behaviour, so it’s ambiguous
VPN or having many IPs for one physical location isn’t artificial, and doesn’t change any default behaviour of any network. You’re stretching up the explanation to what you want, instead what can be really read in it (because both are not explicitly forbidden in the remainder of the ToS).
Also the intent of preventing multiple segments within one physical location is nowhere mentioned in the ToS.
There is now such a thing like a rating, making your node being picked over the other.
So, it’s nonsense again. Not willing to be rough, harsh, whatsoever.
But let’s start with something like:
"Since the default behaviour of the storage network is to disperse data across as many of storage nodes operators as possible [to be resistant to termination of one or some operators] as well geographical dispersed [to be resistant to natural disaster, fire, war, and so on] both in a random manner,
Every storage node operator (SNO) must be identifiable by the same STORJ-wallet and/or email address across the whole network, in such a way that a single SNO always is to be determined to be the same if one of them are the same;
Storj also implements means to disperse the data as much as possible to geographical different store nodes using the IP-address of the storage nodes, which implies a SNO has to use as few as possible IP-addresses for one physical location (a building in it’s widest interpretation) to run all storage nodes within it; not using means to artificial increasing the number of IP-addresses, for example by using VPN or multiple ISPs.
Any situation in which multiple nodes of one storage node provider aren’t identifiable as belonging to the same operator by wallet address or email address, will lead to suspension in case it’s being traced back."
At first, I think wallet address and email address are better than /24-IP. Because you don’t just want geographic dispersion, but also dispersion over SNOs.
Another idea could be to set out some default setups, with the obligations to ask permission first in case of another setup than outlined in the standard setups. So you can accept data centers / professional operators and home operators along each other, with different requirements.
Table SNOs: ID (auto-increment), …
Table nodes: ID , SNOID.
Table SNOEmailAddresses: SNOID, MailAddress .
Table SNOWallets: SNOID, WalletNumber .
Every time, when a node signs in with its credentials it also delivers the walletnumber and mailaddress besides the nodeID. This script assumes, only the real node operator is able to combine at least two of them succesfully. So, if two of these values have been a match previously (directly or indirectly), than there are two options: if the new value belongs to an already existing usere than he must be using multi-credentials, if the new values doesn’t belong to any user he aparantly wants to update a value or just add a new node.
# Check node using at least two identiable factors
int checkNode(sNodeID, sMailAddress, sWallet){
validMailAddress(sMailAddress) or return E_INVALID_MAIL
int iSNO = SQL.FetchField("SELECT SNOID FROM Nodes WHERE (ID='{sNodeID}')")
int iSNOByMailAddress = SQL.FetchField("SELECT SNOID FROM SNOMailAddresses WHERE (MailAddress='{SQL.Escape(sMailAddress)}')")
int iSNOByWallet = SQL.FetchField("SELECT SNOID FROM SNOWallets WHERE (WalletNumber='{SQL.Escape(sWallet)}')")
# First sign-up of node
if( isNull(iSNO) ){
if( isNull(iSNOByMailAddress) and isNull(iSNOByWallet) ) iSNO = SQL.Insert("SNOs () VALUES ()") # First sign-up of SNO
if( isNull(iSNOByMailAddress) or isNull(iSNOByWallet) ) {
# No two factors
return E_NO_TWO_FACTORS
} else if( iSNOByMailAddress <> iSNOByWallet ){ # non-matching values
# Don't suspend anything, because only one factor (someone trying to get someone else out of business for example?)
return E_NON_MATCHING_SNOS
} else iSNO = iSNOByWallet
SQL.Insert("Nodes (ID, SNOID) VALUES ('{sNodeID}', {iSNO})")
}
if ( iSNOByMailAddress and ( iSNOByMailAddress <> iSNO ) ) { # Two disagreeing factors
if( iSNOByMailAddress == iSNOByWallet ){ # Someone knows the two factors of wallet and mailaddress, so almost sure someone with multi-credentials
suspendAllNodesBySNOID(iSNOByMailAddress)
# Don't block SNO of NodeID, because only one factor pointing to that SNO
} else if( iSNO == iSNOByWallet ){ # Someone knwos the two factors of this wallet and node, so almost sure someone with multi-credentials
suspendAllNodesBySNOID(iSNO)
# Don't block SNO of mailaddress, because only one factor pointing to that SNO
} else {
warnSNOForAbuse(iSNO, 'NodeID')
warnSNOForAbuse(iSNOByMailAddress, 'MailAddress')
if( iSNOByWallet ) warnSNOForAbuse(iSNOByWallet, 'Wallet')
return E_NO_TWO_FACTORS
}
return E_NON_MATCHING_SNOS
}
if( iSNOByWallet and ( iSNOByWallet <> iSNO ) ){ # Two disagreeing factors
if( iSNOByMailAddress == iSNO ){ # Someone knows the two factors of mailaddress and node, so almost sure someone with multi-credentials
suspendAllNodesBySNOID( iSNO )
# Don't block SNO of wallet, because only one factor pointing to that SNO
} else if ( isNull( iSNOByMailAddress ) ){
warnSNOForAbuse(iSNO, 'NodeID')
warnSNOForAbuse(iSNOByWallet, 'Wallet')
return E_NO_TWO_FACTORS
}
# other possibilities are already handles a bit up
return E_NON_MATCHING_SNOS
}
# Update of mailaddress, wallet-address or it's just the first time the satellite sees thise node operator
if ( isNull( iSNOByMailAddress ) ) SQL.Insert("SNOMailAddresses (SNOID, MailAddress) VALUES ({iSNO}, '{SQL.Escape(sMailAddress)}')")
if ( isNull( iSNOByWallet ) ) SQL.Insert("SNOWallets (SNOID, WalletNumber) VALUES ({iSNO}, '{SQL.Escape(sWallet)}')")
return E_SUCCESS
}
So, if I first sign up with (nodeID, mailadress, wallet): 1ba, a@a.com, 0xAO > addition to database, as SNO 1
Then, with: 1Ug, a@a, 0xA0 > 1Ug is being bound to also user 1
Then, someone else with: 1Os, b@b.com, 0xB9 > addition to database as SNO 2
Then, that someone else signs up with these six variants:
1Ug,b@b.com,0xCA > no two factors of same nodeID > just an error
1Ug,b@b.com,0xA0 > SNO 1 is suspended, because apparently trying to use a mailaddress of another SNO or using multi-credentials (more likely).
1Ug,b@b.com,0xB9 > SNO 2 is suspended, because trying to bind an already existing node of SNO 1 to itself (it is debatable, whether SNO 1 shouldn’t be blocked too in this case; depending on how that nodeID is being delivered, does it prove that the user actually has the signed cert?).
1Ug,a@a.com,0xB9 > SNO 1 is suspended for using wallet address of SNO 2, so he’s probably using multi-credentials.
1Ug,a@a.com,0x6D > wallet 0x6D is added to the known wallets of SNO 1
1Ug,c@c.com,0xA0 > mailadress c@c.com is added to known mailaddresses of SNO 1
If you keep this record:
You can easily select at maximum one node per storage node operator (also rendering the use of many IP-addresses per SNO useless) for any segment.
You can enforce within the maximum extent, that SNOs aren’t using multiple credentials.
You even could enforce the use of one wallet address per SNO.
People only should be aware, that if they want to change mail address or wallet they must first make the network acquanted to the new wallet or mail address by keeping (so, only one change at a time).
This can only be avoided if people keep an extensive track record for their nodes, making every combination distinct in all three factors.
I’m sure already a certain extent of these values are already being kept, in order to send us emails in case of being offline too long.
Some email provider have their own rule for alias, ie: yourusername@provider.com and any combination of yourusername+39@provider.com will also be your alias. You would have to learn all the rules for each email provider and filter it correctly. If it come to that, what prevent an SNO to create a new email service and share with everyone with an unpredictable email alias forward to your main email to prevent tracking? Sooo… email is out…
Since we now know, StorJ track IP where you original register, just VPN when you register ., IP when running node already subject to /24 and frankly cannot be reliably count - since most ISP using dynamic IP.
Some CEX allow me to have multiple wallet address (Binance for example - though I don’t know how many wallet is the limit), so no transaction fee occur when combining multiple wallet together, and that the final circumvent…
Enforcement is not that important. There will always be few who try to cheat in any system, there is no good reason to waste disproportionate amount of resources to make life of disproportionally small number of cheaters temporarily slightly more frustrating. It is, and always will be, possible to cheat if one is determined enough.
Most people, however, will honor the agreement in good faith. But the agreement must be unambiguous and coherent, that’s the main problem here.
It should not be in ToS, because it’s a current implementation of the node selection, described in the whitepaper. The current implementation is described in the documentation and blueprints. The ToS is changing much much less often, then the implementation, so no point to include the particular aspects of the node selection to the ToS, because it may change.
For example, we didn’t have a /24 rule, so there was a requirement to do not run more than a one node on the same server. This was a default behavior of the network.
Then we implemented the /24 rule to the node selection and documented it, now it’s a default network behavior. And now it’s not forbidden to run more than a one node on the same server, but they must be in the same /24 network to do not risk the customers’ data.
Then we implemented the choice of n in the node selection, and now the more performant nodes selected more often than before, basically reducing the effectiveness of bypassing the /24 rule by some, because with a higher latency the VPN nodes will be selected less often, reducing the risk… And so on.
I would expect that VPN nodes will have a worst score over the time and this would make them obsolete, because likely we would still improve things here.
It’s changing the default network behavior, now your VPN nodes running on the same server have a higher chance to get pieces of the same segment, reducing the durability of the network. This is harmful for the network and you know that.
If you behind CGNAT it is ok to use a VPN but all nodes on the same server/the same ISP/the same power supply/etc. still should be in the same /24 subnet, this would not reduce the durability and wouldn’t change the current default network behavior.
Yes, because it is in the whitepaper and in the documentation, including blueprints and implementation in code. It also published and explained here, on the forum multiple times.
So, you cannot say, that you didn’t know the default network behavior.
Running multiple nodes with multiple IPs to bypass the security measures on the same hardware without at least high availability is harmful for the network and to you in the end, and it’s changing the default network behavior.
If you want to run many nodes with different node selection rules you can conclude a contract as a Commercial Node Operator, you will discuss all conditions with the team to make it profitable for you and being not harmful for the network. If you are not eligible then all the current conditions will apply.
There is. It’s implemented in the node selection, which is part of the default network behavior. It has improved over time, but it still cannot effectively prevent Byzantine behavior by some operators who circumvent security rules by, for example, artificially presenting nodes running on the same server as being in different locations.
I like this suggestion, but do not like that you also included details of implementation, specifically educating malicious actors how to bypass them. Details of implementation shouldn’t be included in ToS in my opinion. ToS should be separated from the implementation.
we actually do not want a dispersion over SNOs, only the hardware, to make storing pieces reliable and independent of each other as much as possible. I would agree, that sometimes the downtime can happen because of SNO too, for example they may update the whole cluster with a bug and all servers would be down in the same time, but it’s too strict as I think.
Using the wallet and especially the email address as a determinate factor will actually have a reverse impact. It is much easier to generate several wallets and/or emails to introduce a bigger problem than we have now with VPN nodes running on the same server, because it doesn’t require to also increase costs of running a Sybil attack.
This is exactly how the Commercial Node Operators are examined (one of the conditions).
At first, there is no reference to it (=whitepaper or any document describing the intended principles of the network) making the definition ambiguous at first.
Second, you’re actually saying it’s not static, so the SNO should be notified for each relevant change and cannot be held to track relevant changes himself.
So there is an undefined, changing network delivering storage.
There is one reference about the storage service software:
These documentations are strict practical, don’t tell anything about the intended use of the network and have no normative connotation in it. Besides, there is a inconsistency: if you need to comply to a recommendation, it’s not a recommendation anymore but a requirement. Furthermore, “not limited to” is quite vague: just tell what belongs to it and keep it up-to-date. Inform everyone on relevant changes.
The forum is nowhere called to be normative in the ToS. Besides, are your words normative in the forum? Since you’re officially no rep of Storj in the first place? But even then, would that mean that every SNO has to scrutinize the forum for any piece of information of Storj-related persons?
I know this. But it’s not forbidden by the ToS, aside from the discussion at which point the network starts and ends (is my node itself part of it, or do I just attach to that network and how does OS and any other software on my node relate to it). And again, apparently there is no “default network” because undefined, which previous sentence explicitly tried to make clear.
But again, running multiple nodes with the same IP is forbidden -as you know- by the ToS. Although apparently just one time, instead of ‘manipulating or altering the (undefined) behaviour of the Storage network’ which is two times forbidden in the same section. Leaves me asking when a score is artificial and when it’s real…
We can debate it many times, and I like you taking this real serious. But come on, if you’re honest with yourself: that ToS is ancient, a little bit more ancient than the promise to update it. Any reference to it, is like referencing to a ghost in the past which you can use like Google: it says what you want it to say.
Juridical, it’s all worth nothing. But don’t think anyone is really up to trying to test it.
So, let just forget about it, since we want to do it just by software. Besides, I think most SNOs are of good faith and fair dealing. ToS shouldn’t be needed in those cases and let’s hope that the undefined network has the resilience to cope with those few SNOs with malicious or selfish intents.
Edit:
Apparently if the discussion can’t be won on the content part, then there is always the solution to overrule this by marking your own post as the solution and closing the thread. Apparently, is the main thing missed here is that the ToS just isn’t aligned to current practice and references forum and github aren’t referenced in the ToS as normative.
., IP when running node already subject to /24 and frankly cannot be reliably count - since most ISP using dynamic IP.