my question is why not running 6 real nodes and use them as backup for the real ones?
Because redundancy is already built in to the storage network design. Each node does not need to be reliable.
1 Like
These SSDs are perfect for storj usecase ā the drawbacks donāt matter because storagenode writes very little and never rewrites any data. Even accounting for data amplification, 1TBw per months is nothing. The benefits ā low cost, noise and power consumption.
Iām not sure why are they ābadā in your book.
I would argue booting from 2TB 980āProā is an obnoxious overkill, as is raid6 (!?) of just eight drives, let alone for a storage node (I would do raid0) ā but itās not my money so why not. If done carefully, manually overriding 512e bullshit when deploying the filesystem ā it could work.
We had eight of those SSDās and the Broadcom RAID controller laying on the shelf, decommissioned from another server project.
Each backup node shares a very small amount of hdd space. In my setup itās ~42 GB per node.
The idea is not to get much ingress or egress, but instead to get pass the held back period of 15 months and eventually get all nodes vetted.
So when/if any of the primary nodes gets DQāed due to eg. a bad HDD, it can be replace with one of the backup nodes, and should not have to wait the held back or vetting periods.
Of course, I have to wait for new data to be send to the replaced primary node, since all data from the DQāed node are permanently gone.
Currently, the CPU is running between 5-10% serving 501 nodes, and has been running pretty steady for the last 131 days.
root@backup-parking-slot001:~# lsb_release -a
Distributor ID: Ubuntu
Description: Ubuntu 22.10
Release: 22.10
Codename: kinetic
Th3Van.dk
3 Likes
Thankās alot for the amazing answer. Now I understood why you prepare such backup-server. So it also means you are having already like 500 nodes running? You could, when you expand, always feed from this pool because as you said they would be vetted by then.
the whole idea (incubating nodes and then deploying them elsewhere) and implementation (running them all on one host) appears to violate both the letter and the spirit of Node Operator Terms & Conditions.
1 Like
Yes, Iām running 501 backup nodes.
I could, but currently I have no plans on setting up more primary nodes, than the 105 primary nodes i already have, as long as thereās free space on them.
Dev (SM01) Available hdd space Used hdd space Free hdd space
Totals : 1.986.332.595.376.128 1.190.175.825.043.456 (59.9183 %) 796.156.770.332.672
Th3Van.dk
This is unbealivable. When I will have big node it will be some little nodes with different ISP. How I see there is no other way to give 100% guaranty that hardware works fine everytime.
I believe you can. You must modify a parameter in config.
My understanding is that thatās intended for debugging purposes only rather than something that should actually be lowered by an SNO. The SNO TOS and documentation both explicitly state 500GB minimum.
SNO Terms of Service (edit: raert beat me to linking this while I wasnāt looking):
- 4.1.4.2. Have a minimum of 500 GB of available Space per Storage Node;
Quick Start Docs:
Minimum of 550 GB with no maximum of available space per node
The docs technically say 550GB, since itās accounting for the recommended 10% overhead, but itās pretty clear that less than 500GB allocated on a normal node is a TOS violation.
Also, depending on how itās interpreted, this part of the TOS could be taken as ādonāt adjust the minimum allocation without permissionā, since you could argue that youāre manipulating the performance of it.
4.1.3. You will not modify or attempt to modify the Storage Node Software for any purpose including but not limited to attempting to circumvent the audit, bypass security, manipulate the performance of, or otherwise disrupt the Storage Services for any reason, including but not limited to attempting to increase the amount of data stored or bandwidth utilized or the amount of Storage Node Fees, as defined herein, and you will not otherwise interfere with the operation of the Storage Services.
2 Likes
Eh, it can be done without modifying Storj software while being fully compliant with T&C. There are enough corner cases in T&C for that. Which is why Iām very curious what will be the new proposed document.
Besides, itās not like Storj Inc. does not benefit from the work of Th3Van.
2 Likes
Could you point out some of these corner cases? Iāve read through it and I donāt see anything thatād allow it, just stuff that disallows doing it.
Specifically;
4.1.2. You will operate the Storage Node in strict compliance with terms of this Agreement and will not take any action not expressly authorized hereunder.
You will operate the Storage Node in strict accordance with the terms of this Agreement and in no other manner. Without limiting the generality of the foregoing, you will not: [ā¦] 5 Operate a Storage Node that does not meet all of the Minimum Requirements;
5.1.12. Manipulate or alter the default behavior of the Storage Network to artificially increase or decrease the value of any reputation factor of any Storage Node;
5.1.17. In any other way attempt to interfere, impede, alter, or otherwise interact in any manner not expressly authorized hereunder with the Storage Services or the operation of any other Storage Node(s).
These are all quite clear.
Further, Iād say itās actually a detriment to have big farms like this, rather than Storj benefitting from it - it reduces the distributed nature that the marketing is built on, and, as weāve seen, petabytes of capacity can be wiped out in one go.
According to this site, there are 21,685 active nodes. 501 nodes is 2.31% of that total. That is a hugely disproportionate amount for one person to be running. When the payout changes hit, the network saw a significant dropoff in the number of SNOs participating (and I believe I read that most didnāt exit gracefully? I could be wrong on that). This site (having a GitHub link to an official repo, implying itās at least somewhat official or using official data sources) shows a free capacity on the network roughly in line with what the graphic in the first post shows. The graphic shows 3PB of free space lost. This is just shy of 10% of the networkās free space lost with those nodes being taken offline.
Caveat; thatās reported free space. 3PB spread across 500 nodes would be 6TB per node, so based on Th3Vanās post about āsharingā the space, Iām assuming in this instance the nodes are configured to deliberately lie about their capacity to the network, in the absence of any other information.
So, letās consider knock-on effects - on one hand, we have whales dropping out of the network causing disproportionate impacts. If one leaves permanently, the stats donāt look as good in marketing, and Storj becomes less appealing to customers. It may even be that over time, when free space isnāt 33PB, the eleven nines guarantee canāt be legitimately offered anymore. Storj has to offer the new, reduced service for cheaper, and pay SNOs less. More people leave, service quality degrades, it gets cheaper, etc etcā¦ itās a snowball effect. This can be compounded by individuals without silly money to invest seeing the whales and deciding not to join the project, further centralising the distribution of data and weakening Storjās marketing.
When thereās people not playing by the rules - especially so brazenly, on such a large scale - it inherently harms the project rather than benefits it. Itās not unreasonable for SNOs - and the customers who are paying for the service as advertised and whoās data is at risk - to expect enforcement of the TOS as itās written (and this is ignoring the fact that relying on edge cases to do things that clearly arenāt in the spirit of a legal document is braindead at best).
If you operate even a medium-sized data center, itās easy to have thousands of drives in operation. Allocating half a terabyte on each would not be a huge deal if theyāre already decently sized. Besides, nothing in T&C actually denies using the space allocated, but not yet filled, for other purposes, as long as the space is available when it is needed by the node. And given the nodes operate behind a single IP, they are not going to be filled quickly.
The only problem is with T&C mentioning a separate IP per node, but this has been already covered on the forums.
Going away from artificial requirements made by T&C into real needs of the network. 2.31% of all nodes? Does not matter. What matters is amount of data actually stored. And this is limited by the fact itās a single IP address. With the current operation what counts is number of unique IP /24 blocks that a given piece of hardware operates in, and these 500 nodes just donāt matterāexcept, probably, slightly increasing resources needed by satellites for accounting.
These 500 nodes dropping out of the network? Not a big deal, itās just 500 Ć 42 GB = 21 TB of data. I host more, with likely worse uptime and latency parameters, simply because Iām operating with just a residential connection. Iāve likely caused more repair traffic with my single Microserver gen7 setup, than these 500 nodes. You will just not see this in official stats because itās a low number of nodes. So you wonāt cry about that.
Thereās one more factor that might, or might not be at play here. Random people like you and me join the network with expectaction that we will follow the regular T&C. But large operators may just as well negotiate different setups. I wouldnāt be surprised at all if Storj Inc. had some special arrangements, for example to maintain stable supply of nodes in presence of regular node churn. With the right type of contracts even regular node operators would benefit from presence of these operators. I recall someone from Storj Inc. stating here on the forum that they were approached by some companies already.
1 Like
In a zero trust situation, complaining about violations of TOS is like antelope complaining about the speed of the leopard
3 Likes
With great respect to your work.
In my opinion: if one of the backup nodes goes live, grows and churns, it will not have sufficient heldback to cover the churn repair.
It would be perfectly fine, if they would grow normaly to 500gb. And grow more later or never.
Here i see the biggest problem with heldback,
Since this would be perfectly fine with everything if it were like 24 backup nodes. Wich would be sufficient as backup also, but also would not have enough heldback in case of going live.
Also i can understand the cry of the community.
Lets wait for the new TOS, then rethink again.
My guess: @Alexey :we have to work at the held amount, to make it more reliable. Or let it go completely.
People have all done this with held amounts its not something new having new nodes with small amounts of space waiting for the 15month mark to not have held amounts to worry about has been happening since ive started storjā¦ But the fact of the matter is these whales have large amounts of storagenodes in a single location on different subnets isnt ok, they can control almost all of the data if they see fit they can shut down all there nodes and really screw storj if storj doesnt get some kinda control over this they can go bankrupt real quick, and have to tell there customers all there data doesnt belong to them.
One thing I know for certain is the whales arent doing it legit and they can for sure run every single node on its own subnet inside a single datacenter.
Just a small amount of the whales getting paid out. More then 50% of the network is controlled by whales.
31 wallets received over 1000 tokens for july, with 3 addresses receiving over 6K, 12K and 13K tokens. Thatās a lot! I received 400 tokens for 63TB of data. Those 3 SNOs have like 5PB of data, if the math is correct. Aprox 23% of the entire network.
your math is wrong, but may be not very much, I have 280tb and I got little over 2k, but it is with 10% bonus.
Also it can be a contractor that get salary in tokens.
I hope they are something else than SNOs, but the payment date is the same as for SNOs.
Well, this is pure cheating: you bypass most of the reasons behind the held amount structuring, and completely undermine the reasons for the vetting process.
Exactly. When Joe Shmoes do that here and there on their raspberry pi nodes, itās probably tolerable and unenforceable. However it seems a rather big players are openly stating about doing this now, and get no reaction from Storj.
I would like a definitive asnwer Storj staff, is such blatant undermining of vetting mechanism and circumvention of incentive/held back program OK to do? Yes or no? @Alexey @heunland? Because if thatās OK ā I have a few unsavory ideasā¦