Bandwidth utilization comparison thread

Storage Node Operators FAQ
, , ,
#1059

Come on my friend,

  1. Vetting node ingest is extra ingest
  2. Defend your subnet; If you have many nodes on “your” ip then you will not be affected by an upstart “stealing” your ingress
  3. If you have a disaster on your main. You want a vetted, even 15 month if possible, node ready to go.
  4. Expansion. When storj takes off you need to be ready to scale out.
  5. There are limits to a node. The file walk means that a node above 100TB is probably prohibited.

Besides, I know you’ve got more than one already.
Anyway, my storj friend, I’ve typed too much and I don’t want to challenge you as master of the tldr!
And.

1 Like
#1060

totally agree with this post. I currently have 3 nodes at the moment. Started each one in a staggered fashion. First one in Aug 2019, second in April 2020, and the third back in late Sep/early Oct 2020. The first two have 12TB HDDs with 10.8TB available for Storj. First one has about 8.8TB used, second one has 4.4TB used. Then for the third node, I’ve just been using a 1TB portable, external HDD (2.5" running on only power from USB port of Raspberry pi 4) as a temporary drive to “seed” the new node, with 500GB allocated to Storj. That has about 250GB, currently. And actually I’m currently in the process of migrating that data to a new Raspberry pi with a 4TB HDD attached to it. I’m thinking I’ll probably leave that allocation at 500GB for the time being, and then repurpose the 1TB HDD for a new 4th node, so I can get that seeded. I luckily have another old 4TB WD Red HDD laying around, along with another 10TB WD Elements HDD that I can continue to use for expanding offered space.

I have high confidence in the future of Storj, and can certainly see using both spare drives at some point, once the others start filling up. Although, for the time being, I’m just planning on keeping them shelved as spare drives if I need to replace one of the active nodes. I also anticipate having to replace two of my four drives in my plex server at some point over the next year or two, so that will give me two WD Red 6TB HDDs to use for Storj if need be.

1 Like
#1061

i got 2 tiny ones that barely counts and they are all on the same subnet, so basically on node anyways.
so cannot really compare how it behaves when i don’t have two nodes created on the same time, on the same subnet, thats what i mean… and when i start running multiple nodes… more subnet because right now i’m basically just one.

100TB is unreachable unless if deletion pr month would be like sub 1% and ingress is larger than 1TB
if we can hit 30TB we might be lucky at present, ofc this will be a moving goal line.
depending on customer behavior and storj labs ofc.

vetting nodes still takes from the total ingress of a subnet

yeah ingress have been rather dead lately, so starting new nodes right now is a rather inopportune time, as you might also have seen during the townhall, we had gone from 600 or whatever nodes up to like 4000…

so essentially test data is dead, the network is growing at a pace where storj most likely doesn’t have the ability or capital to pump into it anymore to artificially inflate nodes.

been kinda stuck on one subnet because i refuse to setup anything less than perfect :smiley:
with a foreign storagenode…

one weird thing about my tiny new nodes, is that the 3rd one or 2nd one of the new ones, doesn’t seem to have completed vetting now some 2 months later… for some reason it is getting less ingress than the others even tho, it’s like 2 months now, but i guess with 4 nodes on a subnet that kind of stuff goes slow… even if people seem to say it doesn’t…

should have looked into that, but haven’t bothered because i’ve been trying to get my former new ISP to cooperate, but that was a bust… so going to get a 3rd one on monday, or atleast initiate talks to figure out if they will actually do what the others claimed they would before i actually signed up and they just basically didn’t care until after nearly 14 days of trying to get them to do what they promised, they finally let me up the support level so somebody was not drooling into their keyboards enough that they could define if they wanted to sell the solution to me or not.

which then would cost me hundreds of dollars just for the connection… the same connection…
so now the goal is to bother them enough to make them give me my money back for the connection i never used, doubt that will happen… but i can try :smiley: i do have my mastery to rely on…
tldr kungfu

and the funny part of the story is that my other choice of ISP will actually do the setup i wanted, cheaper than the first one… and only reason i saw the others as a possible solution was because i doubted a big company would bother doing that setup for a consumer, and when i called the others they said it wasn’t a problem.

@dragonhogan
but yeah it sure does look like we might be in for a long haul while we compete with the semi enterprise setups… tardigrade got popular, now many people are running nodes, some won’t persist.

atleast it’s most likely good for the customers :smiley: so there is that

1 Like
#1062

image
image984×1610 99.3 KB

My first week stats.

7 audits total which seems really low with 98% up and 92% down success rate.

1 Like
#1063

does seem kinda low… but it also requires data for you to get audits…
so it will be slowest at first… and pickup exponentially…
duno if this is to low, it’s certainly not high.

image
image829×762 35 KB

this node is 2 months… but it’s one of 4 on this particular subnet.

you can check this if you are alone on the subnet
not 100% accurate but like 95-98%
and should be 100% for older larger nodes, something to do with how it works.

your node looks fine… it’s just slow at first… very very slow at first.
and the network have just had huge growth in the last quarter so many more people to take data.

if there are others on your IP/24 subnet then the data will be shared between all node equally,

#1064

no. Vetted nodes get 95% of global traffic, unvetted nodes 5%.

(You even answered directly below my post linking to that answer in a different thread… Ingress traffic splitting between vetted and non-vetted node on same subnet)

#1065

that might actually explain why my 2 month old node isn’t vetted yet and why coldsc
has only 7 audits over a week.

if global traffic is fixed at 5% for vetted nodes, and there have been an increase of 3000 nodes over the last quarter… that could essentially stall the vetting of new nodes due to the global traffic remaining the same, but the number of nodes going up by 5 fold…

that would mean that ingress is about to drop like a stone when all the nodes vetting comes online…
only reason why vetting would take months is because there is an insane amount of nodes being vetted.

not a good prediction that…
wonder if they made it that way… the 5% of global instead of subnets to ensure a slow growth of the network, because it would create a sort of static amount of new nodes that can vet in a certain time equal to the total ingress / repair ingress on the network.

thus if the network grows faster it will vet more nodes faster and if its not growing node vetting will slow down…

that would actually be pretty smart for the stability of the network and profits of SNO’s
ofc it also means that if somebody in theory now created 10000 nodes they could get something like 4% of the total network bandwidth routed to them…

which is kinda crazy… ofc they would have to maintain all those nodes, because most will go into held amount, which then becomes a critical stop gap measure… but still seems a bit crazy still.

ofc when the vetting nodes comes out of vetting then the whole 10k node thing goes down hill … :smiley: so it would be kinda pointless i suppose…

1 Like
#1066

lots of things to keep track of, sorry my bad… pretty sure it’s stuck now… should be, because now it makes sense why it is like it is :smiley:

#1067

my unvetted nodes got 2.7GB ingress yesterday (both combined since same subnet). My vetted node got 12GB. So currently that’s 22.5% of the ingress of a vetted node. So it’s still a long way from a vetted node actually only receiving 5% of a vetted node. So this system is actually faster for unvetted nodes.

However, it also depends on generall infress:

I have 2 unvetted nodes on the same subnet so vetting will take them longer. They are both 1 month old now and vetting is done 11% (europe-north) up to 54% (on saltlake).

#1068

i should try and check how far mine are with vetting.
doesn’t seem to get anywhere really… maybe the other guy on my subnet is vetting nodes also.
the one in the screenshot is my newest 2month old node. passing 2 full months in 2 days.

it got 2.7gb ingress for the 12th, which would mean either my 3 month old node isn’t finished vetting yet… but that doesn’t get the same low ingress as this latest one does, it’s usually equal to my main node so i just assumed it was fully vetted.

so if you are right about the a subnet having it’s own share of the vetting 5% global, then me having the same number as you would mean either my subnet neighbor is vetting 1 new node.
or in fact all vetting nodes gets an equal share of the global independent of subnet.

cannot say which… but that would be fairly easy to figure out by comparing ingress across more vetting nodes from other people.

coldsc’s
graph also seems to be around 2.7gb for 12th
so that would indicate that the global 5% is split between equally between all nodes, or he also have somebody vetting on his subnet :smiley: which seems less likely.

so no your two nodes should vet just as fast as every other vetting node. only the network ingress and total number of vetting nodes will change the time it takes to fully vet a node, atleast to how i understand it and what the numbers seems to say.

ofc it does seem i have an alarming tendency to be wrong when dealing with this stuff…lol
but hey we are all learning.

#1069

nah my 2 nodes combined have 2.7GB, each one has 1.35GB. So you’re fine and the only unvetted node.

1 Like
#1070

there is another interesting fact about that tho… if we know the semi accurate number of unvetted nodes we can also calculate the total ingress onto the network, because it would be number of unvetted x current ingress = 5%

ofc the neighbor tracking thing works best on larger older nodes, so thats not going to pick it up.

doesn’t really have the information i want, but that andromeda thing looks kinda interesting.

#1071

While ingress is low the current nodes seem to be suffering quite a bit with the delete pressure- for sure. While staying below 10GB of loss, it still was enough to negate any ingress gains for several hours (12hr window show, zoomed to 10GB window height). …
image image

#1072

yeah does look like my node have been shrinking for the last few days… but difficult for me to tell since i had 24 hours of dt recently and still got 137gb in trash…

i was reading that thread from the Q3 townhall question
linked a post or two above, it talked about how there was like 1300 or whatever nodes reporting free space and 8400 in total, the numbers are irrelevant for my point… :smiley: just saying

so if we imagine deletes hitting the network on a large scale… this would have a MASSIVE impact on ingress, due to us being able to go from like say 1300 nodes that share ingress to 8400 nodes that share ingress…

so it would sort of start an avalanche when doing large deletes on the network, avalanche might be a bad example… more like digging into an ant hill and workers will swarm to take care of stuff…

much in the same way the network would from big deletions see massive drops in ingress months after as all the nodes getting extra capacity is filled back up.

just like the ant hill would see massive activity of workers in a removed area of the ant hill until the structure has been repaired and is back in equilibrium.

since the majority of storagenodes are like the bottom of an iceberg… 90% is below the surface / full
wouldn’t surprise me if the storagenodes of the network follow the same ratio… 90% of nodes will usually be filled… :smiley: ofc just a gut shot guess, but math and nature has certain ways it likes to behave.

2 Likes
#1073

Actually, if there’s merit to what you’re saying (large number of “sleeping” active but full nodes), then this would coincidentally coincide with some of the depressions in ingress that seem to somewhat follow around when massive delete storms start happening.

#1074

yeah that was what i was thinking, it sure would explain some of what we are seeing and then if there is a large uptick of incoming / vetting nodes

it would also explain why they went from like 800 or 1300 nodes in the last townhall and then this time said there was 3700 over whatever which seemed like an immense uptick in the number of nodes…

but really it was or might have been related to the big deletions… which also may have been done by storj labs to equalize customer data over more nodes… in case large numbers of them contained mostly test data…

so yeah atleast from a hypothetical point of view, it would make a whole lot of sense.

#1075

They consider the pieces unhealthy after a node has been offline for 4 hours. There is a separate system to track when a node is offline for reputation scores, which counts the node as offline as soon as an audit is sent to an offline node. I’m sure you knew this, but just making sure people don’t think they can be offline without penalty for 4 hours.

No, repair only recreates and reuploads the pieces that are unhealthy at that time. So other nodes that are online can keep their pieces.

There has been some confusion around this. There is a system that audits unvetted nodes with at least one piece with priority. This should help all unvetted nodes equally. But this accounts only for a small part of total audits. All other methods still rely on how much data you have and since unvetted nodes share ingress, it is still expected to take a while longer for multiple unvetted nodes to be vetted.

None of this really matters as long as you make sure you always have a vetted node with free space available as well.

2 Likes
#1076

yes, sorry I was only referring to traffic, not the uptime system.

but a piece consists of multiple erasure code shards which have to be downloaded from multiple nodes to recreate the piece and then upload all shards to the nodes, so nodes that have the old shards will have to delete it. So the nodes that store shards of a different piece of a file will be unaffected by repairs.
(Maybe I got the terminology of shards and pieces wrong but I think you understand what I meant)

#1077

A little bit. What you call a piece is actually a segment. A segment is erasure encoded into pieces which are spread out across the nodes. The satellite knows which pieces are healthy and which aren’t. If a segments number of healthy pieces drops too low, repair is triggered. That repair downloads 29 healthy pieces, recreates the encrypted segment and from there recreates the unhealthy pieces and uploads them to new nodes. So this leaves everything on healthy nodes in tact. The only thing they may see as a result of this repair action is repair egress if they were used to download one of the 29 healthy pieces. They get to keep the pieces they already had. Otherwise you would be penalizing and removing data from reliable nodes. Luckily that doesn’t happen.

1 Like
#1078

well that’s good, makes for a much better system over all, it also seemed like a very wasteful approach.

there are four nodes on my subnet, i said that before kevink brought up the 5% of global for unvetted nodes on a particular subnet.

does seem like this node is taking it’s sweet time, started one a month earlier and it finished in the usual month, but this one i made 2 months ago tomorrow has just been churning away, haven’t gotten around to checking how far its gotten because my network is messed up atm and haven’t bothered to fix it because my new ISP are being …'s
apparently vetting nodes are going slow atm, even if there is a minimum vetting speed, you think that is an indication of there being many new nodes being vetted?
or is it just down to the very low traffic we seem to have been seeing for a long time…?