How to limit / cap bandwidth (inbound & outbound transfer speed)

Node Operators
21

limiting tcp connections doesn’t work like that to my understanding.

one way to think about it would be like this, imagine having multiple internet connections, this allows you to do more parallel transmissions, else they would simply be in a cue / sequence / serial data transmission.
in most cases having multiple active connections only serves to make stuff move quicker… because the system will basically not have to wait upon initiating a new connection with a certain peer or something like that… just like your browser sometimes opens many connections at one time when data is stored in many different locations, so instead of waiting for one to finish it will simply open more until having started loading everything one a website.

if the connections are limited, then it will load fewer things at the same time, but in most cases the end result is more or less the same, so you aren’t really denying any connections… just telling the system how many it has to work with and then it will switch between them… sure in some very extreme cases it can limit stuff to much or create issues because of it…

but this is not a datacenter level transmission we are talking about here… so i’m sure limiting the connections to 10 won’t harm a thing, aside from making everything else on the network able to actually use the internet.

changing max concurrent doesn’t work well, it will make all kinds of weirdness happen with the node, it can work as an emergency stop gap measure, and i haven’t seen it damage my own node while running with that, but it did make it act all weird from time to time.

limit the tcp and check the logs after you make the change… if there is any problems it should be more of less immediately apparent…

but i doubt there will be, if you set it to 10

if you do change max concurrent you shouldn’t set it to low… maybe around 40 i think use to be a good spot, ofc that will most likely change depending on node size because it will start doing more concurrent stuff.

ofc if its to high it won’t have any effect and if its to low it will refuse a lot of pieces, which is also kinda bad… max concurrent should be avoided if you can…

also because pieces have different sizes its kinda relative how much bandwidth it takes… not sure what it does for tcp connections tho… might affect it, might not…
when i used max concurrent it was to reduce latency because i had some bad hdd’s that got overloaded, setting the max concurrent to 20-30 at the time worked pretty good, but did cause about as many issues as it solved.
its good for reducing peaks giving a better avg, but you ofc are refusing data, which is not optimal…

just don’t, not worth the hassle

22

i dug a bit more into the subject of how to configure it in linux, and i’m not sure what the guy was doing in link i first posted was correct.
he doesn’t seem to be limiting the tcp connections, but maybe what limit there is on controlling them, which we most certainly don’t want to tinker with.

found this… personally i would start by using the

net.ipv4.tcp_low_latency

The normal TCP stack behavior is set to favor decisions that maximize network throughput. This parameter, when set, tells TCP to instead make decisions that would prefer lower latency.

The default value is 0 (off). For workloads or environments where latency is a higher priority, the recommended value is 1 (on).

taken from the link below, this should make the protocol reduce the number of active connections to improve latency which in turn would reduce the network connection congestion created from the storagenodes.

and the added benefit is we don’t have to tinker with all kinds of configurations that can have all sorts of unforseen effects, this might solve the problem without any limits at all, which would be the preferred solution.

23

Thanks for your reply :slight_smile:

I’m sorry, your answer got me confused.

doesn’t work like what?

I am not sure we are talking about the same thing.
@Toyoo talked about setting the --storage2.max-concurrent-requests Storj parameter.
@SGC, in your previous post, you told me “it’s basic tcp/ip configuration”, so I’m assuming you are talking about a general TCP configuration (not a Storj configuration). That’s why I’m not sure you are talking about the same thing as @Toyoo :confused:

To sump up the discussion, I understand there are 2 ways to address my need (which is: being able to limit bandwidth consumed by Storj in order to keep some available bandwidth for other things: streaming, browsing, etc.):
1. Limit bandwidth (speed) with linux tools such as tc or wondershaper.
2. Limit number of concurrent connections by using --storage2.max-concurrent-requests Storj parameter.

@SGC, I don’t understand if your recommendation refers to option 2 or to something else.
@Toyoo, your recommendation would be option 2, right? If so:

  • how should I use it first
  • My understand is that this option only controls uploads in order to avoid to deny audits (which are/include downloads). And I shouldn’t try to find a way to reduce download connections, right?

Thanks for your help :slight_smile:

24

the config.yaml max-concurrent set to 20-40 will affect how well the nodes operate, ran that myself for a few months… got all kinds of weirdness in logs from running that, and i’m not even sure it would solve your internet performance issue.

ill give the tcp connection thing a whirl setting it up on my own debian vm/container, so i can give you a more concise way to implementing it.

1 Like
25

looks like you do the following.
atleast for setting up low latency feature, which might solve the problem without using limit’s
because the default is throughput which means many connections and thus will choke other internet activity as previously explained, with the connection ratio’s affecting network / internet bandwidth share

or whatever we want to call it :smiley:
in the terminal do

nano /etc/sysctl.d/99-sysctl.conf

then add the following line

net.ipv4.tcp_low_latency=1

and then save by pressing ctl x and confirm the filename by enter

and then you do a

sysctl --system

which will reload the tcp configuration defaults with the added parameter.

i know there is an easier way to do this, but i cannot remember the command for writing directly into a file from the command line

this might or might not solve the problem, but it certainly won’t hurt anything nor affect the performance of your storagenode, if anything it might only make it run better, while also giving you enough internet access that you won’t feel the storagenodes are running.

if it doesn’t work, i can figure out how to do the tcp max connections setup also.
its basically the same place, just a matter of exactly what line you need to add.

26

This is the only switch at the storage node level. If you want to control downloads, you need to do it at the OS/VM/hypervisor/network level.

Besides, I can’t tell you what you should be doing, sorry. You should know your environment better.

Whatever is healthy for your node and network. Mine is at 500, and given it’s per node, I essentially expect thousands of connections before the limit is triggered. My network is 10 years old home grade devices, so really nothing fancy… and it would still handle at least that much.

27

@Toyoo at 500 i doubt it will ever stop any… i ran my storagenode when it was at 7tb with 200gb ingress a day at max concurrent of 20 or less ended up a bit higher than that, but it only during boot would reject pieces…

so i’m sorry, but at 500 it certainly won’t solve his problem, nor is it likely to do much for your storagenode, tho in extreme cases it may, but when i was testing max concurrents i ran anything from 10 to 200 and keep track of how many it was rejecting, and at 200 even at around 200gb ingress some days, it wouldn’t reject anything… except maybe a boot, ofc it also depends on the speed of your harddrives, because their iops will never be able to compete with network iops.

usually an enterprise hdd today can do maybe 400-800 read iops… in most cases its lower than that… down to maybe 100-200 writes or less, so capping at 500 max… well your hdd’s will most likely limit your utilization before the max concurrent

1 Like
28

Thanks!

Actually, it was more a general question than a question related to my own environment. Maybe I didn’t ask it properly. I meant: “if my Internet connection is saturated because of Storj, would reducing uploads concurrent connections be enough or should I also consider downloads concurrent connections?”. Sure, I will test myself and see.
I was surprised by your answer since, at the beginning, I assumed I should more focus on reducing downloads than uploads if my laptop connection is slow (in my understanding, a common usage of laptop for browsing or streaming involves more downloads than uploads, but maybe I’m wrong). So my question was more a general question to better understand how Network works in general.

29

Throttling connections is a bad idea, and it will eventually cause issues with your nodes reputation, and possibly disqualification.

Only real way to solve this issue is by using one of the tools or options suggested for shaping TRAFFIC.

30

Network might be saturated in multiple ways. Ingress, egress, number of connections, various queue problems, routing… there’s no single answer to all problems, sorry.

1 Like
31

Don’t be. I’m here to learn, thanks :blush:

1 Like
32

What? 500 concurrent connections?! This will never ever be reached, will it?
I had to configure one of my nodes with a maximum of 4 concurrent connections for the poor quality SMR 2.5" drive dedicated to it to follow when there were a lot of uploads (ingress).

I must say I agree with @SGC on this one.

Vocabulary should be clarified here, potentially.
I’m wondering if you are both refering to Storj wording which is from customers’ point of view (POV), that is:

  • “uploads” from customers POV, that is “ingress” for SNOs
  • “downloads” from customers POV, that is “egress” for SNOs
33

Actually I had some major IOPS problems some time ago on some otherwise very healthy non-SMR setup. To the point where this limit of 500 was actually useful.

Given that we’re on a Storj forum, I’m following Storj usage. Though I know it might be confusing.

34

Indeed you are right :slight_smile:, I wasn’t clear.
When I said “I was surprised by your answer since, at the beginning, I assumed I should more focus on reducing downloads than uploads if my laptop connection is slow”, I was talking about reducing ingress Internet traffic (“uploads” from customers’ point of view).
My thought was (but maybe wrong, I’m still learning a lot about how network works):

Browsing, Streaming and other ‘common usage’ of my laptops/desktop require more ingress Internet traffic than egress ; So I should reduce this ‘ingress’ Internet traffic.

But maybe I’m totally wrong and ‘common Internet activity’ requires more egress traffic than I think…

My mistake, @Toyoo. I spoke (wrote) too fast and didn’t think about how “downloads” and “uploads” words would be interpreted.

35

did the low latency tcp configuration help…? it should be pretty immediate if it works… and very clear.
after you do the sysctl --system to load the changes

i haven’t gotten around to setting up the ipv4 tcp connection limit, but i did realize i actually may have a VM that could be useful to test it on, which chokes even my fiber or local network with to many connections, tho maybe it’s related to that it uses encrypted tunnels and thus affects cpu… no matter i’m sure if i restrict the connections of the vm’s tcp then it should stop disrupting the other vm’s.

36

I didn’t test it yet.
Before changing any network setting, I want to understand all of the potential impacts.

Now, I have 3 options:

  1. Limit bandwidth (speed) with linux tools such as tc or wondershaper (on OS/system level).
  2. Limit number of concurrent connections by using --storage2.max-concurrent-requests Storj parameter (on Storj level).
  3. Set “low latency mode” (OS/system level). More explanation here about “net.ipv4.tcp_low_latency” setting.
37

tcp_low_latency - BOOLEAN
This is a legacy option, it has no effect anymore.

https://www.kernel.org/doc/Documentation/networking/ip-sysctl.txt

Out of the remaining two - “max-concurrent-requests” if it’s still doing something will only limit ingress.

tc can limit both, though IMO you’d be better off replacing your router with one that can do QoS/shaping and then giving the node a lower priority, so that it would be able to saturate your connection, when you are not using it, but your traffic would get priority. I have done this for my backup line that only has 12mbps upload - I just prioritized other traffic over the node.

2 Likes
38

Unless you’re explicitely sending data to a remote server (sending files to the cloud, sending huge e-mails, etc.) that’s correct.

However, another thing to keep in mind is that on slow connections normal browsing could become difficult if one process saturates egress.
And that’s because even though surfing the web or watching video streams mostly uses ingress traffic, very small (but frequent) packets of egress are needed to acknowledge network queries and request next packets.

If egress is saturated, these small packets tend to stack up somewhere, and get sent too late for a smooth communication between your brower and remote servers.
That’s usually not an issue on fiber connections, but it’s a real problem on ADSL or modest VDSL connections…

So IMHO, ideally your node should be allocated something like 80% (to be fine tuned) of both your egress & ingress bandwidth when you need to use the Internet.

1 Like
39

That is exactly the kind of explanation I needed :slight_smile:
Even if this is what I assumed, I wasn’t sure at all this would be the way it works.
Thank you so much :slight_smile:

1 Like
40

the 80% limit on upload might work, it might need more… ofc it will reduce your node’s ability to upload stuff then, but i suppose if you turn it on and off depending on usage it might be a fairly okay way to solve the issue.

the tcp overhead issue is pretty common on DSL mostly because in many cases you get like 50/5 mbit meaning if you are doing full download then your entire upload will be saturated just for overhead, which is usually around 10% on avg (a bit less but 10% is easy to remember)

thus in cases of ISP lacking a fundamental understanding of the requirements of their customers, so if we imagine two connections one being the storj node trying to upload ( egress) and the user trying to download(ingress) and for simplicity we will imagine this as two connections.

trying to describe this as simply…
storj node uploading, so thats upload + 10% of overhead coming back through download
user trying to download, so that download + 10% of overhead going out as upload

both having 1 connction so would get 1/2 in either direction as minimum.
so the storj node would get 50% of the upload and then take 10% of the download

the user would then get access to 90% of the upload, however that would required 100% of the upload, so because only 50% is allotted for his connection, his max of the download is then 50%. of the initial 50mbit even tho the storj is only uploading with 2.5mbit.

you really want a lot more than 1/10 upload to download… from a internet connection…
2/10 is acceptable, but kinda the lowest i would ever accept especially if i expect to be uploading anything, you really want like 4/10 upload to dl.

not sure running a storage node on DSL is that viable in most cases… i mean you get paid for egress, which is usually very low… and if you use the egress to hard you cannot get ingress through…
and if its really crappy DSL then no matter how little you upload, you will cut down in your download speed, no matter how much you bandwidth limit it…

so really should take that into account…

what speed did you say your internet was?