Should I change max-concurrent-requests? >> Note: The implementation has changed. You shouldn't use this setting any longer. Leave it commented out

There is not one answer for everyone. If you have low powered hardware, you probably shouldn’t. In worst cases you might want to lower it. But if your hardware is fast and your connection is as well, it could be beneficial to raise this number.

How do I change the setting?

# Maximum number of simultaneous transfers
storage2.max-concurrent-requests: 7

Add these lines at the end of the config.yaml and play with the number a bit. You have to stop the node, make the change, then start the node again.

Why are there larger numbers in the error messages than what is set in the config.yaml?
The node only rejects uploads when the limit is reached, since you don’t want to deny paid download traffic. Additionally there is much more concurrency and overhead in upload traffic, so rejecting isn’t as much of a problem there.

How do I determine what the right setting is?
This really depends on hardware and connection. The biggest bottlenecks are connection speed and IO, but on low RAM devices RAM may play a role as well. It’s also useful to keep in mind your data cap if you have one. You’ll want to balance the rate of accepting transfers against the rate of interrupted transfers because your node was too slow or saturation of system resources. A few of us SNO’s have created a script together in order to monitor the success rates of your node based on the logs. You can find it here. https://github.com/ReneSmeekes/storj_success_rate You’re mostly looking to get the acceptance rate for uploads as high as possible, without the success rate for uploads and downloads going down. If your success rate is already low and especially if you have a data cap, it would be interesting to lower the concurrency number to make sure your bandwidth is focused on fewer transfer that will in fact succeed more often. I myself have an uncapped high speed connection with fairly fast hardware and found that 40 is probably a good setting for me, but a friend of mine has the exact opposite. Slow connection, lower powered hardware and a data cap. We are still testing his setup, but he has seen great results by dropping the number to 4 and the success rate has increased, which wastes less bandwidth on interrupted transfers and eventually will be more profitable. I hope this helps a bit. Higher is definitely not always better.

12 Likes

Is this advice still relevant? I think the current default value for this setting is zero, meaning “unlimited”.

It’s no longer accurate. With the introduction of dRPC you should leave this setting alone. Unlimited is the best setting for pretty much everyone. I haven’t spent much time looking into fringe cases since this changed. The only exception might be if you have super low success rates and high traffic while also having a low speed connection. Lowering this may get you better success rates, but the loss of traffic due to rejections is probably higher anyway. So basically, just leave it alone.

Heh, actually I just noticed one of my nodes was started being hammered by upload requests. I eye-balled it to at least a hundred simultaneous requests. This made impact on my I/O and the node became barely usable. I’ve set this switch to 30 for now and I’m observing how the situation develops—now at least I can browse the files.

Try to remove it at all

It was removed before I set it.

i tried setting this… shortly after starting my node, after looking a bit more at the config.yaml recently i found this.

config.yaml
!- # how many concurrent requests are allowed, before uploads are rejected. 0 represents unlimited.
!- # storage2.max-concurrent-requests: 0

excuse the !-

however it seems changes in the config.yaml before after the “end of file” makes a node reboot until the changes are removed.

so i would recommand making a config.yaml.bak if you are editing it.

and it does kinda make me wonder if one can even change it from the storage/config.yaml location at all, i think one might have to go through the docker commands to change the container config.yaml

i just cannot find where i read that guide in the documentation … lol

The only way to break it is with faulty settings. Check the logs to find out what’s wrong.

pretty sure i just removed the # from the “# storage2.max-concurrent-requests: 0”
line in the config.yaml, then when it didn’t work, i did a couple of retries and then restored config.yaml from my config.yaml.bak to successfully put the node online again.

i’ll give it another test next time i have to shut down my node.

and ill look at the logs tomorrow, maybe i should try and set my logs to debug, that would be an easy test to see if the storage/config.yaml actually does anything at present.

Made a small script to analyze the rate of success vs. the number of in-flight data transfers (both uploads and downloads) at the time the data transfer started. The result for the last 24-hour period:

 0- 9     -   99%   99%   99%   99%   99%   99%   98%   97%   96%
10-19   95%   90%   84%   67%   68%   69%   56%   65%   66%   63%
20-29   50%   48%   30%   52%   30%   39%   36%   41%   46%   42%
30-39   38%   57%   43%   62%   39%   31%   45%     -     -     -

So, clearly, my node’s success rate starts dropping at around 12 simultaneous connections and it still does make sense to set max-concurrent-requests.

1 Like

that’s interesting… the number of transfers completed also seems kinda relevant…

like say 66% success rate at 18, which is 200% more than 9, which is 96% success rate
however 96% is barely even 1.5x improvement from 66%

so in theory atleast the successful transfers would be higher at 18 than 9 over the same time.

then however one also have to take into account that the cancelled transfers will happen during max or near max load… thus the success rate number at 18 active transfers might much lower and then rest of the time when its not a peak load, the higher success rate improves the avg.

so yeah you really need to include successful transfers also, to make an even more informed decision.
or maybe your script does account for that…

did you upload the script somewhere… i would love to try and see what it says for my node.

Can’t upload the script, sorry. But it’s pretty simple. For each line of log:

  • if it’s upload started or download started, increment the counter for the number of current in-flight requests,
  • if it’s uploaded, downloaded, upload cancelled or download cancelled, decrement the above counter.
  • if it’s upload started or download started, memorize the satellite id, piece id and the current value of the above counter,
  • if it’s uploaded or downloaded, retrieve the memorized value of the counter for the given satellite id/piece id pair and memorize that there was a successful transfer at that number of in-flight requests.
  • if it’s upload canceled or download canceled, retrieve the memorized value of the counter for the given satellite id/piece id pair and memorize that there was a canceled transfer at that number of in-flight requests.

Then print out the rates, noting to avoid computing rates if there aren’t enough samples (I picked a threshold of 100, which very roughly sets the error bars at ±10%).

that was a waste of a perfectly good explanation…
i’m barely staying alive in command line linux… going to hold off on the custom scripting for a few weeks or months… thanks for trying tho.

and your info will most def make me reevaluate what the best setting is… but atm i’m keeping 80%+ with either 200 or unlimited, but maybe i can squeeze out a bit more performance out of my node with a better max connections,

maybe i should put it back at 40 see if that changes anything… ofc each shutdown will also affect my nodes performance… i really need to make a little test node… that i can torture and experiment upon.

I think this kind of diagnostics should be available in the node dashboard, so that every node operator can take informed decision. Maybe try submitting an idea for that at https://ideas.storj.io?

1 Like

i’m sure stuff like that will be integrated as people develop it… just like in the next version they will start to integrate the storj earnings utilities, who knows maybe they are using code that @BrightSilence wrote… xD this is open source… so if people make some good stuff for it… it will be assimilated into the storj collective… lol

i just like having lots of information to help me make more informed decisions…
been using netdata, which is pretty cool… i’m not totally happy with the mechanics of it… but it sure does give me access to more server information than i could have imagined lol…

not exactly storj related, but its helped me a lot in tuning my server for storj…

I am running 5 storagenodes behind 1 ip address with very weak broadband (compared to what I’ve seen here)
I’ve set max-concurrent-requests to 1 for each node but this is equivalent to 5 since there are 5 nodes. My plan is to add a storagenode for each disk but this means that eventually my broadband will be flooded. The only thing left to do seems to be to combine disks onto the current nodes?

A lot of people here discourage combining drives into a single storage node, but given a weak Internet connection and a single IP address, I’d probably do so as well. With five drives it might even make sense to make a RAID5—the overhead becomes small and your storage node will be able to survive a drive failure, which would otherwise be the biggest risk with pooled storage.

Are your nodes vetted?

1 Like

Unfortunately, I have a motley collection of different size and type drives.
2 nodes vetted, 1 node vetted except on stefan-bentan, 1 only vetted on saltlake and 1 new

That sounds like your connection isn’t up to snuff for the minimum requirements.

  • 5 Mbps bandwidth upstream
  • 25 Mbps bandwidth downstream

Having a single node would speed up vetting, then, but this ought to be a one-time concern…

It is possible to do an equivalent of RAID5 with drives of different sizes, e.g. by using btrfs or manually combining storage with LVM and device mapper in a similar way to Synology’s SHR. But I’d be worried about stability of the solution. Separate nodes aren’t as bad in this context.