Just saw this. We are aware that a good portion of users are encountering the rate limiting. You can request a rate limit increase, but we are also working on making the rate limit much higher.
The rate limit is annoying, but other than my tests I am not really doing anything important with it, so, not really worth wasting the time of support to ask for increase.
What more annoying is that the limit appears to be both new connections/ms and simultaneous connections. I could get around the first by simply inserting âsleepâ between opening new sessions, I could get around the second by using âxargs -Pâ to start all the sessions, but not trigger either of them is more difficult.
I am currently waiting for the nextcloud integration to get fixed. After that, if that rate limiting wonât be differently by then, I might have to request a rate limit increase. Because right now I canât even delete 10 files in nextcloud because half of those will still be there when I refresh, because it hits the rate limit and just doesnât perform its tasksâŚ
1 Like
Uhm, this is a fail, if it is rate limited, shouldnât the uplink queue these actions instead of assuming you never wanted them done?
1 Like
No! Queue the action would give you a slow response. It would look like the performance is bad. Instead we should just increase the limit to the point you donât hit it with normal operations.
Well with limiting there is no performance so it would be infinitely better. 
Rate limiting shouldnât have been added in the first place; when I mentioned IOps amplification of the system, nobody saw it as a problem.
If you intend to keep the limiting, but set it higher, I would still suggest to implement some kind of wait and retry instead of cancel as the mechanism. Cancel just makes for a bad user/customer experience.
2 Likes
A queue would be better than the current way of just rejecting the request. If I want to delete a bunch of files (and not do it one by one as that takes one second per file), I end up with only some files deleted and have to retry doing it.
Whatever the new limit is, when I hit it, Iâd rather the uplink waited in line instead of just failing, making me run the script again.
5 Likes
Thatâs also my point. A limit rejecting commands is still a limit, no matter how high. And apparently the S3 endpoints are not designed to handle the way the storj s3 gateway does the rate limit, which results in operations not being executed. This can never be what you want. You should rather have customers complaining about poor performance than customers complaining about only parts of their commands being executed.
4 Likes
I totally agree. From a users perspective I would prefer slow execution over no execution.
To me, a good middle ground would be the Leaky bucket algorithm: https://en.m.wikipedia.org/wiki/Leaky_bucket
So that everyone is free to issue tons of queries for a short amount of time when needed, and it still would prevent âmeanâ users from continuously spamming the server.
That seems OK. The problem with the current limit and behavior is that it is complicated to limit both the request rate and number of simultaneous requests.
Letâs say I want to upload, download or delete a lot of small files. Deleting one file takes about a second, so it would take rather long to delete a lot of them. I could run the commands in parallel, but now I have a problem:
If I do simple
for ...; do
sleep 0.1
uplink .... &
done
this limits the rate to 10 connections/s, but if the request takes longer, the number of open connections goes up to the point where new connections are dropped.
OK, so how about
echo "$list" | xargs -P29 uplink ...
But now xargs starts the 29 connections at once, tripping the rate limit, while staying within the simultaneous open connection limit. This results in the first commands being dropped, but the rest working fine.
Your proposal would work with the xargs method - allow a burst of connections and new connections will be opened only when old commands are completed.
If i upload a file, i need to know its uploaded. It should retry and wait and etc. Not just give up.
My point was that the limits on both simultaneous connections and connections per millisecond is a bit difficult to do using standard tools.