And have no idea what this error means or why it’s happening.
*This code worked untill today just fine.
This is the deletion code:
Parallel.ForEach(chunks, new ParallelOptions { MaxDegreeOfParallelism = 10, CancellationToken = cancellationTokenSource.Token }, (chunk) =>
{
// max chunk size is 100
// Build the request object with the keys in this chunk
var request = new DeleteObjectsRequest
{
BucketName = bucketName,
Objects = chunk.Select(k => new KeyVersion { Key = k }).ToList()
};
// Try to delete the objects in this chunk
try
{
DeleteObjectsResponse response = s3Client.DeleteObjectsAsync(request).Result;
if (response.HttpStatusCode != System.Net.HttpStatusCode.OK || response.DeleteErrors.Count > 0)
{
// This is a mechanizem for retry delete if needed
cancellationTokenSource.Cancel();
return;
}
}
catch (DeleteObjectsException ex)
{
cancellationTokenSource.Cancel();
throw;
}
catch (AggregateException ae)
{
ae.Handle(ex => {
if (ex is DeleteObjectsException)
{
cancellationTokenSource.Cancel();
throw ex;
}
else
{
throw ex;
}
});
}
});
Again this code used to work untill today, not sure what is going on…
I tested it with different numbers of files (5,10,100) and they all failed with the same error (the only difference is “Deleted objects: 100”, “Deleted objects: 5” etc.)
I want to emphasize this happened on any number of files I try to delete (5,10,100 it doesn’t seem to do a difference), the thing is, this exact code worked until today (worked for at least a week), so this is probably some new change in your side
We now have internal parallelism in the DeleteObjects handler of the S3 gateway. So you shouldn’t use your client-side parallelism, i.e. keep it at 1. Otherwise, you will exhaust your API rate limit.
I get the same error with MaxDegreeOfParallelism =1 (as long I try to send DeleteObjects and not DeleteObject).
In your S3 par, you specify you have full compatibility with DeleteObjects, and if this is true, you need to support the deletion of 1000 files in 1 DeleteObjects request.
Currently you are far from this (I can’t delete even 100 files in one request)
It seems we’ve set this new internal parallelism to be too aggressive. The team is working on reducing the parallelism level so you don’t hit the API rate limit. I’ll update when this change is deployed.
Thanks, Just to make sure, are you going to open the possibility to send 1000 files in a single DeleteObjects request to achieve the full compatibility stated here: gateway-st/s3-compatibility.md at main · storj/gateway-st · GitHub and in addition you need to support 100 DeleteObjects pers second (src from your limit page: “100 request per second” Usage Limits - Storj DCS Docs) so in total the new PR should allow deleting of 100000 files per second (100 DeleteObjects requests in each we have 1000 files)?
We will fix the issue where a single DeleteObjects request to the S3 gateway exhausts the API rate limit on the satellite.
This will allow passing large lists of objects (e.g. 1000 objects) to this single DeleteObjects request, but it will take a proportional time to complete, i.e. the larger the list, the more time it would take.
I doubt this would make it to deleting 100,000 objects per second.
If you implement additional parallelism on the client-side, it’s very likely to increase the chance of hitting API rate limit errors, so you should also implement some logic for handling them and throttling down the delete requests.
Why? in my client side I will throttle to not send more than 100 request per second, and since each request contain 1000 file delete request it effectively means that I can send 100,000 file delete request/second (obviously, it will take a bit longer to complete DeleteObjects of 1000 files then DeleteObjects of 100 file but it should be an ms different not more than this).
Currently, you are hitting the rate limit on the satellite, which is different than the rate limit on the S3 gateway.
The satellite API currently supports only deleting a single object or an entire bucket. So the S3 gateway implements DeleteObjects by calling the respective satellite API for each object separately. Improving the satellite API is on the roadmap, and when we get there, the delete performance will improve.
The team updated me that the change for reducing the internal parallelism in the S3 gateway has been deployed. Could you verify on your side that a single DeleteObjects request with a list of 1000 items does not fail anymore?
In addition, what happened if I will send 2 DeleteObjects with 1000 files parallel? it will translates to 2000 single DeleteObject requests and hit the rate limit on the satellite? (it’s better than the current situation, but we still need proper support for 100 requests/sec).
There is part of my code that does upload and a part that does delete, so if I send 1 DeleteObjects (of 1000 files) it means I can’t send any more requests (put/create/delete) until the processing of the DeleteObjects finished? this is a major problem…
Sending a single DeleteObjects of 1000 files should not exhaust the API rate limit on its own. We tested it on several satellites from several locations.
The internal parallelism is now set to 50. This means no more than 50 parallel requests to the satellite per DeleteObjects request. The default API rate limit is 100 req/s per satellite API pod. The satellite has several API pods, so the total API rate limit is much higher than 100 req/s.
If your application is so loaded that you often hit the rate limit, you can request an increase by opening a support ticket.
Is Storj production ready?
I’m considering using Storj in my production environment, and frankly, this made me a bit nervous, what is your SLA for me as a business?
I read the limit link and the S3 comparability link.
Until this conversation, I had no idea (this is not written in any formal place) that the way you implement DeleteObjects has this implication on my rate limit (And this is not trivial).
Do you have formal comprehensive promises on how the S3 should behave?
Do you have a formal channel where you update enough time ahead (need to be a minimum ~2 months) before changes that going to impact the S3 promises, how they will impact it, and give me enough time to prepare my production environment for the change?
Where in those documents I can find that a single DeleteObjects request translates to 50 behind the scene which effects my rate-limit?
I’m sorry but we don’t have this limit described in documentation yet but we are constantly working to improve our docs so I hope you will see it there soon.