Delete many S3 Files at once

[web4yougmbhch]

How to delete many files at once in a S3 Tardigrade Bucket?

aws s3 --endpoint=http://localhost:7777/ rm s3://bucket-name/*.pdf

does NOT work.

[Alexey]

As usual for s3:

aws s3 --endpoint http://localhost:7777 rm s3://test1/ --recursive 
aws s3 --endpoint http://localhost:7777 rb s3://test1

Or

aws s3 --endpoint http://localhost:7777 rm s3://test1/nested
delete: s3://test1/nested
aws s3 --endpoint http://localhost:7777 rb s3://test1

Or

aws s3 --endpoint http://localhost:7777 rb s3://test1 --force 

[web4yougmbhch]

@Alexey

–recursive

I cant delete all Entrys and also not with --force. I get this error on gateway terminal:

2020-02-17T14:18:06.521+0100 ERROR gateway error: {“error”: “bucket: metainfo error: Too Many Requests”, “errorVerbose”: “bucket: metainfo error: Too Many Requests\n\tstorj.io/uplink/metainfo.(*Client).GetBucket:218\n\tstorj.io/uplink/metainfo/kvmetainfo.(*Project).GetBucket:106\n\tstorj.io/storj/lib/uplink.(*Project).GetBucketInfo:142\n\tstorj.io/storj/lib/uplink.(*Project).OpenBucket:162\n\tstorj.io/storj/pkg/miniogw.(*gatewayLayer).DeleteObject:114\n\tstorj.io/storj/pkg/miniogw.(*layerLogging).DeleteObject:113\n\tgithub.com/minio/minio/cmd.deleteObject:241\n\tgithub.com/minio/minio/cmd.objectAPIHandlers.DeleteObjectHandler:1484\n\tnet/http.HandlerFunc.ServeHTTP:2007\n\tgithub.com/gorilla/mux.(*Router).ServeHTTP:212\n\tgithub.com/minio/minio/cmd.securityHeaderHandler.ServeHTTP:657\n\tgithub.com/minio/minio/cmd.rateLimit.ServeHTTP:642\n\tgithub.com/minio/minio/cmd.pathValidityHandler.ServeHTTP:602\n\tgithub.com/minio/minio/cmd.httpStatsHandler.ServeHTTP:541\n\tgithub.com/minio/minio/cmd.requestSizeLimitHandler.ServeHTTP:66\n\tgithub.com/minio/minio/cmd.requestHeaderSizeLimitHandler.ServeHTTP:91\n\tgithub.com/minio/minio/cmd.crossDomainPolicy.ServeHTTP:51\n\tgithub.com/minio/minio/cmd.redirectHandler.ServeHTTP:243\n\tgithub.com/minio/minio/cmd.minioReservedBucketHandler.ServeHTTP:301\n\tgithub.com/minio/minio/cmd.cacheControlHandler.ServeHTTP:270\n\tgithub.com/minio/minio/cmd.timeValidityHandler.ServeHTTP:371\n\tgithub.com/rs/cors.(*Cors).Handler.func1:200\n\tnet/http.HandlerFunc.ServeHTTP:2007\n\tgithub.com/minio/minio/cmd.resourceHandler.ServeHTTP:492\n\tgithub.com/minio/minio/cmd.authHandler.ServeHTTP:300\n\tgithub.com/minio/minio/cmd.reservedMetadataHandler.ServeHTTP:134\n\tgithub.com/minio/minio/cmd/http.(*Server).Start.func1:108\n\tnet/http.HandlerFunc.ServeHTTP:2007\n\tnet/http.serverHandler.ServeHTTP:2802\n\tnet/http.(*conn).serve:1890”}

[Alexey]

Yes, I know. Because of this:

@littleskunk

Rate Limiting
We have added a rate limit of 30 requests per project per second. If you need a higher limit please contact us.

You can request to increase the rate limit: https://support.tardigrade.io/hc/en-us/requests/new
Sorry for inconvenience!

This is discussed here:

Fixing Too Many Requests Error in s3 gateway

Is there plan to have S3 gateway support rate-limitting so we dont get errors saying “Too Many Requests”. I am using s3 gateway and some operations (backup pruning) are resulting in way too many rate-limit errors. I feel like I am DDOS-ing the service because of the number of errors I am seeing. Can S3 Gateway delay requests so it does not hit the rate-limit?

And the POLL is here:

[POLL] We recently introduced a rate limit for all requests to the satellite. What has been your experience so far?

We recently introduced a rate limit for all requests to the satellite. What has been your experience so far? poll

At the moment just run it several times, it will finally delete the content or/and bucket