Gateway: metainfo error: Too Many Requests

I was deleting some files using the gateway and came across the error “metainfo error: Too Many Requests”.

What exactly does this mean?

2020-01-29T20:29:03.524+0100    ERROR   gateway error:  {"error": "metainfo error: Too Many Requests", "errorVerbose": "metainfo error: Too Many Requests\n\tstorj.io/uplink/metainfo.(*Client).Batch:1118\n\tstorj.io/uplink/storage/streams.(*streamStore).Delete:452\n\tstorj.io/uplink/storage/streams.(*shimStore).Delete:56\n\tstorj.io/uplink/storage/objects.(*objStore).Delete:86\n\tstorj.io/uplink/metainfo/kvmetainfo.(*prefixedObjStore).Delete:49\n\tstorj.io/uplink/metainfo/kvmetainfo.(*DB).DeleteObject:144\n\tstorj.io/storj/lib/uplink.(*Bucket).DeleteObject:116\n\tstorj.io/storj/pkg/miniogw.(*gatewayLayer).DeleteObject:120\n\tstorj.io/storj/pkg/miniogw.(*layerLogging).DeleteObject:113\n\tgithub.com/minio/minio/cmd.deleteObject:241\n\tgithub.com/minio/minio/cmd.objectAPIHandlers.DeleteObjectHandler:1484\n\tnet/http.HandlerFunc.ServeHTTP:2007\n\tgithub.com/gorilla/mux.(*Router).ServeHTTP:212\n\tgithub.com/minio/minio/cmd.securityHeaderHandler.ServeHTTP:657\n\tgithub.com/minio/minio/cmd.rateLimit.ServeHTTP:642\n\tgithub.com/minio/minio/cmd.pathValidityHandler.ServeHTTP:602\n\tgithub.com/minio/minio/cmd.httpStatsHandler.ServeHTTP:541\n\tgithub.com/minio/minio/cmd.requestSizeLimitHandler.ServeHTTP:66\n\tgithub.com/minio/minio/cmd.requestHeaderSizeLimitHandler.ServeHTTP:91\n\tgithub.com/minio/minio/cmd.crossDomainPolicy.ServeHTTP:51\n\tgithub.com/minio/minio/cmd.redirectHandler.ServeHTTP:243\n\tgithub.com/minio/minio/cmd.minioReservedBucketHandler.ServeHTTP:301\n\tgithub.com/minio/minio/cmd.cacheControlHandler.ServeHTTP:270\n\tgithub.com/minio/minio/cmd.timeValidityHandler.ServeHTTP:371\n\tgithub.com/rs/cors.(*Cors).Handler.func1:200\n\tnet/http.HandlerFunc.ServeHTTP:2007\n\tgithub.com/minio/minio/cmd.resourceHandler.ServeHTTP:492\n\tgithub.com/minio/minio/cmd.authHandler.ServeHTTP:300\n\tgithub.com/minio/minio/cmd.reservedMetadataHandler.ServeHTTP:134\n\tgithub.com/minio/minio/cmd/http.(*Server).Start.func1:108\n\tnet/http.HandlerFunc.ServeHTTP:2007\n\tnet/http.serverHandler.ServeHTTP:2802\n\tnet/http.(*conn).serve:1890"}

I’m also getting a “Too Many Requests” error after I tried to upload some files with the gateway. Now my gateway won’t start. This is on the US satellite.

2020-01-30T11:56:08.612-0500 WARN Failed to contact Satellite. Perhaps your configuration is invalid?
2020-01-30T11:56:08.612-0500 FATAL Unrecoverable error {“error”: “bucket: metainfo error: Too Many Requests”, “errorVerbose”: “bucket: metainfo error: Too Many Requests\n\tstorj.io/storj/uplink/metainfo.(*Client).ListBuckets:311\n\tstorj.io/storj/uplink/metainfo/kvmetainfo.(*Project).ListBuckets:136\n\tstorj.io/storj/lib/uplink.(*Project).ListBuckets:138\n\tmain.checkCfg:181\n\tmain.cmdRun:156\n\tstorj.io/storj/pkg/process.cleanup.func1.2:278\n\tstorj.io/storj/pkg/process.cleanup.func1:296\n\tgithub.com/spf13/cobra.(*Command).execute:826\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:914\n\tgithub.com/spf13/cobra.(*Command).Execute:864\n\tstorj.io/storj/pkg/process.Exec:74\n\tmain.main:365\n\truntime.main:203”}

I read in another post that this is related to a bug that they are fixing:

2 Likes

How are these connected?
The full storagenode has nothing to do with the gateway not working correctly.

It was my impression that the devs disables some stuff on the satellites to temporarily prevent more data from going into the nodes while they work on a real fix to prevent nodes from over filling. I could be wrong. Maybe it’s unrelated. My gateway does seem to be connecting now.
Edit. I guess they are still working on it. Connects but failed to upload a file.
Error: metainfo error: not enough nodes: requested 110 found 0

Seems like that could be true… So it is an indirect connection between those issues.
Not all posts from heunland were posted or read by me when I answered.

However, as you can see my issue in this thread is older than the last release. It is almost a day old.
So while current issues could be related to the current problems, my initial problem was not connected.

I just got my Tardigrade invite today and am testing out restic to the s3 backend. I’m getting the exact error:

2020-02-03T20:59:33.168-0700    ERROR   gateway error:  {"error": "metainfo error: Too Many Requests; metainfo error: Too Many Requests", "errorVerbose": "group:\n--- metainfo error: Too Many Requests\n\tstorj.io/uplink/metainfo.(*Client).Batch:1139\n\tstorj.io/uplink/storage/streams.(*streamStore).Put:195\n\tstorj.io/uplink/storage/streams.(*shimStore).Put:49\n\tstorj.io/uplink/stream.NewUpload.func1:53\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57\n--- metainfo error: Too Many Requests\n\tstorj.io/uplink/metainfo.(*Client).Batch:1139\n\tstorj.io/uplink/storage/streams.(*streamStore).Put:195\n\tstorj.io/uplink/storage/streams.(*shimStore).Put:49\n\tstorj.io/uplink/stream.NewUpload.func1:53\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}

Gateway is v0.31.12. The backup is still proceeding despite these errors so I’ll let it continue and see how far it gets.