IAM in S3 Gateway

Developer Discussions troubleshooting
1

I’m attempting to proxy the S3 Gateway HTTP interface, and I’ve noticed that there is no way to set public read permissions on buckets/objects. In order to provide authorization to the S3 Gateway, I’ve installed the ngx-aws-auth nginx plug-in from https://github.com/anomalizer/ngx_aws_auth; however the plug-in requires the IAM access key.

Are public read permissions or IAM available in S3 Gateway? I’m unable to find a modern solution that doesn’t require IAM.

Many thanks in advance for anyone with any input!

3

If I don’t understand wrongly the IAM access key that you mentioned is the aws_access_key configuration parameter.

Assuming that, you can generate a restricted readonly scope and used in the gateway configuration for getting the aws_access_key to be used.

For generating a restricted scope you need use uplink share --readonly pointing to the directory that contains the gateway configuration with the --config-dir command line argument.

4

I finally had a chance to test this out:

./uplink share --readonly returned API Key, Scope among other parameters.

nginx configuration:

server {
    ...
    aws_sign;
    aws_access_key [key from uplink];
    aws_key_scope [scope from ngx-auth-aws generate_signing_key script];
    aws_signing_key [key from ngx-auth-aws generate_signing_key script];
    aws_endpoint host:port;
    ...
}

I’m noticing that while the terminology is the same, Uplink is using a different set of mechanisms than the S3 Gateway, so I don’t think it’s possible to use access tokens, scopes, etc from Uplink in S3 Gateway.

I could be wrong, I’ll keep testing this out but this is how it seems to be now.

5

I’ve just tried and I realized that the AWS access key and secret key are the same if you change the scope, however the restrictions of the new scope applies.

If you replace the scope of the gateway configuration file by the the scope that uplink share --readonly... outputs and you run the gateway then the AWS S3 clients can only read the data.

What I observed is that the operations are denied as expected and the gateway log shows it, for example

2020-01-15T09:03:26.406+0100    ERROR   gateway error:  {"error": "metainfo error: Unauthorized API credentials; metainfo error: Unauthorized API credentials", "errorVerbose": "group:\n--- metainfo error: Unauthorized API credentials\n\tstorj.io/storj/uplink/metainfo.(*Client).Batch:1118\n\tstorj.io/storj/uplink/storage/streams.(*streamStore).Delete:452\n\tstorj.io/storj/uplink/storage/streams.(*streamStore).Put:85\n\tstorj.io/storj/uplink/storage/streams.(*shimStore).Put:49\n\tstorj.io/storj/uplink/stream.NewUpload.func1:53\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57\n--- metainfo error: Unauthorized API credentials\n\tstorj.io/storj/uplink/metainfo.(*Client).Batch:1118\n\tstorj.io/storj/uplink/storage/streams.(*streamStore).Delete:452\n\tstorj.io/storj/uplink/storage/streams.(*streamStore).Put:85\n\tstorj.io/storj/uplink/storage/streams.(*shimStore).Put:49\n\tstorj.io/storj/uplink/stream.NewUpload.func1:53\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}

But the AWS S3 client doesn’t get an error message that indicates the unauthorization. For example, trying to create a new bucket, I’ve got

make_bucket failed: s3://test-2 An error occurred (InternalError) when calling the CreateBucket operation (reached max retries: 4): We encountered an internal error, please try again.
6

So that we’re on the same page, what do you mean by:

If you replace the scope of the gateway configuration file by the the scope that uplink share --readonly... outputs and you run the gateway then the AWS S3 clients can only ready the data.

7

Sorry a typo in the end, I fixed it. I wanted to say

… can only read the data.

I meant

  1. You generate a limited readonly scope with uplink share --readonly.
  2. Open gateway configuration file and replace the scope field value by the scope value that you got in step 1.
1 Like
8

I tried changing the S3 Gateway to use the scope provided by uplink share --readonly with no further success.

I believe the problem lies within the generate_signing_key script from ngx_aws_auth.

Attempting to use the scope/keys generated by uplink share --readonly in the nginx configuration results in this error:

<?xml version="1.0" encoding="UTF-8"?>
<Error>
  <Code>AuthorizationQueryParametersError</Code>
  <Message>Error parsing the X-Amz-Credential parameter; the Credential is mal-formed; expecting "&lt;YOUR-AKID&gt;/YYYYMMDD/REGION/SERVICE/aws4_request".</Message>
  <Key />
  <BucketName />
  <Resource>[redacted]</Resource>
  <RequestId>[redacted]</RequestId>
  <HostId>[redacted]</HostId>
</Error>

…so obviously those can’t be used. But using the values from the script results in nginx results in:

<?xml version="1.0" encoding="UTF-8"?>
<Error>
  <Code>SignatureDoesNotMatch</Code>
  <Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message>
  <Key />
  <BucketName />
  <Resource>[redacted]</Resource>
  <RequestId>[redacted]</RequestId>
  <HostId>[redacted]</HostId>
</Error>

As for aws_access_key in nginx, it doesn’t seem to matter what I use there, the errors are the same. I’ve tried using the Access Key and Secret Key from S3 Gateway as well as API Key and “Enc Access” from the uplink share --readonly output as well.

I’m not sure if the problem is being caused by the nginx mod, the values I’m using or the results of the generate_signing_key script.

All I’m trying to do is proxy the bucket in the same way that a “static hosted website” would be proxied but I think the IAM requirement of this nginx mod isn’t compatible with S3 Gateway (yet).

9

Unfortunately I don’t have any knowledge in such nginx mod, so I cannot say.