Programmatic Granting of new Access keys

awcchungster · November 9, 2021, 6:31am

The flow I am trying to create is have an API server create shop term write only access grants for users. These users will leverage the S3 gateway to upload a blob to a preassigned bucket and folder path they have access to. Their access token should expire within 30 minutes.

What access grant permissions have the ability to make new access grants?

Alexey · November 9, 2021, 8:29am

Hello @awcchungster ,
Welcome to the forum!

You can create a derived access grant from any access grant, but it will be limited by the scope of the original access grant. For example, if you have a root access grant, you can create a new access grant limited to any buckets/objects.
If you have a limited access grant, for example for the one bucket, then you can create a new access grant, which will be limited by this bucket too, but you can make it more restrictive and/or allow an access to only some prefixes inside the bucket or even single objects.
See Sharing Your First Object - Storj DCS

Regarding usage of s3 protocol for sharing - it’s not great idea, because you skip all key Storj features like p2p distribution, related to the location of your clients, not to the selected region as with S3-compatible, granular permissions for shared buckets/prefixes/objects (because you cannot change the access on s3 level you will need to generate a new access grant with needed permissions and new keys), using the client-side encryption (with s3 it will be a server-side encryption).
If you would start fresh, I would recommend to use a native integration instead of limited s3 protocol.
See available libraries: Storj Client Libraries - Storj DCS and Storj - Third Party · GitHub

awcchungster · November 10, 2021, 12:10am

Could you clarify what you mean by the S3 access grant permission limitations?

From the Python bindings, couldn’t I create a new access grant with more restrictive permissions (such as to one folder) and get a S3 key for this?

granular permissions for shared buckets/prefixes/objects (because you cannot change the access on s3 level you will need to generate a new access grant with needed permissions and new keys)

Alexey · November 10, 2021, 7:02am

Of course you can. You need to create an access grant, then register it on s3-gateway to get new s3 keys, rather you can create only access grant and it would be enough in case of native integration.

The usual behavior for AWS s3: you creates keys once then attach permissions, here it would not work - you need to create a new access grant with required permissions, register it on s3-gateway and receive a new s3 keys. So you can’t manage permissions by AWS methods.
This is what I mean by “you cannot change the access on s3 level” - you always need to create a new access grant first anyway.

awcchungster · November 10, 2021, 7:18am

Thanks for the super fast response!

awcchungster · November 20, 2021, 5:30am

I’m having an issue with this process.

When I create an S3 access grant with a folder limitation, my S3 clients return an “access denied”.

uplink share --readonly=false --disallow-deletes --not-after +1h --register sj://[bucket]/[folder]/ --auth-service=https://auth.us1.storjshare.io

When I navigate to the bucket/folder, I am given the Access Denied. My Storj access granting the S3 has full permissions on the entire bucket. If I remove the folder limitation, the access grant can go into S3 bucket as expected.

Alexey · November 20, 2021, 7:48am

Is the [folder] exist? The grant can be generated even if the path doesn’t exist (to create it later for example).
I just checked and it works:

PS> ./uplink ls sj://web-test
OBJ 2020-08-25 09:58:05            0 tardigrade.png
PRE src/
OBJ 2020-08-25 09:28:41            0 index.html

PS> ./uplink ls sj://web-test/src
OBJ 2021-07-29 23:02:47           70 src/index.html
OBJ 2021-07-29 23:03:35        37284 src/StorjDCS.png

PS> ./uplink share --readonly=false --disallow-deletes --not-after +1h --register sj://web-test/src --auth-service=https://auth.us1.storjshare.io
Sharing access to satellite 12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@europe-west-1.tardigrade.io:7777
=========== ACCESS RESTRICTIONS ==========================================================
Download  : Allowed
Upload    : Allowed
Lists     : Allowed
Deletes   : Disallowed
NotBefore : No restriction
NotAfter  : 2021-11-20 11:39:51
Paths     : sj://web-test/src
...
========== CREDENTIALS ===================================================================
Access Key ID: ***
Secret Key   : ***
Endpoint     : https://gateway.us1.storjshare.io
Public Access:  false

PS> $Env:AWS_ACCESS_KEY_ID="***"
PS> $Env:AWS_SECRET_ACCESS_KEY="***"

PS> aws s3 --endpoint https://gateway.us1.storjshare.io ls
2020-08-25 09:27:58 web-test

PS> aws s3 --endpoint https://gateway.us1.storjshare.io ls s3://web-test/

An error occurred (AccessDenied) when calling the ListObjectsV2 operation: Access Denied.

PS> aws s3 --endpoint https://gateway.us1.storjshare.io ls s3://web-test/src
                           PRE src/

PS> aws s3 --endpoint https://gateway.us1.storjshare.io ls s3://web-test/src/
2021-07-29 23:02:47         70 index.html
2021-07-29 23:03:35      37284 StorjDCS.png

awcchungster · November 20, 2021, 3:28pm

My folder does not exist because I believe S3 is all one folder, but keys (file names) can be given a prefix. Is there an API call to create an empty folder in Storj?

The use case I have is a server creates the derived access grant with access to only a specific folder, and then the client uploads files to the folder. In case of interruptions, the client can list the files in that folder and compare against local to resume.

awcchungster · November 20, 2021, 4:20pm

I can confirm Alexey’s implementation using the S3 API. It must be something wrong with the clients I am using.

Thanks again. I really appreciate how fast your responses are

Alexey · November 21, 2021, 9:42am

There is no folders in the filesystem meaning. You have only buckets.
The path to the object is its name. For example sj://my-bucket/the/long/path/to/the/file.mp4:

sj://my-bucket is a bucket;
the/long/path/to/the/file.mp4 is a key of the object (name of the file);
- parts divided by slash are prefixes of the object;
the value of this key is a content of the file.

So, you cannot create an empty folder neither in AWS S3 nor in Storj. The only way to do this is to create an almost empty object with the needed prefix(es) and maybe hide it in the UI, if its name or metadata have attributes of stub object (there is no special meaning on the platform level - it doesn’t have an access to encrypted info anyway, so this is possible only on client side).

And there are several implementations of methods to create a bucket, for example:

github.com

storj/uplink-c/blob/27800a6fadf5e83921d0d3afcc16dea08467b532/bucket.go#L43

    
      
          	}
          
          
	bucket, err := proj.StatBucket(proj.scope.ctx, C.GoString(bucket_name))
          
          
	return C.UplinkBucketResult{
          		error:  mallocError(err),
          		bucket: mallocBucket(bucket),
          	}
          }
          
          
//export uplink_create_bucket
          // uplink_create_bucket creates a new bucket.
          //
          // When bucket already exists it returns a valid Bucket and ErrBucketExists.
          func uplink_create_bucket(project *C.UplinkProject, bucket_name *C.uplink_const_char) C.UplinkBucketResult { //nolint:golint
          	if project == nil {
          		return C.UplinkBucketResult{
          			error: mallocError(ErrNull.New("project")),
          		}
          	}
          	if bucket_name == nil {

See more in Storj Client Libraries - Storj DCS and Storj - Third Party · GitHub

If in the CLI, then mb - Storj DCS or mb — AWS CLI 1.22.10 Command Reference