S3 Compatability Copy

I’ve been working on a photo/video sharing application where assets are private unless shared. I am using lambda on a node runtime so I’ve been using the S3 compatible commands and so far so good. However now that I am trying to implement sharing I’ve learned that while I can make a “public” bucket by sharing a path with the Uplink CLI on my local machine I am unable to copy file to it using the S3 API according to this: Supported S3 Commands | Storj Docs.

So right now I have a bucket set up the way I want it but have no way to move files from another bucket to it without using the Uplink CLI which is a real headache in my node lambda environment as the Node SDK is just being a wrapper for the Go SDK one meaning I need to worry about including a Go runtime with node-gyp. Either that or take a crash coarse in Go and using a lambda running on the Go runtime.

The copy command not being included seems wrong to me as it is pretty basic and oft used functionality for S3. I wanted to check with everyone here to see if I’m missing something and there is a way to move files between buckets with AWS API or if I really need to explore on of the options above.

Hi @Larryfromalaska, welcome to the Storj Forum! Currently server side copy is under construction and it might be available soon! Here is a link to the server side copy design draft discussion here in our forum Design Draft: Server-side Copy.

1 Like

Thank you for the update. I have decided to attempt using the Go SDK rather than trying to make the node wrapper work in AWS Lambda.

I do have another question if anyone will to help. I have some function that will take an access and object key and attempt to make a publicly accessible URL. I think I know how to make the grant I need and serialize it for further use but do not know how to get the SDK to give me a browser URL like CLI does.

func ShareObject(ctx context.Context, accessGrant, bucketName, uploadKey string) (string, string, error) {
	var sharedAccess *string = nil
	var publicUrl *string = nil

	// Request access grant to the satellite with the API key and passphrase.
	access, err := uplink.ParseAccess(accessGrant)
	if err != nil {
		return *sharedAccess, *publicUrl, fmt.Errorf("could not request access grant: %v", err)
	}

	// Create the path to be share with permissions and generate a grant.
	sharedPrefix := uplink.SharePrefix{ 
		Bucket: bucketName,
		Prefix: uploadKey,
	}
	permission := uplink.ReadOnlyPermission()
	shareGrant, err := access.Share(permission, sharedPrefix)

	// Serialize that grant for further use and to later revoke if needed.
	serialized, err := shareGrant.Serialize()
	sharedAccess = &serialized

	// How do I get the browser url I would get with the CLI by including the --url flag? 

	return *sharedAccess, *publicUrl, nil
}

Again I just starting using Go yesterday so I’m open to any pointers here. I’ve decided since I’m using the uplink CLI I might as well not copy the object to a bucket with universal shared link that i just append an object key to. Instead I will create a public URL for that one object in place and store the serialized access used to make in case I later need to revoke it.

You need to use uplink/access.go at c4e80d7a3a8703d4976eaad4a0e1de32ca48c4f8 · storj/uplink · GitHub
Here you can see an example: uplink/edge_test.go at c4e80d7a3a8703d4976eaad4a0e1de32ca48c4f8 · storj/uplink · GitHub

You can of course do not register the created Access on edge services, and use the serialized access grant directly like https://link.us1.storjshare.io/s/1fhfurhurifofkrfrgtgtbpvgbgobgbpgbpbobpotihjth9gjrferf8tgjr[....]/8f379b26ee4e7f3a7e48aaef2e8369256a44de67.jpeg, but the access grant have a derived API key, derived encryption phrase and can be abused with some code change. Especially if you would use a root access grant, not derived - the malefactor can get an access to entire project.

So it’s better to register and get an access key and secret access key from the edge service and got an URL like https://link.us1.storjshare.io/s/jvj7t3mp6ingvswycgqzqsmwmrdq/forum/8f379b26ee4e7f3a7e48aaef2e8369256a44de67.jpeg
It’s more safe, because here you will use only the access key. The access grant will be decrypted using this access key and used to gain an access to file(s)/prefix(es)/bucket(s)

1 Like

Hello Alexey,

Thanks again for the quick response, I think I am very close now. My function now looks like this:

func ShareObject(ctx context.Context, accessGrant, bucketName, uploadKey string) (serializedAccessGrant *string, credentials *edge.Credentials, err error) {
	// Request access grant to the satellite with the API key and passphrase.
	access, err := uplink.ParseAccess(accessGrant)
	if err != nil {
		return nil, nil, fmt.Errorf("could not request access grant: %v", err)
	}

	// Create the path to be share with permissions and generate a grant.
	sharedPrefix := uplink.SharePrefix{
		Bucket: bucketName,
		Prefix: uploadKey,
	}
	permission := uplink.ReadOnlyPermission()
	shareGrant, err := access.Share(permission, sharedPrefix)

	// Serialize that grant for further use and to later revoke if needed.
	serialized, err := shareGrant.Serialize()
	serializedAccessGrant = &serialized

	edgeConfig := edge.Config{
		AuthServiceAddress: "auth.us1.storjshare.io:443",
	}
	edgeOptions := edge.RegisterAccessOptions{
		Public: true,
	}
	// error happens here "context was cancled"
	credentials, err = edgeConfig.RegisterAccess(ctx, shareGrant, &edgeOptions)
	if err != nil {
		return nil, nil, fmt.Errorf("error registering access grant: %v", err)
	}

	return serializedAccessGrant, credentials, nil
}

It works up to the point that I attempt to register the share grant. I’ve even tried using the serialized grant in the link as you have mentioned and that works as expected. However, for the reasons you stated, I would like to register that grant and use the public key in the link instead.

However RegisterAccess() is returning this error:

uplink: register access for edge services failed: context canceled

I am not using a certificate in my edge config options and I’m wondering if that may be the reason it is failing. I didn’t need one when using the CLI to do the same thing so I was hoping that would be the case here as I’m not really sure what I should use there.

Again, thanks for the help!

I should mention that the context used is Context.Background() and I am not canceling it anywhere so I believe it is happening within the function referenced above.

Could you try changing the port of auth.us1.storjshare.io:443 to auth.us1.storjshare.io:7777?

That did it! I will say that I got the port from the docs for Edge.Config so that may need to be updated unless I’m missing something.

// AuthServiceAddress: set a fixed DRPC server including port.
// valid is auth.[eu|ap|us]1.storjshare.io:443
// or a thirdparty-hosted alternative.

Thanks all the help!

2 Likes

Great to hear! I also tested out the code and made a simple example: share file programatically using storj linksharing service · GitHub

1 Like