PutObject fails when going through self-hosted gateway-st

Developer Discussions troubleshooting
1

I have setup gateway-st and i can confirm that the setup is working properly as i can list buckets and see all my buckets.

However, when i try to put objects I get an error on the client side: dispatch failure

Details:

The error doesn’t give much information. Please let me know how i can gather more details

2

Did you specify the endpoint of your Gateway-ST instance? It’s http://localhost:7777 by default, so you need to use it and run your load on the same host or reconfigure it to listen on a network interface, if you want to connect from another host.

Also, please check your access grant with uplink:

uplink access inspect 12fkgjfgiogrgkrgji.....

It would print all integrated permissions. Perhaps you configured it with a read only permissions.

3

uplink access inspect returns the following:

{
  "satellite_addr": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777",
  "encryption_access": {
    "default_key": "<manually redacted>",
    "default_path_cipher": "ENC_AESGCM"
  },
  "api_key": "<manually redacted>",
  "macaroon": {
    "head": "DGWOU7jv4C3X3FHLB00Aa9MtDakYMW5QrSp6wgsxUmo=",
    "caveats": [
      {
        "not_before": "2024-12-31T14:58:39.796Z",
        "nonce": "UCOl+g=="
      }
    ],
    "tail": "O0A-8kYBiar2y8b3HigTLyl2rKxGikCLnZ16a4GX1VU="
  }
}

We also confirmed that the configuration is correct by uploading a file through the aws cli. Which means the error only occurs when using aws’ rust sdk.

I also ran the gateway locally once with debug mode and got the following trace when sending a PutObject request:

API: SYSTEM()
Time: 13:53:56 UTC 01/01/2025
DeploymentID: 88309807-75f2-4eb1-83ff-9b5fe248aecf
Error: XML syntax error on line 2: invalid character entity & (no semicolon) (*xml.SyntaxError)
       5: /go/pkg/mod/storj.io/minio@v0.0.0-20241126095350-0039fe402364/cmd/handler-utils.go:54:cmd.parseLocationConstraint()
       4: /go/pkg/mod/storj.io/minio@v0.0.0-20241126095350-0039fe402364/cmd/auth-handler.go:328:cmd.CheckRequestAuthTypeCredential()
       3: /go/pkg/mod/storj.io/minio@v0.0.0-20241126095350-0039fe402364/cmd/auth-handler.go:274:cmd.checkRequestAuthType()
       2: /go/pkg/mod/storj.io/minio@v0.0.0-20241126095350-0039fe402364/cmd/bucket-handlers.go:537:cmd.ObjectAPIHandlers.PutBucketHandler()
       1: net/http/server.go:2220:http.HandlerFunc.ServeHTTP()
4

Interesting. What’s difference between S3 SDK and aws CLI?

Seems the error

do not have any relation to the permissions. Perhaps you really have a syntax error in the call of the method?

Would you mind to share a code? Then I would be able either to analyze myself or forward it to the developers team?

5

I am not sure if there’s any difference between the sdk and cli other than the fact that one is an sdk and the other is cli lol. but they are both from aws itself and can be considered source of truths.

Here’s a small snippet using aws’ rust sdk to put files:

    let creds = Credentials::new(
        "<access key id>",
        "<secret key>",
        None,
        None,
        "custom",
    );

    let shared_creds = SharedCredentialsProvider::new(creds);

    let config = aws_config::load_defaults(BehaviorVersion::v2024_03_28())
        .await
        .into_builder()
        .credentials_provider(shared_creds)
        .endpoint_url("<gateway url>")
        .region(Region::new("global"))
        .build();

    let client = Client::new(&config);

    let res = client
        .put_object()
        .bucket("test-bucket")
        .body(ByteStream::from_path("<file path>").await.expect("file to be loaded as bytes"))
        .key("<key>")
        .send()
        .await
        .expect("the operation to succeed");

    println!("{res:?}");

NOTE: This code works when putting object in gateway-mt, it only fails for gateway-st.

6

Yes, which is why I asked you to inspect an access grant from the Gateway config - it could be a read-only, or limited to some buckets.

When you created S3 credentials for Gateway-MT, they can be used only against Gateway-MT, and will not work with Gateway-ST, unless you provide them in the config of the Gateway-ST instead of autogenerated ones.

They both works in the same way and shared the same code, so, you have an access grant under S3 credentials. The difference where and how this access grant is stored on the Gateway. For Gateway-MT your access grant is encrypted with S3 Access Key and stored on our servers, so Gateway-MT authenticate you with provided S3 Access Key and S3 Secret Key, decrypt your access grant and use it to access your data.
In the case of Gateway-ST your access grant is stored in the Gateway-ST config, everything else is working as with a Gateway-MT.
If you have used different access grants for registering on Gateway-MT and in the Gateway-ST, they may have different caveats (permissions) and even an encryption phrases (then it will see only objects, encrypted with that other encryption phrase and vice versa).

If you used the inspected access grant, then I do not see any restrictions except “not_before”. So, it should be able to put the object.
Could you please set to use a path style instead of virtual hosts for S3 in the SDK? Gateway-ST doesn’t work with a virtual hosts style without an additional configuration related to a reverse-proxy.

7

can you elaborate on that

8

The generic information is there: Virtual hosting of buckets - Amazon Simple Storage Service
The closest answer to how to enable paths style:

And here is a some example (I think you use Rust):

1 Like
9

This log output seems to indicate it’s calling PutBucketHandler, with the specific error being that the location constraint is invalid. Are you sure your code is calling the object endpoint, not trying to call the bucket endpoint with the XML for PutObject, perhaps?

1 Like
10

It turns out indeed that the client is trying to use vhost-style requests but it ends up sending the request to the endpoint like http://localhost:7777/myfile.txt (with Host header mybucket.localhost) instead of http://localhost:7777/mybucket/myfile.txt. It may be possible to set things up to use vhost style with gateway-st, but it would require properly set up domain including wildcards for subdomain (as Alexey mentioned above).

Here’s a way to force path style requests when constructing the client:

use aws_config::BehaviorVersion;
use aws_sdk_s3::Client;

#[::tokio::main]
async fn main() {
    let sdk_config = aws_config::defaults(BehaviorVersion::v2024_03_28())
        .region("us-east-1")
        .load()
        .await;

    let config = aws_sdk_s3::config::Builder::from(&sdk_config)
        .force_path_style(true)
        .endpoint_url("http://localhost:7777")
        .build();

    let client = Client::from_conf(config);

    // do something with client
2 Likes