I am unable to upload files to storj with neither gateway-mt nor gateway-st using aws-cli. I have two instances of gateway-st running, 1. locally, 2. remotely on fly. They are all backed by the same account.
However, I can upload using the uplink cli. Moreover, I can not upload using aws’ rust sdk with a different error, so i will open a separate issue later.
Not sure if this is relevant but the gateway running locally and on fly returns 503 when i try to get clusters’ health:
Could you please try to open http://localhost:7777 in your browser?
Perhaps you need to allow this port in the firewall.
Also, for aws CLI please add an endpoint --endpoint http://localhost:7777 to the commands, because it would work from a profile only if you setup an endpoint plugin: Setting Up AWS CLI Endpoint to Storj - Storj Docs.
It’s not a cluster instance, it’s a single node, so you will always get 503:
https://min.io/docs/minio/linux/operations/monitoring/healthcheck-probe.html#id3
A response code of
200 OKindicates that the MinIO cluster has sufficient MinIO servers online to meet read quorum. A response code of503 Service Unavailableindicates the cluster does not currently have read quorum.
You need to use a Node Liveness healthcheck instead.
I wanna repeat that i am using three different instances for testing:
And in the screenshot you can see that aws s3 ls works for all three of them. The issue we are trying to resolve is the fact that the server returns You must provide the Content-Length HTTP Header when using aws s3 cp
Please indicate if you are using boto3 version >= 1.36.0
I am using aws cli. which internally uses botocore/1.36.23.
aws-cli/1.37.23 Python/3.13.2 Linux/6.13.4-arch1-1 botocore/1.36.23
Please downgraded to v1.35.99 for now. You can find the explanation of this breaking change in boto3 in this github issue.
Specifically, please note
Disclaimer: The AWS SDKs and CLI are designed for usage with official AWS services. We may introduce and enable new features by default, such as these new default integrity protections prior to them being supported or handled by third-party service implementations. You can disable the new behavior with the WHEN_REQUIRED value for the request_checksum_calculation and response_checksum_validation configuration options covered in Data Integrity Protections for Amazon S3.
It is preferable to downgrade rather than disabling the new behavior with WHEN_REQUIRED, and this may not even work with Storj anyway.
Does storj have tracking issue for implementing the new integrity checks?
you can follow AWS CLI uses streaming trailer chunked requests on newer versions · Issue #89 · storj/gateway-st · GitHub