S3 put stopped working with missing Content-Length HTTP header

:red_exclamation_mark: PutObject fails with “Missing Content-Length” — previously worked for years

We’re consistently getting the following error when uploading to Storj using the S3-compatible gateway. This code previously worked for months without modification.

Error:
An error occurred (MissingContentLength) when calling the PutObject operation: You must provide the Content-Length HTTP header.


:white_check_mark: Previously Working (s3fs + pandas)

import pandas as pd
import s3fs

storage_options = {
    "key": "YOUR_ACCESS_KEY",
    "secret": "YOUR_SECRET_KEY",
    "client_kwargs": {"endpoint_url": "https://gateway.storjshare.io"},
}

df = pd.DataFrame({"sensor": ["temp"], "value": [72.5]})
df.to_parquet("s3://lookout/test_upload.parquet", storage_options=storage_options)

This approach worked reliably until recently. We’re now consistently seeing errors that indicate Content-Length is missing.

Current Attempt (boto3, explicitly setting ContentLength)

import boto3
import pandas as pd
import os
import tempfile

df = pd.DataFrame({"a": [1, 2], "b": [3, 4]})
with tempfile.NamedTemporaryFile(suffix=".parquet", delete=False) as tmp_file:
    df.to_parquet(tmp_file.name)
    file_size = os.path.getsize(tmp_file.name)

    with open(tmp_file.name, "rb") as f:
        s3 = boto3.client(
            "s3",
            endpoint_url="https://gateway.storjshare.io",
            aws_access_key_id="YOUR_ACCESS_KEY",
            aws_secret_access_key="YOUR_SECRET_KEY",
        )
        s3.put_object(
            Bucket="lookout",
            Key="test_upload.parquet",
            Body=f,
            ContentLength=file_size,
        )

This fails with the error even when ContentLength is explicitly passed and all other parameters are correct. We suspect something has changed recently in the gateway implementation or its HTTP handling.

Any guidance from the Storj team or community would be greatly appreciated.

Hello @stonematt ,

Thank you for alerting us.
I’ve forwarded your post to the team.

If you have not already done so, please open a support ticket and our Support Team will be happy to help you from there

This is related to the broke change by AWS, you need to downgrade a client below the boto3 version to v1.35.99

You may subscribe to AWS CLI uses streaming trailer chunked requests on newer versions · Issue #89 · storj/gateway-st · GitHub

2 Likes

I’m going to have to refactor a bit to use boto3 directly instead of the current s3fs storage options embedded in pandas. I’ll work on it over the next few days. (I got into a bit of a version downgrade hell…)

I think you may downgrade s3fs too, at least until we would implement this new not neccessarily needed functionality.
Or use rclone mount or Object Mount instead, they are faster anyway.

this code runs in a streamlit.io app. I use Mountain Duck for local/interactive stuff. I don’t mind using native boto3 to for more direct control. I just have to schedule it. going to be in the next sprint (starting next week)

You may use any S3 SDK, just make sure that it’s not updated to the latest version after 1.35.99 (or analogue, see Announcement: S3 default integrity change · Issue #4392 · boto/boto3 · GitHub), where they introduced these breaking changes.

1 Like