PutObject fails with “Missing Content-Length” — previously worked for years
We’re consistently getting the following error when uploading to Storj using the S3-compatible gateway. This code previously worked for months without modification.
Error:
An error occurred (MissingContentLength) when calling the PutObject operation: You must provide the Content-Length HTTP header.
Previously Working (s3fs + pandas)
import pandas as pd
import s3fs
storage_options = {
"key": "YOUR_ACCESS_KEY",
"secret": "YOUR_SECRET_KEY",
"client_kwargs": {"endpoint_url": "https://gateway.storjshare.io"},
}
df = pd.DataFrame({"sensor": ["temp"], "value": [72.5]})
df.to_parquet("s3://lookout/test_upload.parquet", storage_options=storage_options)
This approach worked reliably until recently. We’re now consistently seeing errors that indicate Content-Length is missing.
Current Attempt (boto3, explicitly setting ContentLength)
import boto3
import pandas as pd
import os
import tempfile
df = pd.DataFrame({"a": [1, 2], "b": [3, 4]})
with tempfile.NamedTemporaryFile(suffix=".parquet", delete=False) as tmp_file:
df.to_parquet(tmp_file.name)
file_size = os.path.getsize(tmp_file.name)
with open(tmp_file.name, "rb") as f:
s3 = boto3.client(
"s3",
endpoint_url="https://gateway.storjshare.io",
aws_access_key_id="YOUR_ACCESS_KEY",
aws_secret_access_key="YOUR_SECRET_KEY",
)
s3.put_object(
Bucket="lookout",
Key="test_upload.parquet",
Body=f,
ContentLength=file_size,
)
This fails with the error even when ContentLength is explicitly passed and all other parameters are correct. We suspect something has changed recently in the gateway implementation or its HTTP handling.
Any guidance from the Storj team or community would be greatly appreciated.