S3 gateway list with delimiter throw exception

Developer Discussions troubleshooting
1

Hi everyone!
Maybe someone is faced with that kind of an issue.
I am trying to download several files from Storj with different prefixes. With aws CLI everything works fine with that command:

aws --endpoint-url=https://gateway.eu1.storjshare.io s3 cp s3://<bucket> . --recursive --exclude "*" --include "<one>" --include "<two>" --profile storj

But when I am trying to do that with .net code:

new ListObjectsRequest { BucketName = resp.Buckets.First().BucketName, Delimiter = "*", Prefix = "<one>*<two>" }

I got an error:

A header you provided implies functionality that is not implemented

Thanks and have a nice day!

2

Hello @11011 ,
Welcome to the forum!

I’m not sure that < and > are valid symbols. Even so, I think you need to escape them.

aws s3 --profile eu1 --endpoint https://gateway.eu1.storjshare.io cp s3://web-test . --recursive --exclude "*" --include "*.png"
download: s3://web-test/src/my-image.png to src\my-image.png
download: s3://web-test/tardigrade.png to .\tardigrade.png
download: s3://web-test/src/StorjDCS.png to src\StorjDCS.png

aws --version
aws-cli/2.4.1 Python/3.8.8 Windows/10 exe/AMD64 prompt/off
3

The S3 gateway currently supports only the / delimiter in object listing. I see you have already found the ticket to expand this support to an arbitrary delimiter: Support arbitrary delimiters · Issue #43 · storj/gateway-st · GitHub

There is currently no ETA for this improvement. In the meantime, I would recommend implementing your .NET app in a way that does not require this functionality. Perhaps, you could do the listing without the delimiter, and then filter it client-side? Not optimal, but it would work until we deliver that improvement.

2 Likes
4

My intent here is to be able to download multiple files.
For example, I want to get

2021/12/1/file.bin

AND

2021/11/29/file.bin

Is this a wrong API at all? Like it is limitation on a S3 side and I cannot do that in a one call

5

Oh, you mean you want to use * as a wildcard, not as a delimiter? I do not think the S3 API support wildcards for listing objects, even on Amazon S3.

6

As @kaloyan pointed out, filtering using wildcards is something that AWS CLI does by listing the entire bucket and matching keys to the pattern locally. It’s not something that AWS S3, Gateway-MT, or Gateway-ST have/will have.

Other than that, arbitrary prefix and delimiter support (or making the ListObjects and ListObjectsV2 actions fully S3-compatible) should be in the next Gateway-ST release (13th of December). For Gateway-MT, it should be there after the production deployment.

Here’s the changeset associated with this issue:

https://review.dev.storj.io/c/storj/gateway/+/6434
https://review.dev.storj.io/c/storj/gateway/+/6448
https://review.dev.storj.io/c/storj/gateway/+/6449
https://review.dev.storj.io/c/storj/gateway/+/6450
https://review.dev.storj.io/c/storj/gateway/+/6452

If anyone has questions about these, let me know!

1 Like