Managed S3 Gateway Up/down speed?

I use the Storj managed S3 gateway for downloading our files. They’re often a few hundred MBs zips. This week my speeds are now hitting 8-10MB/s. Prior to this it was many times over. I have a fiber that is pretty consistent on Speedtests.

Why is the Storj network slower?

there was a bunch of old data deletions, this overloaded lot of nodes.
but today it shold be mostly resolved.

You can use parallelism and more big chunks to increase your speed, see

1 Like

I’d like to avoid tampering my settings since they were already finetuned before. My download speeds are still quickly slow. Ranging from 9MB to 20MB, with a median of around 10MB.

They network is becoming very slow. As a paying customer, this is concerning to me. :frowning_face:

This has been noticed before on clients with IPv6 connections. Has anything changed on your end as far as you know?

My client is on a standard ipv4.

In that case how are you downloading the file? Have you tried alternate methods?

I am downloading through my s3 gateway credentials (hosted by Gateway MT). My client is rclone, but even the direct AWS s3 api has a similar experience.

What gateway endpoint are you using?
What’s chunk size and parallelism (--s3-chunk-size and --s3-upload-concurrency)?
How many parallel transfers (--transfers)?
What is your upstream bandwidth?

I am only using the upload cutoff I found in the documentation before. From looking at the logs, it’s running 4 concurrent connections and each one is hitting about 2.6MB/s.

Are there other parameters I should be considering?

–s3-upload-cutoff 5G

--s3-chunk-size and --s3-upload-concurrency should help greatly.
The --s3-chunk-size option should not be less than 60MB for best performance (we uses 64MB segment size, 60MB size of the chunk + encryption + erasure codes should get roughly 64MB as a result).
The --s3-upload-concurrency option should allow to upload/download chunks of the one file in parallel, this will allow to use your bandwidth at full.
The --transfers option will upload/download files in parallel, utilize your bandwidth even more.

With those changes, I’m now at 5MB/s. I’m pretty sure this is related to the s3 gateway and not my client settings. Like I said before, this has worked before at the max speed of my ISP. I can get the same slow-ish speeds on the cloud too.

Could you please provide an information, what Gateway endpoint you have used?
Is it gateway.storjshare.io or regional-specific like gateway.us1.storjshare.io or gateway.eu1.storjshare.io or gateway.ap1.storjshare.io?

1 Like

To reiterate @Alexey’s ask - it’s very important for us to know what gateway endpoint you’re using. We can’t see any degradation of service on our end, but this would greatly help in nailing down your problem.

Also, if you @awcchungster can include an output of mtr to the gateway (run mtr gateway.storjshare.io), it would be easier to say if the problem isn’t some internet links along the way.

1 Like