I use the Storj managed S3 gateway for downloading our files. They’re often a few hundred MBs zips. This week my speeds are now hitting 8-10MB/s. Prior to this it was many times over. I have a fiber that is pretty consistent on Speedtests.
I’d like to avoid tampering my settings since they were already finetuned before. My download speeds are still quickly slow. Ranging from 9MB to 20MB, with a median of around 10MB.
They network is becoming very slow. As a paying customer, this is concerning to me.
I am downloading through my s3 gateway credentials (hosted by Gateway MT). My client is rclone, but even the direct AWS s3 api has a similar experience.
What gateway endpoint are you using?
What’s chunk size and parallelism (--s3-chunk-size and --s3-upload-concurrency)?
How many parallel transfers (--transfers)?
What is your upstream bandwidth?
I am only using the upload cutoff I found in the documentation before. From looking at the logs, it’s running 4 concurrent connections and each one is hitting about 2.6MB/s.
Are there other parameters I should be considering?
--s3-chunk-size and --s3-upload-concurrency should help greatly.
The --s3-chunk-size option should not be less than 60MB for best performance (we uses 64MB segment size, 60MB size of the chunk + encryption + erasure codes should get roughly 64MB as a result).
The --s3-upload-concurrency option should allow to upload/download chunks of the one file in parallel, this will allow to use your bandwidth at full.
The --transfers option will upload/download files in parallel, utilize your bandwidth even more.
With those changes, I’m now at 5MB/s. I’m pretty sure this is related to the s3 gateway and not my client settings. Like I said before, this has worked before at the max speed of my ISP. I can get the same slow-ish speeds on the cloud too.
Could you please provide an information, what Gateway endpoint you have used?
Is it gateway.storjshare.io or regional-specific like gateway.us1.storjshare.io or gateway.eu1.storjshare.io or gateway.ap1.storjshare.io?
To reiterate @Alexey’s ask - it’s very important for us to know what gateway endpoint you’re using. We can’t see any degradation of service on our end, but this would greatly help in nailing down your problem.
Also, if you @awcchungster can include an output of mtr to the gateway (run mtr gateway.storjshare.io), it would be easier to say if the problem isn’t some internet links along the way.