I am testing out Storj, following the advice in Hotrodding Decentralized Storage. I have a 10G test file and have configured rclone with a native config and a Gateway MT config on a GCP instance with 12 vCPUs and 16GB of RAM. When I upload the file to a bucket using the native config (rclone copy --progress test10G native:testing) I get ~16MiB/s. Using the s3 config and following the ‘Hotrodding’ post (rclone copy --progress --s3-upload-concurrency 40 --s3-chunk-size 256M test10G s3:testing) I get ~12KiB/s! This improves when lowering the s3-chunk-size flag (~250KiB/s with 128M), which makes me think something is wrong with my configuration. I followed the rclone configuration here. Any ideas on how I could improve this?
Hello @thomas-rd,
Welcome to the forum!
Perhaps you have a capping on your VM instance for the bandwidth.
You may try to use a lower value for a concurrency with the same chunk size.
Thank you for the welcome and suggestion @Alexey! I tried as you suggested and it didn’t seem to change anything. I double checked on my bandwidth limits and they are much higher than what I’m seeing from the transfer (multiple Gb/s per core). I tried on my physical machine (M2 Macbook with 16GB RAM) and got the expected result for my machine and internet connection, so it definitely seems to be related to the VM. I tried with a different disk attached to the VM, still no luck. Not sure what to try next 
For the concurrency 40 and chunk size of 256MiB, you need to have 256MiB * 40 = 10GiB of memory only for this attempt. By the way, did it finish upload and how long it took in reality?
It may start slow, while chunking the file, but should upload fast, if you do not have limits on concurrent connections from the hypervisor or the cloud provider.
Since your VM has 16GiB of RAM, it has enough memory for this task, and since reducing the chunk size gives an improvement, but reducing concurrency is not, I would suggest to try without a concurrency at all, does it improve something?
Are you hitting the vm’s IOPS limits? They can be quite low with smaller disks.