Rclone throws lots of errors when trying to upload

Developer Discussions troubleshooting
1

I followed Rclone with Hosted Gateway - Storj DCS (S3 compatible) to use rclone for synchronizing my files to a bucket in Storj. But when I run the command rclone sync --progress /myfolder/ remotename:/bucketname/remotefolder/ it seems to successfully upload some files (I can see them in the bucket and their content seem ok) but it’s throwing lots of errors (Internal Server Error) and it takes forever to upload just a few files (uploading 11 files with a total of 26Kb took more than 30 mins).

Using the latest version of rclone (1.55.1).

This is the verbose output:

2021/05/30 13:09:54 DEBUG : Creating backend with remote "rclonename:bucketname/www/somedomain.com/html/views/"
2021/05/30 13:09:55 DEBUG : pacer: low level retry 1/10 (error InternalServerError: Internal Server Error
        status code: 500, request id: 1683D3A49F39EBA6, host id: )
2021/05/30 13:09:55 DEBUG : pacer: Rate limited, increasing sleep to 10ms
2021/05/30 13:09:57 DEBUG : pacer: low level retry 2/10 (error InternalServerError: Internal Server Error
        status code: 500, request id: 1683D3A4CA86DA39, host id: )
2021/05/30 13:09:57 DEBUG : pacer: Rate limited, increasing sleep to 20ms
2021/05/30 13:09:57 DEBUG : pacer: low level retry 3/10 (error InternalServerError: Internal Server Error
        status code: 500, request id: 1683D3A5502C9378, host id: )
2021/05/30 13:09:57 DEBUG : pacer: Rate limited, increasing sleep to 40ms
2021/05/30 13:10:00 DEBUG : pacer: low level retry 4/10 (error InternalServerError: Internal Server Error
        status code: 500, request id: 1683D3A553149904, host id: )
2021/05/30 13:10:00 DEBUG : pacer: Rate limited, increasing sleep to 80ms
2021/05/30 13:10:00 DEBUG : pacer: low level retry 5/10 (error InternalServerError: Internal Server Error
        status code: 500, request id: 1683D3A606AA4ED6, host id: )
2021/05/30 13:10:00 DEBUG : pacer: Rate limited, increasing sleep to 160ms
2021/05/30 13:10:00 DEBUG : pacer: low level retry 6/10 (error InternalServerError: Internal Server Error
        status code: 500, request id: 1683D3A60CF4D7A9, host id: )
2021/05/30 13:10:00 DEBUG : pacer: Rate limited, increasing sleep to 320ms
2021/05/30 13:10:01 DEBUG : pacer: low level retry 7/10 (error InternalServerError: Internal Server Error
        status code: 500, request id: 1683D3A616D84E68, host id: )
2021/05/30 13:10:01 DEBUG : pacer: Rate limited, increasing sleep to 640ms
2021/05/30 13:10:01 DEBUG : pacer: low level retry 8/10 (error InternalServerError: Internal Server Error
        status code: 500, request id: 1683D3A629F40E3A, host id: )
2021/05/30 13:10:01 DEBUG : pacer: Rate limited, increasing sleep to 1.28s
2021/05/30 13:10:02 DEBUG : pacer: low level retry 9/10 (error InternalServerError: Internal Server Error
        status code: 500, request id: 1683D3A6502FD174, host id: )
2021/05/30 13:10:02 DEBUG : pacer: Rate limited, increasing sleep to 2s
2021/05/30 13:10:03 DEBUG : pacer: low level retry 10/10 (error InternalServerError: Internal Server Error
        status code: 500, request id: 1683D3A69D0CD0A6, host id: )
2021/05/30 13:10:03 DEBUG : fs cache: renaming cache item "rclonename:bucketname/www/somedomain.com/html/views/" to be canonical "rclonename:bucketname/www/somedomain.com/html/views"
2021-05-30 13:10:09 DEBUG : pacer: Reducing sleep to 1.5s
2021-05-30 13:10:09 DEBUG : S3 bucket bucketname path www/somedomain.com/html/views: Waiting for checks to finish
2021-05-30 13:10:09 DEBUG : S3 bucket bucketname path www/somedomain.com/html/views: Waiting for transfers to finish
2021-05-30 13:10:11 DEBUG : pacer: Reducing sleep to 1.125s
2021-05-30 13:10:12 DEBUG : pacer: Reducing sleep to 843.75ms
2021-05-30 13:10:12 DEBUG : pacer: Reducing sleep to 632.8125ms
2021-05-30 13:10:14 DEBUG : pacer: low level retry 1/10 (error InternalServerError: Internal Server Error
        status code: 500, request id: 1683D3A91AF79182, host id: )
2021-05-30 13:10:14 DEBUG : pacer: Rate limited, increasing sleep to 1.265625s
2021-05-30 13:10:14 DEBUG : pacer: low level retry 1/10 (error InternalServerError: Internal Server Error
        status code: 500, request id: 1683D3A940EF4D59, host id: )
2021-05-30 13:10:14 DEBUG : pacer: Rate limited, increasing sleep to 2s
2021-05-30 13:10:14 DEBUG : pacer: Reducing sleep to 1.5s
Transferred:       22.449k / 26.533 kBytes, 85%, 3.970 kBytes/s, ETA 1s
Transferred:            0 / 7, 0%
Elapsed time:        20.5s
Transferring:
 *                         topic/holy-spirit.php:100% /17.687k, 3.532k/s, 0s
 *                              layouts/main.php:100% /2.096k, 429/s, 0s
 *                                site/about.php:100% /358, 70/s, 0s
 *                              site/contact.php:100% /2.317k, 475/s, 0s
2

Hello @StoryCoder ,
Welcome to the forum!

What is gateway you have used?
Did you try a native rclone integration instead?

3

Gateway? You mean the endpoint https://gateway.eu1.storjshare.io?

I guess I could try the other method… Have tried now - that one does work.

4

Yes, you also could use a self-hosted gateway, so I wanted to be sure what gateway you have used.
Thank you for information, I would notify the team to look into it.

If you still want to use an s3 protocol, you also could try to use a different gateway endpoints: https://docs.storj.io/api-reference/s3-compatible-gateway#regions-and-points-of-presence

5

The possible reason - the default chunk size of 5M in rclone S3 integration
It’s better to configure 64M instead (via Advanced menu or text editor, or specify --s3-chunk-size 64M option in rclone sync command)

1 Like