Upload takes eternity

Product Discussions
1

Hi all. Please find enclosed a part log from a recent upload. This is my first attempt to make use of rclone. Am interested in your observations comments. It was very slow. The link to the internet here is wireless and can vary from 20Mbit to 40Mbit.

2022-10-28 10:56:24 ERROR : FS sj://uploads2022/xxx/Documents: cp input ./Recipe Pizza-2.png [HashesOption()]: uplink: upload done: already committed
storj.io/uplink.errwrapf:99
storj.io/uplink.(*Upload).Abort:161
github.com/rclone/rclone/backend/storj.(*Fs).Put.func1:546
github.com/rclone/rclone/backend/storj.(*Fs).Put:579
github.com/rclone/rclone/fs/operations.Copy:501
github.com/rclone/rclone/fs/sync.(*syncCopyMove).pairCopyOrMove:423
2022-10-28 11:03:29 ERROR : FS sj://uploads2022/xxx/Documents: cp input ./Tools Brush On Elect Tape.png [HashesOption()]: uplink: upload done: already committed
storj.io/uplink.errwrapf:99
storj.io/uplink.(*Upload).Abort:161
github.com/rclone/rclone/backend/storj.(*Fs).Put.func1:546
github.com/rclone/rclone/backend/storj.(*Fs).Put:579
github.com/rclone/rclone/fs/operations.Copy:501
github.com/rclone/rclone/fs/sync.(*syncCopyMove).pairCopyOrMove:423
2022-10-28 11:24:53 ERROR : Speed Test/Speedtest.ods: corrupted on transfer: sizes differ 14611 vs 14712
2022-10-28 11:29:09 ERROR : FS sj://uploads2022/xxx/Documents: cp input ./Poems /Henry Text.png [HashesOption()]: uplink: upload done: already committed
storj.io/uplink.errwrapf:99
storj.io/uplink.(*Upload).Abort:161
github.com/rclone/rclone/backend/storj.(*Fs).Put.func1:546
github.com/rclone/rclone/backend/storj.(*Fs).Put:579
github.com/rclone/rclone/fs/operations.Copy:501
github.com/rclone/rclone/fs/sync.(*syncCopyMove).pairCopyOrMove:423
2022-10-28 13:02:21 ERROR : FS sj://uploads2022/xxx/Documents: cp input ./Mens Shed Docs/Purchasing/Spring Fair2021.png [HashesOption()]: uplink: upload done: already committed
storj.io/uplink.errwrapf:99
storj.io/uplink.(*Upload).Abort:161
github.com/rclone/rclone/backend/storj.(*Fs).Put.func1:546
github.com/rclone/rclone/backend/storj.(*Fs).Put:579
github.com/rclone/rclone/fs/operations.Copy:501
github.com/rclone/rclone/fs/sync.(*syncCopyMove).pairCopyOrMove:423
2022-10-28 13:11:50 ERROR : Attempt 1/3 failed with 1 errors and: corrupted on transfer: sizes differ 14611 vs 14712
2022-10-28 13:11:55 ERROR : Attempt 2/3 succeeded
Transferred: 2.127 GiB / 2.127 GiB, 100%, 12.018 KiB/s, ETA 0s
Checks: 780 / 780, 100%
Transferred: 782 / 782, 100%
Elapsed time: 3h28m30.8s

2

With low upstream bandwidth it’s better to configure rclone with hosted S3 Gateway.

3

Thanks - I will look into rclone S3 configuration and try it out.

1 Like
4

Hi All,

Just leaning more into this topic but from the browser perspective. For some reason, with the newer UI refresh, the upload takes a really really long time and it is very glitchy. I uploaded 26 PDFs with those sizes (nothing too crazy):

image
image229×573 3.31 KB

It looks like the system is uploading the files in batches, and freezes between each batch upload; but there is no feedback of when the files are ‘actually’ uploaded. The bar just goes straight to 100%. Also, in between one of those ‘batch uploads’ I saw a pop up error with code but it went away too quick for me to capture it.

Is there any reason it is taking really long to upload? I feel its something to do with the frontend as with CLI it seems pretty quick, and Storj is pretty solid in the backend :slight_smile:

5

The objects browser is not meant to be a primary interface and to upload bunch of files, it’s for a quick start and to try out several files.
The primary interface is a library and tools with integration, like rclone, Cyberduck, Filezilla or sliksafe, etc.

However I’ll pass your feedback to the team.