After a few files have been uploaded, the upload stops

Edit: Today I tried the hosted gateway pattern and uploaded those ~8 GB files within ~20 minutes without any errors at all. Didn’t know why I thought I had already tried this pattern. Maybe I was confused.

Original post:

Hi,

My upstream bandwidth is about 200 Mbps. I’m using restic 0.11.0 and rclone 1.58.0 to backup ~1000 files (~8 GB) via ap1.storj.io (I tried both the native and hosted gateway pattern). However, after some files have been uploaded, a lot of errors (successful puts less than success/repair threshold) appear, and the backup will eventually fail.

Since restic can resume the backup job, I created a cron job to backup every hour. However, only ~400 files (~3.2 GB) had been uploaded after a night.

I thought this had something to do with rclone, so I tried FileZilla instead. Unfortunately, FileZilla also throws errors after some successful uploads.

Any ideas on how to troubleshoot this issue?

Thank you.

Do you have any outbound firewall rules? Like limiting what services, IPs or ports you can connect from your PC?
If so - you need to remove them. The native integration uses p2p mechanism to spread pieces across the globe. The end storage nodes could have literally any IP and port, so you must not limit your outgoing connections.

Take into account that the native integration creates 110 connections for each segment (less or equal 64MB), the restic and rclone uses several threads to upload for each big file and several parallel file uploads, so you can multiple number of connections to the number of threads and to the number of parallel uploads. Your router will have a hard time to manage so much connections and if it’s not capably - most of the “excess” connections will fail.
This is especially true if you use a WiFi connection - it’s not capable for many heavy connections (highly depends on your router though).
The same is true for FileZilla - it uses several parallel uploads by default.
You can manage that by limiting number of parallel uploads and threads to the acceptable for your router level and/or use a wired connection where is possible.
The s3-compatible integration (via GatewayMT) uses one connection per upload, it can use more connections for multipart uploads though, however it’s not hundreds of connections like with p2p.
See also Hotrodding Decentralized Storage

1 Like

My router (Huawei HG8245W5 with ISP firmware​) doesn’t have the firewall option. The option that resembles a firewall the most seems to be IPv4/6 Address Filtering, to which I only added a few IPv6 rules to it (didn’t enable IP filter for IPv4 though).

I assumed that my ISP hid this option in their firmware. Maybe to prevent BitTorrent from dominating their network?

I used a wired connection when using restic+rclone and a WiFi connection when using FileZilla.

These filters for incoming (Downstream) connections not outcoming (Upstream). So I can assume that your router doesn’t have outbound filtering rules.
And I can assume that you doesn’t have a local firewall on the device where you run restic or FileZilla, or their firewalls doesn’t have outbound rules too.

I would recommend to experiment with number of parallel uploads and number of threads to find the allowed limit or use a GatewayMT.

1 Like

I just read an article about throttling and tried connecting to a VPN before transferring the files via FileZilla.

Results: No more errors! :tada:

2 Likes

Seems it’s your ISP. Weird. Even in Russia they do not interfere with traffic… yet…

1 Like

By the way, please take a look:

1 Like