Edit: Today I tried the hosted gateway pattern and uploaded those ~8 GB files within ~20 minutes without any errors at all. Didn’t know why I thought I had already tried this pattern. Maybe I was confused.
Original post:
Hi,
My upstream bandwidth is about 200 Mbps. I’m using restic 0.11.0 and rclone 1.58.0 to backup ~1000 files (~8 GB) via ap1.storj.io (I tried both the native and hosted gateway pattern). However, after some files have been uploaded, a lot of errors (successful puts less than success/repair threshold) appear, and the backup will eventually fail.
I thought this had something to do with rclone, so I tried FileZilla instead. Unfortunately, FileZilla also throws errors after some successful uploads.
Do you have any outbound firewall rules? Like limiting what services, IPs or ports you can connect from your PC?
If so - you need to remove them. The native integration uses p2p mechanism to spread pieces across the globe. The end storage nodes could have literally any IP and port, so you must not limit your outgoing connections.
Take into account that the native integration creates 110 connections for each segment (less or equal 64MB), the restic and rclone uses several threads to upload for each big file and several parallel file uploads, so you can multiple number of connections to the number of threads and to the number of parallel uploads. Your router will have a hard time to manage so much connections and if it’s not capably - most of the “excess” connections will fail.
This is especially true if you use a WiFi connection - it’s not capable for many heavy connections (highly depends on your router though).
The same is true for FileZilla - it uses several parallel uploads by default.
You can manage that by limiting number of parallel uploads and threads to the acceptable for your router level and/or use a wired connection where is possible.
The s3-compatible integration (via GatewayMT) uses one connection per upload, it can use more connections for multipart uploads though, however it’s not hundreds of connections like with p2p.
See also Hotrodding Decentralized Storage
My router (Huawei HG8245W5 with ISP firmware) doesn’t have the firewall option. The option that resembles a firewall the most seems to be IPv4/6 Address Filtering, to which I only added a few IPv6 rules to it (didn’t enable IP filter for IPv4 though).
These filters for incoming (Downstream) connections not outcoming (Upstream). So I can assume that your router doesn’t have outbound filtering rules.
And I can assume that you doesn’t have a local firewall on the device where you run restic or FileZilla, or their firewalls doesn’t have outbound rules too.
I would recommend to experiment with number of parallel uploads and number of threads to find the allowed limit or use a GatewayMT.