I’m new to Storj and wondering where this issue sits currently. I have experienced a large number of orphaned tcp connections, timeouts, and syslog errors:
kernel: [300218.521175] request_module: kmod_concurrent_max (0) close to 0 (max_modprobes: 50), for module tcp_ledbat, throttling...
I was using --transfers 4 at the time and --bwlimit 1M:off. During the rclone sync task my router reported 10-15% packet loss and service for all other hosts on the network was degraded.
** As I continue to play with the rclone settings to make this less of a problem, I’ve found that --transfers 1 and changing my previous use of --delete-during to --delete-after seems to have helped and I have far fewer errors and packet loss in my current rclone sync.
My ISP connection is 300/10. Not a lot of upload bandwidth to play with.
That’s an excellent guide for tuning. For my use case, where most files are < 64MB the guide would suggest that increasing parallel file transfers instead of segment concurrency is important. I will give that a try by limiting s3-upload-concurrency to 1 and increasing transfers back to the default of 4.
Another noob question but I set up my rclone config using Tardigrade as the provider. I’m still not clear on what GatewayMT is and how rclone interacts with it.
The GatewayMT is an S3-compatible gateway to Storj network. Since you have a low upstream, the native integration would not work well - it opens a lot of connections to different nodes across the globe and with low upstream bandwidth the speed of each connection will be too low, and nodes will start to drop connections because of timeouts.
So, in such circumstances it’s better to use an s3-compatible integration, because it uses only one connection per part, not 110 per segment.
You need to configure rclone to use our GatewayMT.
Excellent, thank you. It seems to be working much better now. I will keep the native integration config for my next location as that will have a symmetrical fiber connection.