Uplink: failed to upload enough pieces (needed at least 80 but got 78)

andy01 · December 16, 2023, 7:48pm

The backup failed (uplink -p4) last Sunday 9th of December:

root-node-1  |  214GiB 0:55:08 [89.9MiB/s] [66.5MiB/s] [>                    ]  5% ETA 15:40:42
root-node-1  | failed to upload part 213: uplink: encryption: metaclient: manager closed: closed: read tcp 172.19.0.3:42882->34.172.100.72:7777: read: connection reset by peer
root-node-1  | error getting reader for part 215: context canceled
root-node-1  | failed to upload part 212: uplink: failed to upload enough pieces (needed at least 80 but got 72)
root-node-1  | failed to upload part 214: uplink: context canceled
root-node-1  |  215GiB 0:55:11 [66.5MiB/s] [66.5MiB/s] [>                    ]  5%             
root-node-1  |

And today (16 December 2023):

root-node-1  | 24.6GiB 0:07:15 [62.5MiB/s] [57.9MiB/s] [>                    ]  0% ETA 19:07:57
root-node-1  | 25.0GiB 0:07:20 [76.3MiB/s] [58.1MiB/s] [>                    ]  0% ETA 19:03:45
root-node-1  | failed to upload part 15: uplink: metaclient: manager closed: closed: read tcp 172.19.0.3:55320->34.172.100.72:7777: read: con
nection reset by peer
root-node-1  | error getting reader for part 25: context canceled
root-node-1  | failed to upload part 23: uplink: encryption: context canceled
root-node-1  | failed to upload part 24: uplink: context canceled
root-node-1  | 25.0GiB 0:07:21 [58.0MiB/s] [58.0MiB/s] [>                    ]  0%

So I’ve restarted it again with uplink -p 2 now.

Good point!
Though we are not using the firewall on our MikroTik router.

[admin@MikroTik] > /ip/firewall/connection/print 
<nothing>

[admin@MikroTik] > /ip/firewall/filter/print 
Flags: X - disabled, I - invalid; D - dynamic 
[admin@MikroTik] > /ip/firewall/nat/print 
Flags: X - disabled, I - invalid; D - dynamic 
[admin@MikroTik] > /ip/firewall/mangle/print 
Flags: X - disabled, I - invalid; D - dynamic 
[admin@MikroTik] > /ip/firewall/raw/print    
Flags: X - disabled, I - invalid; D - dynamic 
[admin@MikroTik] > 
[admin@MikroTik] > /ip/firewall/connection/tracking/print 
                   enabled: auto
               active-ipv4: no
               active-ipv6: no
      tcp-syn-sent-timeout: 5s
  tcp-syn-received-timeout: 5s
   tcp-established-timeout: 1d
      tcp-fin-wait-timeout: 10s
    tcp-close-wait-timeout: 10s
      tcp-last-ack-timeout: 10s
     tcp-time-wait-timeout: 10s
         tcp-close-timeout: 10s
   tcp-max-retrans-timeout: 5m
       tcp-unacked-timeout: 5m
        loose-tcp-tracking: yes
               udp-timeout: 10s
        udp-stream-timeout: 3m
              icmp-timeout: 10s
           generic-timeout: 10m
               max-entries: 28672
             total-entries: 0

AFAIK, the connection tracking feature primarily comes into play when the router needs to keep track of the state of network connections, which is typically necessary for certain types of firewall rules.

We don’t use Firewall so the connection tracking limitation can be ignored I guess, nor we have QoS. We have pretty much default and very simple MikroTik router config.

So it might be anything else I guess (e.g. FW/SW/HW, cables, or ISP/upstream issues).

The temperature is less than 80°C which should not be a concern for MikroTik CRS326-24S+2Q+ we have. And the network load is 0% despite it being making the backup right now with Storj’s uplink at this very moment.

arrogantrabbit · December 16, 2023, 8:18pm

What about nat table size? Or max connections on the client? Else it would be likely indeed upstream (isp modems/bridhes/etc)

andy01 · December 18, 2023, 10:20pm

That’d be the max-entries & total-entries as shown above (/ip/firewall/connection/tracking/print & /ip/firewall/nat/print)
We don’t use the firewall/NAT.

I guess that’d then be something else.

Alexey · December 19, 2023, 2:57am

Can you temporary use a different router?
I have several reports about Mikrotik routers which drops connections for some reasons. This was for SNOs, who invoked an old version of the Graceful Exit, when the node attempted to transfer several TB of data to other nodes and GE always failed because of too much failed transfers (thousands). As soon as they connected internet without Mikrotik - GE was finished with zero failed transfers.

andy01 · January 12, 2024, 11:04pm

Great point!

As we don’t have alternative router there, I went ahead and upgraded our MikroTik router (CRS326-24S+2Q+) firmware from 7.10.2 to 7.12.1, then to 7.13.1.
Now I’ve ran the uplink with -p 8 -t 8 - lo and behold, 2.78TiB have been successfully backed up at the Storj!

Alexey · January 13, 2024, 1:46am

I’m glad to read, that the upgrade of the firmware solved an issue!
I hope it’s permanent and would not require restart of the router every time (as many other cheap SOHO routers) to make it working normally with multiple parallel connections.

Pentium100 · January 13, 2024, 2:40am

Would it be possible to make uplink retry the failed uploads? I mean if 79 pieces were successfully uploaded, get one more node from the satellite or retry the failed ones a few times. It would suck if uploading a large file failed because at some point some connections failed.

Alexey · January 13, 2024, 4:05am

We always working on improvements of our tools to speed up uploads and downloads and to processing of nodes’ failures during upload/download.

But if you want to have this feature now, you may use rclone instead, it’s already has this feature.

Pentium100 · January 13, 2024, 4:22am

Cool, I just noticed that threads about incomplete uploads popped up once in a while and though it would be nice if uplink could just retry the failed piece uploads instead of forcing the user to retry uploading the whole file.

Alexey · January 13, 2024, 9:22am

It’s not easy. Otherwise it would be implemented. We do not want to implement it like it’s implemented in rclone - retry the whole file, because for that you may use rclone.

andy01 · January 23, 2024, 2:44pm

Sadly the errors are back

root-node-1  |  507GiB 2:02:02 [7.36MiB/s] [70.9MiB/s] [=>                   ] 12% ETA 14:24:43
root-node-1  |  507GiB 2:02:07 [ 147MiB/s] [71.0MiB/s] [=>                   ] 12% ETA 14:23:54
root-node-1  | failed to upload part 505: uplink: encryption: metaclient: manager closed: closed: read tcp 172.19.0.3:56072->34.172.100.72:7777: read: connection reset by peer
root-node-1  | error getting reader for part 508: context canceled
root-node-1  | failed to upload part 504: uplink: failed to upload enough pieces (needed at least 80 but got 78)
root-node-1  | failed to upload part 506: uplink: encryption: context canceled
root-node-1  | failed to upload part 507: uplink: context canceled
root-node-1  |  508GiB 2:02:10 [71.0MiB/s] [71.0MiB/s] [=>                   ] 12%

See the sent traffic on the screenshot, logged on the proxmox (blade1) host, during which we’ve encountered the RST (connection reset by peer).

I’ll ask the ISP to see if they can find any issue on their end.
Otherwise we’ll likely have to either restart the router every once before the backup (I’d call this as not acceptable workaround though) or get another router perhaps

Alexey · January 24, 2024, 6:55am

Unfortunately your ISP likely cannot solve the issue. If the router started to drop connections again - it’s a hardware/firmware issue of the router.
The temporary solution would be to restart it every time when it starting to happen again… I’m sorry…

In this case you may also switch to S3 integration, if possible, it uses much less parallel connections, but it will use a Server-Side Encryption.

andy01 · January 25, 2024, 9:57am

Alternatively, consider scheduling the backup for a weekday. I have observed that the backup process consistently succeeds on weekdays (Monday to Friday), whereas it tends to fail during the weekend.
Additionally, restarting the router is not necessary since the backup successfully completes on the weekdays.

Alexey · January 26, 2024, 7:48am

This is very weird. Perhaps the router can flush its state after some time?
It’s very unusual, that you didn’t get enough nodes on a specific days.

Do you maybe know, which days are bad?

andy01 · February 18, 2024, 10:57pm

Thought I’d share the update: We haven’t had any network related errors (during Storj upload) since our ISP had an emergency hardware maintenance 9 days ago (09 Feb 2024), even the weekend backup worked fine (2.91 TiB uploaded with avg speed 78.9 MiB/s).

andy01 · April 20, 2024, 6:29pm

It appears that the issue has returned, interestingly after there was a short ISP outage (~5-10 mins) on April 2nd, 2024, around 4:45 PM UTC.
The same uplink version - v1.90.2 and -p 4 -t 4 --progress=false args.

April 16th:

root-node-1  | failed to upload part 1350: uplink: encryption: metaclient: manager closed: closed: read tcp 172.19.0.3:43990->34.150.199.48:7777: read: connection reset by peer
root-node-1  | error getting reader for part 1353: context canceled
root-node-1  | failed to upload part 1351: uplink: encryption: context canceled
root-node-1  | failed to upload part 1352: uplink: context canceled
root-node-1  | 1.32TiB 5:04:20 [ 128MiB/s] [75.9MiB/s] [=====>               ] 29% ETA 11:58:34

April 20th:

root-node-1  | 1.23TiB 4:44:56 [88.4MiB/s] [75.3MiB/s] [====>                ] 27% ETA 12:34:11
root-node-1  | failed to upload part 1256: uplink: encryption: metaclient: manager closed: closed: read tcp 172.19.0.3:45308->34.150.199.48:7777: read: connection reset by peer
root-node-1  | failed to upload part 1255: uplink: metaclient: context canceled
root-node-1  | error getting reader for part 1258: context canceled
root-node-1  | failed to upload part 1257: uplink: context canceled
root-node-1  | 1.23TiB 4:45:00 [75.3MiB/s] [75.3MiB/s] [====>                ] 27%

The ISP DevOps Network Engineer replied to this:

The connection reset was sent from Google. This would be nearly impossible for us to provide details on this event.

I guess we are out of luck here.

Will try switching -p 4 -t 4 --progress=false back to -p 2 --progress=false.

Alexey · April 21, 2024, 11:38am

It could be related to a routing.
Could you please show the MTR results for the gateway.storjshare.io (if you still uses the Gateway-MT instead of a native, where you will not have such issues at all)?
You may use a DM, if you are not comfortable to expose your IPs publicly.

andy01 · April 23, 2024, 9:42am

Hey Alexey
There you go: mtr gateway.storjshare.io

                                        My traceroute  [v0.95]
akash-rpc-archival (184.105.162.171) -> gateway.storjshare.io (136.0.77.2)    2024-04-23T09:40:00+0000
Keys:  Help   Display mode   Restart statistics   Order of fields   quit
                                                              Packets               Pings
 Host                                                       Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. ve1208.core1.fmt2.he.net                                74.7%    76    0.6   3.8   0.4  27.6   7.0
 2. port-channel2.core6.fmt2.he.net                         23.7%    76    0.5   4.1   0.4  35.5   8.0
 3. port-channel12.core2.pao1.he.net                        86.7%    76    1.2   1.7   1.1   4.3   1.0
 4. 64.62.244.149                                            0.0%    76    0.9   1.2   0.9   3.8   0.5
 5. (waiting for reply)
 6. datacamp-218.losangeles2.loa.seabone.net                 0.0%    76    8.4  10.2   8.3  25.8   3.5
 7. vl224.lax-cs2-dist-2.cdn77.com                           0.0%    75    7.2   8.6   7.2  24.5   3.2
 8. 136.0.77.2                                               0.0%    75    7.7   8.9   7.6  25.1   3.0

We are using HE.net provided gateway directly, but through the MikroTik router as we always have. We have never directly attached the server to the HE.net gateway port (around the MikroTik router), and it’s been working at times flawlessly, without the need for MikroTik router reboot.

Update: Hmm I can see some packet loss there, even against the Google (mtr google.com):

                                                My traceroute  [v0.95]
akash-rpc-archival (184.105.162.171) -> google.com (142.250.191.78)                           2024-04-23T09:43:53+0000
Keys:  Help   Display mode   Restart statistics   Order of fields   quit
                                                                              Packets               Pings
 Host                                                                       Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. ve1208.core1.fmt2.he.net                                                66.7%    51   19.4   2.7   0.4  19.4   5.5
 2. 100ge1-2.core2.fmt2.he.net                                               0.0%    51    7.3   2.8   0.3  21.7   5.6
 3. as15169.sfmix.org                                                        0.0%    50    0.9   0.8   0.7   1.6   0.1
 4. 142.251.254.241                                                          0.0%    50    1.3   1.2   1.1   1.5   0.1
 5. 142.251.224.31                                                           0.0%    50    1.3   1.2   1.2   1.4   0.0
 6. nuq04s43-in-f14.1e100.net                                                0.0%    50    1.2   1.1   1.1   1.5   0.1

Checking with the HE.net.

andy01 · April 23, 2024, 9:57am

HE.net ISP replied:

Both of these traces are showing no loss to the destination. Observed packet loss on our hops (and even Google’s servers) are subject to control plane rate limits. As this doesn’t continue to the destination, it’s only cosmetic and not indicative of an issue.
https://www.cloudflare.com/learning/network-layer/what-is-mtr/

Alexey · April 26, 2024, 7:54am

I have passed your information to the team.

But I know, that HE is often has broken routing unfortunately (they often (very) didn’t agree with other providers).