Slow download speed via link.storjshare.io (~1-5MiB/s)

Product Discussions
1

Hi,

I am getting a very slow download speed via link.storjshare.io links.

speedtest

My server is capable of 4221 Mbit/s download and 227.25 Mbit/s upload:

# speedtest
Retrieving speedtest.net configuration...
Testing from Packet Host (147.75.38.11)...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by Netprotect (New York, NY) [5856.89 km]: 1.153 ms
Testing download speed................................................................................
Download: 4221.12 Mbit/s
Testing upload speed......................................................................................................
Upload: 227.25 Mbit/s

mtr link.storjshare.io 

# mtr link.storjshare.io
                                                           My traceroute  [v0.95]
ssh-0 (10.233.70.173) -> link.storjshare.io (185.244.226.4)                                                        2022-12-09T12:49:14+0000
Keys:  Help   Display mode   Restart statistics   Order of fields   quit
                                                                                                   Packets               Pings
 Host                                                                                            Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. sandbox.ny.aksh.pw                                                                            0.0%    21    0.0   0.0   0.0   0.1   0.0
 2. 169.254.57.1                                                                                  0.0%    20    0.2   0.2   0.1   0.2   0.0
 3. 10.253.4.195                                                                                  0.0%    20    0.2   0.2   0.1   0.2   0.0
 4. 10.253.4.153                                                                                  0.0%    20    1.0   2.1   0.1  13.7   3.8
 5. nyk-b4-link.ip.twelve99.net                                                                   0.0%    20    0.4   0.4   0.3   1.0   0.1
 6. nyk-bb1-link.ip.twelve99.net                                                                  0.0%    20    1.3   1.2   1.0   1.6   0.1
 7. rest-bb1-link.ip.twelve99.net                                                                 0.0%    20    6.5   6.6   6.4   6.9   0.1
 8. lax-b23-link.ip.twelve99.net                                                                  0.0%    20   63.2  63.2  63.0  63.5   0.1
 9. port-channel17.core2.lax1.he.net                                                             35.0%    20   60.4  63.6  60.4  92.4   8.7
10. link.storjshare.io                                                                            0.0%    20   60.0  60.0  60.0  60.1   0.0

aira2c - 10 threads

# aria2c -x10 -j10 -s10 "https://link.storjshare.io/s/jvmp4mtjyyx76xn7atifwhhin5kq/akash-snapshots/rpc.nl/akashnet-2_2022-12-08T22:32:30.tar.zst?download=1"

12/09 10:23:13 [NOTICE] Downloading 1 item(s)
[#8a021a 737MiB/1,549GiB(0%) CN:10 DL:69MiB ETA:6h22m2s]                                                                                   
[#8a021a 1.4GiB/1,549GiB(0%) CN:10 DL:89MiB ETA:4h55m58s]                                                                                  
[#8a021a 2.1GiB/1,549GiB(0%) CN:10 DL:92MiB ETA:4h44m41s]                                                                                  
 *** Download Progress Summary as of Fri Dec  9 10:24:13 2022 ***                                                                          
===========================================================================================================================================
[#8a021a 3.8GiB/1,549GiB(0%) CN:10 DL:84MiB ETA:5h12m37s]
FILE: /root/akashnet-2_2022-12-08T22:32:30.tar.zst
-------------------------------------------------------------------------------------------------------------------------------------------

 *** Download Progress Summary as of Fri Dec  9 10:25:14 2022 ***                                                                          
===========================================================================================================================================
[#8a021a 7.6GiB/1,549GiB(0%) CN:10 DL:3.9MiB ETA:110h56m13s]
FILE: /root/akashnet-2_2022-12-08T22:32:30.tar.zst
-------------------------------------------------------------------------------------------------------------------------------------------

[#8a021a 7.6GiB/1,549GiB(0%) CN:10 DL:1.1MiB ETA:387h34m22s]                                                                               
[#8a021a 7.7GiB/1,549GiB(0%) CN:10 DL:3.9MiB ETA:109h47m43s]                                                                               
 *** Download Progress Summary as of Fri Dec  9 10:26:14 2022 ***                                                                          
===========================================================================================================================================
[#8a021a 7.8GiB/1,549GiB(0%) CN:10 DL:8.2MiB ETA:53h2m39s]
FILE: /root/akashnet-2_2022-12-08T22:32:30.tar.zst
-------------------------------------------------------------------------------------------------------------------------------------------

[#8a021a 7.9GiB/1,549GiB(0%) CN:10 DL:3.1MiB ETA:137h10m55s]                                                                               
 *** Download Progress Summary as of Fri Dec  9 10:27:15 2022 ***                                                                          
===========================================================================================================================================
[#8a021a 8.2GiB/1,549GiB(0%) CN:10 DL:3.1MiB ETA:138h27m21s]
FILE: /root/akashnet-2_2022-12-08T22:32:30.tar.zst
-------------------------------------------------------------------------------------------------------------------------------------------

 *** Download Progress Summary as of Fri Dec  9 10:28:15 2022 ***                                                                          
===========================================================================================================================================
[#8a021a 8.4GiB/1,549GiB(0%) CN:10 DL:2.7MiB ETA:157h33m25s]
FILE: /root/akashnet-2_2022-12-08T22:32:30.tar.zst
-------------------------------------------------------------------------------------------------------------------------------------------

 *** Download Progress Summary as of Fri Dec  9 10:29:16 2022 ***                                                                          
===========================================================================================================================================
[#8a021a 8.5GiB/1,549GiB(0%) CN:10 DL:1.5MiB ETA:275h34m35s]
FILE: /root/akashnet-2_2022-12-08T22:32:30.tar.zst
-------------------------------------------------------------------------------------------------------------------------------------------

[#8a021a 8.8GiB/1,549GiB(0%) CN:10 DL:10MiB ETA:41h25m18s]                                                                                 
 *** Download Progress Summary as of Fri Dec  9 10:30:16 2022 ***                                                                          
===========================================================================================================================================
[#8a021a 8.8GiB/1,549GiB(0%) CN:10 DL:2.0MiB ETA:213h25m26s]
FILE: /root/akashnet-2_2022-12-08T22:32:30.tar.zst
-------------------------------------------------------------------------------------------------------------------------------------------

 *** Download Progress Summary as of Fri Dec  9 10:31:17 2022 ***                                                                          
===========================================================================================================================================
[#8a021a 9.0GiB/1,549GiB(0%) CN:10 DL:1.5MiB ETA:281h59m27s]
FILE: /root/akashnet-2_2022-12-08T22:32:30.tar.zst
-------------------------------------------------------------------------------------------------------------------------------------------

 *** Download Progress Summary as of Fri Dec  9 10:32:18 2022 ***                                                                          
===========================================================================================================================================
[#8a021a 9.2GiB/1,549GiB(0%) CN:10 DL:1.9MiB ETA:227h25m30s]
FILE: /root/akashnet-2_2022-12-08T22:32:30.tar.zst
-------------------------------------------------------------------------------------------------------------------------------------------

[#8a021a 9.3GiB/1,549GiB(0%) CN:10 DL:2.5MiB ETA:174h7m51s]                                                                                
 *** Download Progress Summary as of Fri Dec  9 10:33:18 2022 ***                                                                          
===========================================================================================================================================
[#8a021a 9.4GiB/1,549GiB(0%) CN:10 DL:5.3MiB ETA:81h55m53s]
FILE: /root/akashnet-2_2022-12-08T22:32:30.tar.zst
-------------------------------------------------------------------------------------------------------------------------------------------

 *** Download Progress Summary as of Fri Dec  9 10:34:19 2022 ***                                                                          
===========================================================================================================================================
[#8a021a 9.7GiB/1,549GiB(0%) CN:10 DL:3.3MiB ETA:129h1m36s]
FILE: /root/akashnet-2_2022-12-08T22:32:30.tar.zst
-------------------------------------------------------------------------------------------------------------------------------------------

 *** Download Progress Summary as of Fri Dec  9 10:35:19 2022 ***                                                                          
===========================================================================================================================================
[#8a021a 9.9GiB/1,549GiB(0%) CN:10 DL:3.4MiB ETA:126h45m2s]
FILE: /root/akashnet-2_2022-12-08T22:32:30.tar.zst
-------------------------------------------------------------------------------------------------------------------------------------------

 *** Download Progress Summary as of Fri Dec  9 10:36:20 2022 ***                                                                          
===========================================================================================================================================
[#8a021a 10GiB/1,549GiB(0%) CN:10 DL:2.6MiB ETA:163h27m18s]
FILE: /root/akashnet-2_2022-12-08T22:32:30.tar.zst
-------------------------------------------------------------------------------------------------------------------------------------------

 *** Download Progress Summary as of Fri Dec  9 10:37:20 2022 ***                                                                          
===========================================================================================================================================
[#8a021a 10GiB/1,549GiB(0%) CN:10 DL:2.1MiB ETA:203h42m5s]
FILE: /root/akashnet-2_2022-12-08T22:32:30.tar.zst
-------------------------------------------------------------------------------------------------------------------------------------------

[#8a021a 10GiB/1,549GiB(0%) CN:10 DL:2.0MiB ETA:215h57m7s]                                                                                 
 *** Download Progress Summary as of Fri Dec  9 10:38:21 2022 ***                                                                          
===========================================================================================================================================
[#8a021a 10GiB/1,549GiB(0%) CN:10 DL:1.7MiB ETA:249h42m5s]
FILE: /root/akashnet-2_2022-12-08T22:32:30.tar.zst
-------------------------------------------------------------------------------------------------------------------------------------------

 *** Download Progress Summary as of Fri Dec  9 10:39:21 2022 ***                                                                          
===========================================================================================================================================
[#8a021a 10GiB/1,549GiB(0%) CN:10 DL:6.9MiB ETA:63h19m9s]
FILE: /root/akashnet-2_2022-12-08T22:32:30.tar.zst
-------------------------------------------------------------------------------------------------------------------------------------------

[#8a021a 10GiB/1,549GiB(0%) CN:10 DL:4.3MiB ETA:101h20m59s]                                                                                
 *** Download Progress Summary as of Fri Dec  9 10:40:22 2022 ***                                                                          
===========================================================================================================================================
[#8a021a 10GiB/1,549GiB(0%) CN:10 DL:5.6MiB ETA:77h45m47s]
FILE: /root/akashnet-2_2022-12-08T22:32:30.tar.zst
-------------------------------------------------------------------------------------------------------------------------------------------

 *** Download Progress Summary as of Fri Dec  9 10:41:23 2022 ***                                                                          
===========================================================================================================================================
[#8a021a 11GiB/1,549GiB(0%) CN:10 DL:6.3MiB ETA:69h7s]
FILE: /root/akashnet-2_2022-12-08T22:32:30.tar.zst
-------------------------------------------------------------------------------------------------------------------------------------------

 *** Download Progress Summary as of Fri Dec  9 10:42:23 2022 ***                                                                          
===========================================================================================================================================
[#8a021a 11GiB/1,549GiB(0%) CN:10 DL:1.7MiB ETA:251h43m25s]
FILE: /root/akashnet-2_2022-12-08T22:32:30.tar.zst
-------------------------------------------------------------------------------------------------------------------------------------------

[#8a021a 11GiB/1,549GiB(0%) CN:10 DL:5.2MiB ETA:82h55m55s]                                                                                 
 *** Download Progress Summary as of Fri Dec  9 10:43:24 2022 ***                                                                          
===========================================================================================================================================
[#8a021a 11GiB/1,549GiB(0%) CN:10 DL:5.1MiB ETA:85h22m42s]
FILE: /root/akashnet-2_2022-12-08T22:32:30.tar.zst
-------------------------------------------------------------------------------------------------------------------------------------------

[#8a021a 11GiB/1,549GiB(0%) CN:10 DL:5.1MiB ETA:85h22m42s]

uplink v1.68.2

uplink is also giving slow speed ~4-8 MiB/s :/

# uplink cp sj://akash-snapshots/rpc.nl/akashnet-2_2022-12-08T22:32:30.tar.zst .
download sj://akash-snapshots/rpc.nl/akashnet-2_2022-12-08T22:32:30.tar.zst to akashnet-2_2022-12-08T22:32:30.tar.zst
93.16 MiB / 1.51 TiB [>___________________________________________] 0.01% 8.22 MiB p/s
136.59 MiB / 1.51 TiB [>__________________________________________] 0.01% 8.02 MiB p/s
164.40 MiB / 1.51 TiB [>__________________________________________] 0.01% 6.49 MiB p/s
192.00 MiB / 1.51 TiB [>__________________________________________] 0.01% 6.59 MiB p/s
242.87 MiB / 1.51 TiB [>__________________________________________] 0.02% 7.58 MiB p/s
811.63 MiB / 1.51 TiB [>__________________________________________] 0.05% 5.47 MiB p/s

Any ideas?

Update 1

Got a better speed with 20 uplink threads:

# uplink cp -p 20 sj://akash-snapshots/rpc.nl/akashnet-2_2022-12-08T22:32:30.tar.zst .
download sj://akash-snapshots/rpc.nl/akashnet-2_2022-12-08T22:32:30.tar.zst to akashnet-2_2022-12-08T22:32:30.tar.zst
224.77 MiB / 1.51 TiB [>_______________________________________________________________________________________________] 0.01% 21.51 MiB p/s
400.32 MiB / 1.51 TiB [>_______________________________________________________________________________________________] 0.03% 29.96 MiB p/s
706.57 MiB / 1.51 TiB [>_______________________________________________________________________________________________] 0.04% 40.00 MiB p/s
1.98 GiB / 1.51 TiB [>_________________________________________________________________________________________________] 0.13% 49.12 MiB p/s
2.91 GiB / 1.51 TiB [>_________________________________________________________________________________________________] 0.19% 49.97 MiB p/s
3.02 GiB / 1.51 TiB [>_________________________________________________________________________________________________] 0.19% 48.33 MiB p/s
3.40 GiB / 1.51 TiB [>_________________________________________________________________________________________________] 0.22% 44.97 MiB p/s
3.75 GiB / 1.51 TiB [>_________________________________________________________________________________________________] 0.24% 44.09 MiB p/s
4.17 GiB / 1.51 TiB [>_________________________________________________________________________________________________] 0.27% 47.55 MiB p/s
4.36 GiB / 1.51 TiB [>_________________________________________________________________________________________________] 0.28% 45.67 MiB p/s
4.60 GiB / 1.51 TiB [>_________________________________________________________________________________________________] 0.30% 48.37 MiB p/s
4.87 GiB / 1.51 TiB [>_________________________________________________________________________________________________] 0.31% 46.32 MiB p/s
4.89 GiB / 1.51 TiB [>_________________________________________________________________________________________________] 0.32% 44.49 MiB p/s

However increasing it to 40 did not make the difference… would love to see at least 100 MiB/s:

# uplink cp -p 40 sj://akash-snapshots/rpc.nl/akashnet-2_2022-12-08T22:32:30.tar.zst .
download sj://akash-snapshots/rpc.nl/akashnet-2_2022-12-08T22:32:30.tar.zst to akashnet-2_2022-12-08T22:32:30.tar.zst
150.35 MiB / 1.51 TiB [>_______________________________________________________________________________________________] 0.01% 15.82 MiB p/s
447.77 MiB / 1.51 TiB [>_______________________________________________________________________________________________] 0.03% 28.48 MiB p/s
835.27 MiB / 1.51 TiB [>_______________________________________________________________________________________________] 0.05% 37.74 MiB p/s
925.82 MiB / 1.51 TiB [>_______________________________________________________________________________________________] 0.06% 37.95 MiB p/s
1.58 GiB / 1.51 TiB [>_________________________________________________________________________________________________] 0.10% 42.03 MiB p/s
2.13 GiB / 1.51 TiB [>_________________________________________________________________________________________________] 0.14% 41.37 MiB p/s
2.69 GiB / 1.51 TiB [>_________________________________________________________________________________________________] 0.17% 41.20 MiB p/s
3.93 GiB / 1.51 TiB [>_________________________________________________________________________________________________] 0.25% 40.50 MiB p/s
1 Like
2

Looks like I found a sweet spot - got 81.87 MiB/s:

# uplink cp -p 8 -t 8 --parallelism-chunk-size 256M sj://akash-snapshots/rpc.nl/akashnet-2_2022-12-08T22:32:30.tar.zst /dev/null
download sj://akash-snapshots/rpc.nl/akashnet-2_2022-12-08T22:32:30.tar.zst to /dev/null
...
2.35 GiB / 1.51 TiB [>___________________________________________] 0.15% 78.92 MiB p/s
2.85 GiB / 1.51 TiB [>___________________________________________] 0.18% 81.87 MiB p/s
...
11.09 GiB / 1.51 TiB [>__________________________________________] 0.72% 81.57 MiB p/s
11.12 GiB / 1.51 TiB [>__________________________________________] 0.72% 81.29 MiB p/s

Useful link Hotrodding Decentralized Storage

3 Likes
3

Glad you were able to find that link on getting the fastest possible speeds.

Anything else we can help with? Don’t hesitate to reach out.

3 Likes
4

I’ve noticed that the speed drops down to 10 MiB/s instead of the usual ~100 MiB/s when am using bash redirect (>) or pipe (|):

# uplink cp sj://<REDACTED>.tar.zst /dev/null
download sj://<REDACTED>.tar.zst to /dev/null
/dev/null  22.62 MB / 2.33 TB [--------------------------------------------------------------------------------------------] 0.00% 1.09 GiB/s
/dev/null  48.54 MB / 2.33 TB [-----------------------------------------------------------------------------------------] 0.00% 1015.47 MiB/s
/dev/null  67.11 MB / 2.33 TB [--------------------------------------------------------------------------------------------] 0.00% 1.08 GiB/s
^C

# uplink cp sj://<REDACTED>.tar.zst - | pv -petrafb -i 5 -s 2328232517056 > /dev/null
15.9MiB 0:00:05 [3.17MiB/s] [3.17MiB/s] [>                                                                               ]  0% ETA 8:02:15:41
64.0MiB 0:00:10 [9.51MiB/s] [6.36MiB/s] [>                                                                               ]  0% ETA 4:00:22:03
 127MiB 0:00:15 [12.8MiB/s] [8.49MiB/s] [>                                                                               ]  0% ETA 3:00:38:32
^C83MiB 0:00:20 [11.2MiB/s] [9.16MiB/s] [>                                                                               ]  0% ETA 2:19:21:44

And same speed drop with a simple redirect uplink cp sj://<REDACTED>.tar.zst - > /dev/null.

The use case is we have 4 TiB of archive data that needs to be extracted directly from Storj DCS to the disk. The disk does not have extra 4 TiB of disk space to keep the archive file.

Hence we would normally use this command:

uplink cp -p 4 -t 4 sj://<REDACTED>.tar.zst - | pv -petrafb | ZSTD_NBTHREADS=0 zstd -cd | tar xf -

ZSTD_NBTHREADS=0 means “use as many threads as detected cpu cores on local system”. (Refs zstd/programs/README.md at 52bbf1be540a8f9b67fb675e15a645caecfcaf7d · facebook/zstd · GitHub )

I presume the uplink behaves differently depending on whether it’s writing to a file descriptor or directly to a file where it has the control over the writing process. Where with the file descriptor - it would just send the data to the stdout where shell or the pv tool handles the rest.

Let me know if there is any way to get around this?

5

What is version of used uplink?

6

Newer versions of uplink (version 1.57 or later) allow for buffering segments in memory which can help to accelerate download speeds when piping to other processes like TAR. There are many factors involved in the overall speed, including the hardware limitations of the machine where you run the command. That said, if your environment has significant amounts of bandwidth and compute, you should be able to achieve much higher speeds.

You can check out this article for more information.

Please confirm that you are using a newer version of uplink.

2 Likes
7

it’s the latest one:

# uplink version
Release build
Version:            v1.80.10
Build timestamp:    06 Jun 23 17:39 UTC
Git commit:         06b61d0f1930479141b9cc207c7f1c446c7b0e3b

PATH                      VERSION
storj.io/storj            (devel)
storj.io/common           v0.0.0-20230605212838-de3bae813a7c
storj.io/drpc             v0.0.33
storj.io/monkit-jaeger    v0.0.0-20220915074555-d100d7589f41
storj.io/picobuf          v0.0.1
storj.io/private          v0.0.0-20230405095015-9e5bbc1c7ca8
storj.io/uplink           v1.10.1-0.20230524150903-9e61c35f248e
8

I have the latest version v1.80.10 and the server is not running anything else besides the uplink tool.

The speed is pretty much same with --parallelism 32 flag:

This is the upload speed to DCS Storj

root-node-1  |  384MiB 0:00:21 [14.1MiB/s] [17.8MiB/s] [>                  ]  0% ETA 2:00:23:06
root-node-1  |  448MiB 0:00:25 [15.5MiB/s] [17.4MiB/s] [>                  ]  0% ETA 2:01:22:26

CPU: Intel(R) Xeon(R) CPU E5-2650 v2 @ 2.60GHz - 32 cores. 128 GB RAM.

Speedtest tool reports 500 Mbit/s (== 62.5 MiB/s) Up and Down link.

9

Could you please try to use 1.76.2?

10

Sure, I’ve tried 1.76.2 and 1.82.1 (latest as of today) and have recorded different test scenarios here:
https://asciinema.org/a/rqQdU3NlnDr41DDCYuf9kk5CK
asciicast

The average speed is always around 10-17 MiB p/s (the disk read speed is ~ 1.30GiB/s), server’s upload speed is around 346-436 Mbit/s (or 43-54 MiB p/s) depending on the remote server (tested with speedtest CLI tool).

11

This is correct. When the Uplink is able to download to a file descriptor, it actually downloads multiple ranges concurrently out of order. When you force the data through a pipe, it can’t be out of order, and so it can’t have as much concurrency.

We are continuing to improve our single instance download performance, but it will be hard to ever beat the throughput you can get when you have high concurrency. That parallelism flag really just can’t do much if you are forcing all bytes to be sequential like a pipe will.

3 Likes
12

Thank you for confirming this.

I guess the only “solution” for now is to request the double the amount of the storage to avoid piping the uplink so the data can be downloaded (or uploaded) at the fast speed.
Or have a plenty of RAM to leverage tmpfs for the same purpose.

Though, I’m still wondering whether there could be some sort of a “super-pipe” which could handle the bytes concurrently.