Uplink upload test

In short - We are looking into migrating some of our customers Veeam backup archived files to Storj using uplink.
So before migrating, I decided to run some tests to see how things would work out regarding egress (and ingress), and also server side encryption, since we want to encrypt everything in-house before any data are transferred to storj nodes.

So I’ve setup a test system configured with AMD Ryzen Threadripper 3970X (32 core), 128 GB RAM and a Samsung 1 TB 980 PRO NVMe HDD.
NIC is a Intel xx710 running at 25 Gbit/s, linked to one of our Cisco C9500-48Y4C switch, who is up-linked to our 100Gbit/s internet pipe. OS is Ubuntu 22.04.1 LTS

Using uplink (v1.66.1) I was able to get some decent results :

AMD server
----------
root@storj-gw001:~# time uplink cp -p 30 /root/256GB-file sj://test/256GB-file
upload /root/256GB-file to sj://test/256GB-file
238.42 GiB / 238.42 GiB [-----------------] 100.00% 480.69 MiB p/s

real    8m37.866s
user    110m55.502s
sys     147m53.547s

---

root@storj-gw001:~# time uplink cp -p 30  sj://test/256GB-file /root/256GB-file
download sj://test/256GB-file to /root/256GB-file
238.42 GiB / 238.42 GiB [-----------------] 100.00% 600.72 MiB p/s

real    6m59.733s
user    113m46.239s
sys     82m7.271s

Using htop + nload it was clear that the AMD CPU was the bottleneck since egress only avg. about 15 Gbit/s, so I added another (non-HT) 88 core Supermicro server with 512 GB of ram to the test setup, but unfortunately it was only equipped with a single 10Gbit nic.
Ran uplink cp -p 50 /root/150GB-file sj://test/150GB-file which immediately maxed out the 10G nic.

Supermicro Server
-----------------
root@server030:~# time uplink cp -p 50 150GiB.file sj://test/150GiB.file
upload 150GiB.file to sj://test/150GiB.file
150.00 GiB / 150.00 GiB [-----------------] 100.00% 313.03 MiB p/s

real    8m11,932s
user    206m10,590s
sys     247m34,451s
---

root@server030:~# time uplink cp -p 50 sj://test/150GiB.file 150GiB.file
download sj://test/150GiB.file to 150GiB.file
150.00 GiB / 150.00 GiB [-----------------] 100.00% 380.97 MiB p/s

real    7m9,445s
user    168m8,626s
sys     304m6,148s

Total egress hit 25 Gbit/s when uploading from both servers simultaneously :

So migrate/offloading Veeam backup files to storj for archiving, doesn’t seems to be a problem. We just need to scale up on CPU cores or/and servers to get even faster egress…

If finance department doesn’t kill me, I’ll try to acquirer more servers or maybe get a hold of a server with a AMD Ryzen Threadripper 3990X 64 Cores CPU and run the test again.

Th3van.dk

6 Likes

You may also try to use bigger chunks with --parallelism-chunk-size, but it will use RAM.

1 Like

How much data do you have if you can send 25 Gbit/s and you need more?

Backups = unprofitable data since they are usually not pumped back and there is no payment for traffic. Sadly.

1 Like

A lot more profitable than nothing at all.
I, for one, welcome any data coming my way.

Thank you @Th3Van for sharing your findings with us :slight_smile:

2 Likes