In short - We are looking into migrating some of our customers Veeam backup archived files to Storj using uplink.
So before migrating, I decided to run some tests to see how things would work out regarding egress (and ingress), and also server side encryption, since we want to encrypt everything in-house before any data are transferred to storj nodes.
So I’ve setup a test system configured with AMD Ryzen Threadripper 3970X (32 core), 128 GB RAM and a Samsung 1 TB 980 PRO NVMe HDD.
NIC is a Intel xx710 running at 25 Gbit/s, linked to one of our Cisco C9500-48Y4C switch, who is up-linked to our 100Gbit/s internet pipe. OS is Ubuntu 22.04.1 LTS
Using uplink (v1.66.1) I was able to get some decent results :
AMD server
----------
root@storj-gw001:~# time uplink cp -p 30 /root/256GB-file sj://test/256GB-file
upload /root/256GB-file to sj://test/256GB-file
238.42 GiB / 238.42 GiB [-----------------] 100.00% 480.69 MiB p/s
real 8m37.866s
user 110m55.502s
sys 147m53.547s
---
root@storj-gw001:~# time uplink cp -p 30 sj://test/256GB-file /root/256GB-file
download sj://test/256GB-file to /root/256GB-file
238.42 GiB / 238.42 GiB [-----------------] 100.00% 600.72 MiB p/s
real 6m59.733s
user 113m46.239s
sys 82m7.271s
Using htop + nload it was clear that the AMD CPU was the bottleneck since egress only avg. about 15 Gbit/s, so I added another (non-HT) 88 core Supermicro server with 512 GB of ram to the test setup, but unfortunately it was only equipped with a single 10Gbit nic.
Ran uplink cp -p 50 /root/150GB-file sj://test/150GB-file which immediately maxed out the 10G nic.
Supermicro Server
-----------------
root@server030:~# time uplink cp -p 50 150GiB.file sj://test/150GiB.file
upload 150GiB.file to sj://test/150GiB.file
150.00 GiB / 150.00 GiB [-----------------] 100.00% 313.03 MiB p/s
real 8m11,932s
user 206m10,590s
sys 247m34,451s
---
root@server030:~# time uplink cp -p 50 sj://test/150GiB.file 150GiB.file
download sj://test/150GiB.file to 150GiB.file
150.00 GiB / 150.00 GiB [-----------------] 100.00% 380.97 MiB p/s
real 7m9,445s
user 168m8,626s
sys 304m6,148s
Total egress hit 25 Gbit/s when uploading from both servers simultaneously :
So migrate/offloading Veeam backup files to storj for archiving, doesn’t seems to be a problem. We just need to scale up on CPU cores or/and servers to get even faster egress…
If finance department doesn’t kill me, I’ll try to acquirer more servers or maybe get a hold of a server with a AMD Ryzen Threadripper 3990X 64 Cores CPU and run the test again.
Th3van.dk
