Amazon S3 vs. Tardigrade

@flo interesting, let’s see how this improves when we come closer to production. At the moment the network does not feel fast at all, I also did some testing.

@flo Can you share were you are doing your testing from, what satellite, and what Amazon region? The results on performance can vary a ton based on your connection, and where you are. Sometimes faster, sometimes slower, sometimes on par.

Starting this week we have an entire team devoted to making Storj faster than Amazon S3, and increasing performance to its max. Delete is definitely too slow, and we have some changes in the pipeline to make it faster.

4 Likes

@super3 very good to know, keep us updated please :slight_smile:

1 Like

@super3 the server is in Germany, and I’m on the europe-west-1 satellite; S3 region is Germany.

It’s hard to be on par (or even faster) than Amazon if uploading 10 GB requires more than ~34 GB to be uploaded to Tardigrade.

I remember being told that we’re on par with Amazon in the Q3 2019 town hall - I don’t agree :slight_smile:

2 Likes

@flo
Can you describe your setup?
PC? windows, linux? CPU?
Connection?
Router?
Did you uploaded 1 file or it is multiple files?
Does Amazon also encrypted bandwidth?
Your DNS setups? Local dns or Google?

It’s hard to be on par (or even faster) than Amazon if uploading 10 GB requires more than ~34 GB to be uploaded to Tardigrade.

Depends on the your upload bandwidth since the upload is in parallel rather than linear. It can be faster, but it can be definitely harder if you have slower connection.

I remember being told that we’re on par with Amazon in the Q3 2019 town hall - I don’t agree :slight_smile:

Well that is because you are correct. Still work to do. :sweat_smile:

Has anyone tested uncompressed files vs compressed files when upload to Amazon S3 vs Tardigrade?

I test with random noise files which are incompressible. I see upload speeds of slightly below 250mbit, with gaps in-between the segments on a 500mbit upload connection. Assuming those gaps can be removed by doing DNS and caching on the satellite you’d be looking at an effective upload speed of 80mbit after taking RS inflation rate into account.

Edit: location is the Netherlands, using europe-west-1

Shouldn’t make much of a difference.

Upload a 1GB file (openssl rnd generated)

root@server030:~# time uplink cp ./1G.data sj://test/1g-test-file2                                     
2020-01-29T02:30:27.167+0100    INFO    Configuration loaded from: /root/.local/share/storj/uplink/config.yaml
1.00 GiB / 1.00 GiB [------------------------------------------------------------------] 100.00% 18.62 MiB p/s
Created sj://test/1g-test-file2

real    0m55.348s
user    1m41.474s
sys     1m42.360s

Download of the same 1GB file :

root@server030:~# time uplink cp sj://test/1g-test-file2 /tmp/1gb-test-file02.tmp  
2020-01-29T02:33:25.297+0100    INFO    Configuration loaded from: /root/.local/share/storj/uplink/config.yaml
1.00 GiB / 1.00 GiB [------------------------------------------------------------------] 100.00% 23.87 MiB p/s
Downloaded sj://test/1g-test-file2 to /tmp/1gb-test-file02.tmp

real    0m43.283s
user    0m43.905s
sys     0m10.110s

Random spikes up to 1,4Gbit/s while uploading and around 1,2Gbit/s while downloading.

Deleting the 1GB file :

root@server030:~# time uplink rm  sj://test/1g-test-file2      
2020-01-29T02:46:46.086+0100    INFO    Configuration loaded from: /root/.local/share/storj/uplink/config.yaml
2020-01-29T02:46:46.659+0100    INFO    running on version v0.30.5
Deleted sj://test/1g-test-file2

real    0m10.468s
user    0m0.110s
sys     0m0.040s
  • 4 x E5-4669 V4
  • 512GB RAM
  • Intel DC P3700 NVMe
  • 10Gbit/s Interent
  • OS : Ubuntu 18.04.3 LTS
  • Satellite: Europe-West-1
  • Location : Denmark
2 Likes

Might also be useful to see a 10 GB file speed test to compare to @flo

Compared to my last test, upload has improved a lot; downloading the file took a little longer:

Uploading 10 GB:
Tardigrade : 296.5s

Downloading 10 GB:
Tardigrade 535.5s

Deleting one 10 GB file:
Tardigrade: 43.3s

Looks like uploading is 300-400% faster, and deleting is 50% faster. The performance team was just formed this week, so many more improvements to come.

1 Like

@super3
some of it shold be downloading in your words?

1 Like

Uploading big files to tardigrade seem to take longer than uploading smaller files with the same total size

So uploading was 270 Mbps, downloading 150 Mbps and deleting 3.6 segments/s. I guess there’s still room from improvement but this is already much better than it was in some other threads. I lack bandwidth to do these tests myself.

Wondering why the deletion takes so long; all the node should do is send the file ID to delete to the satellite.

1 Like

It actually sends data for every segment or maybe even piece that should be deleted. And I think right now the satellite still sends out the deletes to the nodes before returning a success to the uplink. Although those messages are small, it’s still a lot of them. I have no doubt more optimization is in the works for this.

I see. I thought the instant delete was implemented with v0.31.
Did the original “return success to uplink immediately” fix include the uplink sending just the file ID for delete operations or is the uplink still going to send each segment/piece ID for deletion with that fix?

I didn’t dive into the code. But I imagine it would at least be on the segment level. Since that’s the entity satellites mostly deal with. Hopefully not individual pieces though. And I’ve argued before for giving the node the possibility to send a complete list of segments perhaps even across multiple files as well. So only one round trip to the satellite is needed.

2 Likes

Right, either segment ID list or file ID list, any kind of individual segment deletion is unacceptable due to latency. Shouldn’t be too much to change…

Maybe parallel requests but lists are much neater and it wouldn’t make sense to delete only half the segments in any case, it’s always going to be all the segments…

1 Like