Uplink upload traffic is factor 3-5 larger than data amount

[ppenguin]

I have recently done some testing using Tardigrade as a backend for duplicati backups and also compared it with simple upload of large-ish archives with uplink.

During these tests I have monitored metrics such as traffic through the network interface on the executing host.

This data shows, that I need about 9.5h to upload about 10GB of data to an sj://mybucket, while network upload traffic was consistently about 1.2 MB/s during this time (which amounts to approximately 40GB, or 4 times the data size stored in the bucket.)

I understand this must be due to the uploading of redundant data, to ensure data durability.
However, the whitepaper leads me to expect more something in the range of a factor 1.5 (as in f=data_uploaded / data_stored), assuming a sufficiently large number of nodes. Must I conclude from this that there are “not enough” nodes in the Tardigrade network to achieve this efficiency, or is there another explanation?

TIA for any insights, both practical and theoretical.

Cheers, ppenguin.

[Pac]

@ppenguin

the whitepaper leads me to expect more something in the range of a factor 1.5

The ratio is closer to 2.7.

So, to upload a 1GB file, you would need to upload something like 2.7GB (+ networking overhead) of data.

[jammerdan]

Up- and download quantity for Tardigrade users

The Filezilla testing made one thing apparent for me: Upload to Tardigrade needs improvement. The requirement to upload almost 3 times the data quantity is a huge road block. Upload bandwidth is already limited due to asymmetric DSL and Tardigrade makes this disadvantage even worse. I have no idea if and how this can be improved, but it definitely should. A user should not have to upload more than what the data size is, maybe 1.3 or 1.5 fold, but not 2.7x. Same goes for downloading. However I…

[ppenguin]

@Pac

The ratio is closer to 2.7 .

Thanks for the insight.
Could you shed more light on where this number comes from? Is part of this metadata traffic with the satellite?

And how does it explain the experimental values I’m seeing (unmistakably factor 4)?

[ppenguin]

@jammerdan
Thanks for the link (how did I miss that, I thought I searched before posting )
Really interesting discussion there. I guess for now the pragmatic thing would be to just accept the upload overhead (“jammer dan” ) , anyway if doing de-duplicating backups it’s a bit less painful.

[Pentium100]

@ppenguin

And how does it explain the experimental values I’m seeing (unmistakably factor 4)?

It’s because of the canceled uploads.

The numbers may have changed by now, but:

High cancel rate in upload requests

How is your HDD connected to the PC with storagenode? Canceled uploads or downloads can be result of cut of long tail or canceled by customers. Every upload is cut up into 110 erasure encoded pieces of which 29 are needed to recreate the data. Uploads stop after 80 of those pieces have been uploaded successfully. The other 30 nodes who were the slowest for that transfer will see the upload canceled error in their logs. The same for downloads, but uplink selects 39 nodes for download and stops…

A 29MB file will get cut up into 110x1MB pieces, those pieces will get uploaded to the nodes and uploads will be canceled after 80 of the pieces have been uploaded successfully.

If all uploads are about the same speed, then, when the 80 fastest are done, the remaining ones will be “almost done”, so you may upload 80-110MB for a 29MB file, so, the upload expansion factor is 2.75 - 3.79. This does not count any overhead with headers and stuff, so in reality the upload expansion factor would be a little bit higher than that.

[Alexey]

How much retries was during upload?

[Storgeez]

This should go down a bit with network getting more use (hence less bad nodes) and future (I assume) reduction of expansion factor, however, at the moment it’s pretty high.

In general Storj is absolutely incompatible with AWS users trying to save money because it costs way more to get the data out of the cloud. Only users who do not have their business in the cloud already are financially compatible with this.

[ppenguin]

@Pentium100 Thanks for that insight. This explains quite nicely what I’m measuring.

Since I have a flat rate and no hard time constraints, it’s just painful but no show stopper. Future major improvement on that part would be really nice but I suspect quite difficult if not impossible without structural and conceptual changes (with likely other inconvenient side effects).