I have recently done some testing using Tardigrade as a backend for duplicati backups and also compared it with simple upload of large-ish archives with uplink.
During these tests I have monitored metrics such as traffic through the network interface on the executing host.
This data shows, that I need about 9.5h to upload about 10GB of data to an sj://mybucket, while network upload traffic was consistently about 1.2 MB/s during this time (which amounts to approximately 40GB, or 4 times the data size stored in the bucket.)
I understand this must be due to the uploading of redundant data, to ensure data durability.
However, the whitepaper leads me to expect more something in the range of a factor 1.5 (as in f=data_uploaded / data_stored), assuming a sufficiently large number of nodes. Must I conclude from this that there are “not enough” nodes in the Tardigrade network to achieve this efficiency, or is there another explanation?
TIA for any insights, both practical and theoretical.
@jammerdan
Thanks for the link (how did I miss that, I thought I searched before posting )
Really interesting discussion there. I guess for now the pragmatic thing would be to just accept the upload overhead (“jammer dan” ) , anyway if doing de-duplicating backups it’s a bit less painful.
A 29MB file will get cut up into 110x1MB pieces, those pieces will get uploaded to the nodes and uploads will be canceled after 80 of the pieces have been uploaded successfully.
If all uploads are about the same speed, then, when the 80 fastest are done, the remaining ones will be “almost done”, so you may upload 80-110MB for a 29MB file, so, the upload expansion factor is 2.75 - 3.79. This does not count any overhead with headers and stuff, so in reality the upload expansion factor would be a little bit higher than that.
This should go down a bit with network getting more use (hence less bad nodes) and future (I assume) reduction of expansion factor, however, at the moment it’s pretty high.
In general Storj is absolutely incompatible with AWS users trying to save money because it costs way more to get the data out of the cloud. Only users who do not have their business in the cloud already are financially compatible with this.
@Pentium100 Thanks for that insight. This explains quite nicely what I’m measuring.
Since I have a flat rate and no hard time constraints, it’s just painful but no show stopper. Future major improvement on that part would be really nice but I suspect quite difficult if not impossible without structural and conceptual changes (with likely other inconvenient side effects).
)
) , anyway if doing de-duplicating backups it’s a bit less painful.