The Filezilla testing made one thing apparent for me: Upload to Tardigrade needs improvement.
The requirement to upload almost 3 times the data quantity is a huge road block. Upload bandwidth is already limited due to asymmetric DSL and Tardigrade makes this disadvantage even worse.
I have no idea if and how this can be improved, but it definitely should. A user should not have to upload more than what the data size is, maybe 1.3 or 1.5 fold, but not 2.7x.
Same goes for downloading. However I am not quite sure how this works currently. I am just thinking, if each piece of data can be reconstructed by downloading only the minimum number of chunks, why download the whole thing? But as I am not sure how this is implemented, maybe someone can shed a light if there is room for improvement.
There is no download overhead like with the upload. You only download what you need, any non-finished download will be canceled.
But the upload overhead is indeed a bit annoying as our upload rate is rather limited. However, I’m also not sure how that can be solved because even if you cut down to 1.5x the data, the network will still need the remaining 1.2x to get the data distributed correctly.
The question is how much do you need? This is what I am not sure about. Does a download client act like a satellite, downloading only like 80 chunks and reconstructing the data piece from that. Or does it download the entire thing?
I cannot tell. But I guess when you upload to AWS which also offers redundancy, then you upload it once and it gets mirrored behind the scenes. You don’t have to upload it to the multiple Amazon data centers.
For Tardigrade this would mean, user uploads his stuff once and nodes and satellites take care of chunk creation and distribution. That’s just the idea. I cannot tell at all if it would be feasible or impossible to do it in a way like that.
A similar solution would be that client only encrypts and splits his data into pieces and uploads those to multiple different nodes. And the nodes take care of conversion into shards and distribution on the network, similar to what the repair job does.
I guess it would be possible but complex. The satellite would need to verify that the node created the pieces correctly and distributed them to other nodes. So the satellite would actually need to recreate the original piece in order to know that the node didn’t just send random data to the other nodes. So in this case the satellite would still need more than half the bandwidth usage than if it had done it by itself.
Scenario nodes upload shards: 1GB file from customer to e.g. 10 nodes, 10 nodes create shards but upload them to the satellite for verification -> 2.7GB upload to satellite.
Scenario server distribute shards: 1GB file from customer to satellite, satellite distributes 2.7GB -> 3.7GB bandwidth usage
I don’t see a system that doesn’t need the satellite to verify the distributed shards. But maybe that’s just me.
Ok I just thought of one: The client upload could create all neccessary shards for comparison but never upload them. So the final nodes then send a hash of their shard to the client for verification. Still a complex system as there is more to it than just that but wouldn’t require the satellite to verify pieces.
yeah but it also bears the question of who is going to pay for the extra egress the nodes have to do? Currently it’s free for the customer to upload a 1GB file (2.7GB of data) and nodes just have to download it and as download bandwidth is a lot higher than upload it doesn’t matter. But if nodes now also need to upload all the pieces to other nodes, they generate 2.7GB of egress. Who’s going to pay for that?
Additionally, this moves certain problems from the uplink to node. For example bandwidth constraints. The client uploads 1GB to e.g. 10 nodes, so 100MB per node. Now it is done but has to wait for the confirmation of those nodes to have the data distributed. So those 10 nodes now have to create the shards first, but that should not take long. However, now each of those 10 nodes needs to upload 270MB to other nodes. So even if they all saturate an upload bandwidth of 5MB/s (I guess most home users won’t have more), it would still take 54 seconds! That means the customer has to wait for 54 seconds doing nothing. Try explaining that to a customer… You can of course make that better by uploading to more nodes but it doesn’t scale perfectly. You can’t just upload to 100 nodes as the resulting filesize would be way too small after creating 80 shards.
I guess such a system would work for the repair job. Satellite downloads shards, creates new shards and has the hashes for comparison. Now it sends the job to a node, which downloads all pieces as well and uploads the shards to other nodes, which send their hash to the satellite for comparison. That means, the satellite saved 2/3 of its bandwidth but the shards get downloaded twice, so more money for the nodes
Only 10 nodes is very few. You would need as many as possible and file size parts as small as possible.
Yes, but look, I have just again uploaded a file roughly 1.4 GB to Tardigrade with Filezilla. It took around 21 minutes. Now with a rough calculation, I have uploaded around 4 GB. Normally the upload should have taken only around 8 minutes or so. So the customer sits and waits for over 13 minutes longer. Compare that with 54 seconds. I believe customer would be very happy waiting 54 seconds longer instead of 13 minutes.
But downloading twice and creating shards twice that does not sound super efficient for the centralized satellites. But maybe it would have to be that way.
But how does a repair job work today? Does the satellite reconstruct the entire file when repairing and recreate the shards from it?
If so, then the satellite would only have to reconstruct much smaller files as the shards get created from file parts only. This could mean that a repair job maybe can get executed faster and uses much less bandwidth than before. Maybe?
Small piece sizes would hurt storagenodes (because of performance) so getting ~2MB would be optimal. So 80*2MB=160MB chunk size. So split a file in 160MB chunks. So for a 1GB file you could upload to 7 nodes which is not much. So a piece size of 1MB is still acceptable but smaller, then you would get 14 nodes.
That’s because your upload is slow. on a 5MB/s upload it would take 14 minutes to upload 4GB. But many nodes will have a lower upload speed too. So let’s say you upload to a “distribution” node that only has 1MB/s upload. Using the example above with 1MB piece size it gets 1/14 of your filesize so of those 1.4GB it gets 100MB. So this node will take 100 seconds to upload.
That might be ok for your connection but if you are a big customer with a sever grade connection of 1000MBit/s you could upload with 100MB/s. Then you are a lot faster than the nodes can distribute data and therefore better off uploading the pieces yourself.
Which in turn means, you’d need a system that automatically prefers one of both solutions or at least lets the user decide which one it wants to use.
The satellite (or a different server actually) downloads all pieces, reconstructs the file/chunk and uploads all of them to the nodes.
I’m not sure about that. The file distribution in shards shouldn’t be different in any scenario.
I agree. If you have big upload bandwidth available, you can upload to more nodes in parallel and therefore upload really fast. But still you’d have to upload 2,7 times the data quantity that you originally have. A customer with such a huge connection probably has really really huge amounts of data to be send. So even with large bandwidth, they might not like the fact, that they have to upload almost 3 times the size. If you remember the NASA problem, they wanted to increase their AWS storage by 215 petabytes I am not sure if they would be happy, if they have to upload almost 650 petabytes instead to Tardigrade. Even with a fast connection it does sound absurd.
I am not sure. It should make a difference, if a satellite has to reconstruct a 1 GB file or only a 100MB (or 10 MB) part of it.
