just a question, satellites split data into 80 pieces and those get uploaded to 80 nodes.
What happens if e.g. 3 uploads out of the 80 uploads fail? Does a satellite always make sure that 80 pieces are successfully uploaded?
Thanks very helpful and understandable!
A follow up on that, initially there are 80 healthy pieces (of which 29 are needed to reconstruct the data). This can erode until repair threshold is hit at 35; then download 29 and build another 16 to total 45 healthy pieces, why not rebuild to 80 again?
These numbers are choosen from satellite operators and will be adjusted according to the needs of the network. For any activity the satellite operators pay the sno it is always a question of cost.
But your are right the text is a bit misleading.
…The Satellite re-encodes the segment from the 29 pieces and rebuilds 16 additional pieces. The 45 pieces of the segment are then uploaded to storage nodes.
The satellite start download after reading the threshold of 35, it is downloading 29 pieces and it will created 45 new pieces out of these 29, which will uploaded. 45 + 35 are again 80 the file is back where it was at beginning.
3 Likes
@BlackDuckthanks for the clarification, that is what I thought, so fall down to 35 and then repair so again 80 healthy pieces are there.
The 35 threshold up to the sat operator, so they may choose 45 instead, if they want?
The satellite operater have full controll over this, and can choice what they want, they have to pay for it. Correct if they want run it at 45 they can do this. There are pro and con for both.
1 Like
right, and if available piece count falls to 28 the piece can no longer be reconstructed, but there must be some math behind the choice 35. Although it feels too close to 29 to me…
The moment that number drops below 35, the repair starts. This would only be a problem if 6 or more nodes go offline or lose the data simultaneously. So far this has never happened (the last report on this is a few months old, but I have no reason to believe this has changed). I also remember some mention that the repair threshold was raised in order to better test the repair process. Not sure if that’s still in place.
The whitepaper has some calculations for durability and Storjlabs has a team of data scientist who’s job it is to predict these risks.
As Satelite can only understand that data is lost by Audit, and it makes audit not very often, so speed of replication start can be small. But /24 policy also help.
35 does feel way too close to 29 to me too. Especially considering that sattelites may not be aware of other missing parts as they cannot audit everything in real time.
But if data scientists approve… I’ll leave it to the professionnals 