Short-lived data

I wish that were true… but I’m really doubting this and have been for a while. Since every upload starts 110 pieces but only keeps 80, you would expect 80/110~=73% to be the average success rate we see. But everyone has scores higher than 95% and almost everyone has scores above 99%. So something really isn’t adding up there.
What I think is actually happening is that the transfer finishes on the node end, but the piece still isn’t part of the fastest 80. Meaning it won’t end up in metadata on the satellite end and will be cleaned up during garbage collection.
We’ve kind of known this for a while but assumed that when comparing numbers it would still give some indication. But at this point I’m really starting to doubt whether it still has any use at all.

I’ve looked at my oldest and largest node. That node still grew by around 500GB this month. And with 313GB normal ingress and 353GB repair ingress, that again matches with a 25% loss based on ingress and 0.8% loss based on static storage.

0.8% * 17TB stored = 136GB loss
25% * 313GB ingress = 78GB loss

313GB + 353GB - 136GB - 78GB = 452GB growth
This is even slightly lower than actual growth.

Now we should keep in mind that nodes don’t all behave the same anymore now that geographic restrictions can be used. But these differences still seem way too big to me.

2 Likes