An empirical latency-based geoseparation idea

[Toyoo]

Just had some idea and wanting to share while I remember it. Comments welcome!

Right now storage of erasure shares is distributed by means of /24 IP blocks. We know this can be worked around by node operators to some degree by, let say, using multiple ISPs or setting up VPNs. However, this method has a crucial feature of being extremely simple and fast when selecting nodes.

I’m now imagining a network of, let say, 40 «measurement nodes» controlled by a satellite owner, and located in various parts of the world, preferably so that each node is in a different data center, different city, different geographic area. Each of them tests all nodes periodically and measures response latency (maybe as a side effect of audits, so these wouldn’t be actually new queries?). For each storage node we find the measurement node with the lowest latency and classify it as belonging to the geographic area of the measurement node.

Then, at storage node selection, nodes are selected so that no geographic area is selected more than X times, for some tunable X. The area would be identified by a single small number, one byte, so selection could again be fast.

It would be much more difficult to cheat, as it’s hard to have a single location have low latency to more than a small number of data centers. VPNs only increase latency, and multiple ISPs will likely connect to similarly-located backbone nodes anyway.

As for feasibility of running so many measurement nodes, they could be some smallest VPSes at, let say, Azure, which is already present in 54 regions. Alternatively I suspect that it should be possible to run a smaller number of measurement nodes and consider a geographic area as area defined by two lowest latencies. E.g. an area with the two lowest latencies to Paris and Frankfurt is likely different than an area with low latencies to Paris and Madrid.

While I don’t think the current approach is so bad that it must be changed, the above idea would probably allow for more fair distribution of data and take into consideration the fact that tech hubs are likely to amass large number of IP blocks.

[Erikvv]

I think this is an interesting idea which might give better incentives to operators.

Storj developers aren’t set on any new approach yet but we do recognize that the node selection algorithm possible contains some gains to be found in terms of resilience and fairness.

[BrightSilence]

I definitely like this idea or something very similar to it. Though with some implementations through vpn or vps routing it may not be a perfect solution yet as people could still use that to look like they are spread out. But it’s probably a lot better than IP location alone.

It would also incentivize more node operators to pop up in geographically less crowded areas. Though it would kind of suck for me personally as the Netherlands is one of the most node saturated places on the planet.

[Toyoo]

@BrightSilence

I definitely like this idea or something very similar to it. Though with some implementations through vpn or vps routing it may not be a perfect solution yet as people could still use that to look like they are spread out. But it’s probably a lot better than IP location alone.

Yeah, it’s a lot more difficult to fake being close to something in terms of latency.

@BrightSilence

It would also incentivize more node operators to pop up in geographically less crowded areas. Though it would kind of suck for me personally as the Netherlands is one of the most node saturated places on the planet.

Tech hubs are likely to also have a lot of customers wanting to store data. With this idea it would be indeed more difficult for nodes in tech hubs to be selected as upload candidates, but on the other side, when they are selected, they’d be more likely to win races against nodes further away from these local customers.

[Toyoo]

Sorry, I believe you have misunderstood the context of the post. It’s not about ranking nodes better/worse, it’s about ensuring data is stored on nodes that won’t fail together.

[Toyoo]

@CutieePie

geolocation and latency would not be enough to establish “a node that won’t fail” is all I was saying, and that other characteristics need to be considered.

Storj’s whitepaper strongly depends on independence of failures, and one way to ensure that is making sure erasure shares are placed on nodes as far as possible from each other.

@CutieePie

With regard latency, without posting something that would give too much detail - it is very easy using technologies on AWS to implement a Storj reverse proxy gateway, that can auth on identity information and DRPC calls, and pass through requests for blocks to a node over tunnels - it would be technically easy to create a node response to the standard pre-amble to respond very quickly - it also allows of Edge caching so Storage Nodes could pre-load shards.

Which is why I suggested bundling audits with these queries.

@CutieePie

One of the nice things about the random nature of node selection is, it is technically challenging to game this for an advantage [not impossible] - soo as we get into more technical criteria, it opens up attack vectors for abuse.

Please be careful when talking about randomness in presence of a potentially malicious adversary. I’m pretty sure that, let say, Amazon is still capable of fielding an army of nodes on more class C blocks than the whole current Storj network.

[BrightSilence]

@Toyoo

when they are selected, they’d be more likely to win races against nodes further away from these local customers

In theory this should already be the case, but in practice the connection latency is only a tiny part of the chain and compared to the time to process and store data, it’s ultimately irrelevant.

That said, if it’s better for the network, this shouldn’t stop anyone from implementing that feature. The incentive it creates is good for the network as well as it means the less nodes there are in a certain place, the more you can earn running a node. Which promotes more decentralization.

@CutieePie

geolocation and latency would not be enough to establish “a node that won’t fail”

You omitted the most important word in that quote. @Toyoo said “nodes that won’t fail TOGETHER”. Storj is built to deal with failure as long as they aren’t large and correlated failures. Ensuring wide spread of data ensures that failures if they occur aren’t correlated and are spread out over time.

It’s ultimately irrelevant whether someone uses tricks like you explained to fake a faster ping response as whether you do that or not, the system @Toyoo suggests would still result in the same location seeing the lowest ping times. And nobody is suggesting using it to rank and judge nodes performance. If you want that, you need to base it on time to respond with the data requested. Which could be an extension of how audits work for example. But that is not what is being suggested here.

[Toyoo]

I’ve noticed someone on reddit linked this paper, which seems to be quite relevant for the idea: Estimation of Internet Node Location by Latency Measurements — The Underestimation Problem (PDF). Linking it here for reference.

[Pac]

I think this idea should get more traction! ^^
The /24 subnet thing isn’t really great from an SNO perspective even though it probably gets the Job done at network leve.

Why isn’t this “votable”?

[Toyoo]

This is more a research idea than just a single feature. Though, if it will help gauge interest in doing the research, let’s make it votable…

[peter_linder]

Another idea is to keep track of what autonomous system a node resides in. This is a simple lookup in the BGP table that can be done centrally.

When selecting nodes, make sure that only X nodes end up within one. This is a 32 bit value attached to each node so lookup should be fast for this too.

AS is arguably the largest failure domain present on the Internet and it is not feasible for someone to aquire lots of AS numbers for the reason of running a storage node, yet enough exists and are online to make up a variety.

This will work for the VPN “workaround” too, since a VPN provider is typically their own AS and stuffing several nodes in there will not increase ingress.

It will however be bad for customers/SNOs in very large providers compared to customers of small providers, so it will need to be combined with some other metric I think to compensate for this.