How does th3van do it?

[boelle]

http://www.th3van.dk/SM01-hardware.txt

so one host, but how do you assign different segmented IP’s to each node?
@Th3Van up for sharing the secret sauce ?

[alpharabbit]

Assigning IPs is basic networking stuff. The only question is how to get those IPs for cheap.

[boelle]

Yes but the traffic goes through the same connection….

So for me its a bit “advanced”

[arrogantrabbit]

No, the question is how is such concentration is tolerated by storj.

If I run two nodes on my single host I’m the villain. But if I run 2892 nodes in the same shed — I’m valued member of node operator community. Go figure.

[alpharabbit]

@arrogantrabbit

No, the question is how is such concentration is tolerated by storj.

Because they gave up on real decentralization long ago. Storj select is even worse in my opinion.

[boelle]

@arrogantrabbit

But if I run 2892 nodes in the same shed

And that is what i’m curious about…. Ie how in practice…

And my isp as most are only able to give me one ip, but speed i can just upgrade

So i asume that some “trickery” is done… but again how?

[jammerdan]

@boelle

So i asume that some “trickery” is done… but again how?

Obviously he is a pro with access to all kinds of different pro stuff:

A Follow-up on the New Minimum Usage Fee – and a Request for Feedback

I’ve been thinking about buying a Supermicro 1 PB dedicated server for the Storj Select network, and placing it in our EU DC, that is ISAE 3402 certified (not a SOC2 certification, but more like a EU version of it) But reading this and other threads, I get the feeling that it might be better spending the money on dedicated SM compute hardware than storage, and offer it for a free one or two year period to Storj in our DC. Dedicating it for running some repair and delete services for the public…

[boelle]

Well i hope he or someone else will enlighten me on how it can be done

[Th3Van]

Keyword is PBR (Policy Based Routing in Cisco terms, or routing policy in Ubuntu terms)

I’ve added a bunch of small subnets, supplied from our ISP’s ($$$) and our ASN($$$), to a single interface :

root@server030:~# ip a
2: ens17f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UP group default qlen 9000
    link/ether fe:de:ab:e2:02:50 brd ff:ff:ff:ff:ff:ff
    altname agemam69s0f0
    inet 37.35.98.254/30 brd 37.35.98.255 scope global ens17f0
    inet 37.35.99.254/30 brd 37.35.99.255 scope global ens17f0
    inet 37.35.100.142/30 brd 37.35.100.143 scope global ens17f0
    inet 37.35.101.254/30 brd 37.35.101.255 scope global ens17f0
    inet 37.35.102.10/30 brd 37.35.102.11 scope global ens17f0
    inet 37.35.103.254/30 brd 37.35.103.255 scope global ens17f0
    etc...

Add ip router rules for each subnet :

ip rule add from 37.35.98.254 lookup 1006
ip rule add from 37.35.99.254 lookup 1007
ip rule add from 37.35.100.142 lookup 1008
ip rule add from 37.35.101.254 lookup 1009
ip rule add from 37.35.102.10 lookup 1010
ip rule add from 37.35.103.254 lookup 1011
etc...

Add a default route for each subnet and route to the propper gateway IP using the rules above :

ip route add default via 37.35.98.253 table 1006
ip route add default via 37.35.99.253 table 1007
ip route add default via 37.35.100.141 table 1008
ip route add default via 37.35.101.253 table 1009
ip route add default via 37.35.102.9 table 1010
ip route add default via 37.35.103.253 table 1011
etc...

Alternative use netplan config file, so the PBR is added every time the server reboots :

network:
  version: 2
  renderer: networkd

  ethernets:
    ens17f0:
      addresses:
        - 37.35.98.254/30
        - 37.35.99.254/30
        - 37.35.100.142/30
        - 37.35.101.254/30
        - 37.35.102.10/30
        - 37.35.103.254/30
          # etc...

      routes:
        - to: 0.0.0.0/0
          via: 37.35.98.253
          table: 1006
        - to: 0.0.0.0/0
          via: 37.35.99.253
          table: 1007
        - to: 0.0.0.0/0
          via: 37.35.100.141
          table: 1008
        - to: 0.0.0.0/0
          via: 37.35.101.253
          table: 1009
        - to: 0.0.0.0/0
          via: 37.35.102.9
          table: 1010
        - to: 0.0.0.0/0
          via: 37.35.103.253
          table: 1011
          # etc...

  routing-policy:
    - from: 37.35.98.254
      table: 1006
    - from: 37.35.99.254
      table: 1007
    - from: 37.35.100.142
      table: 1008
    - from: 37.35.101.254
      table: 1009
    - from: 37.35.102.10
      table: 1010
    - from: 37.35.103.254
      table: 1011
      # etc.

Th3Van.dk

[boelle]

So if my isp can’t (or will not) supply more than one ip I’m basically f’ed?

Or are there other ways for us at home?

BTW what do each extra ip cost these days? Just approximate

[RecklessD]

Well, you could use a VPS. Depends on whether you read the Terms and Conditions as to wether it breaches any.

A fellow operator has a well written howto.

[arrogantrabbit]

Well, you can use 300 VPN connections. I don’t see material difference between multiple subnets issued by your ISP, vs routed via other ISPs (aka VPNs).

Look:

• The subnet limit rule was introduced to prevent high node concentration under the same roof. The node math works assuming most nodes are uncorrelated.
• Using VPN to bypass this rule was therefore frowned upon.
• But using multiple subnets from ISP is OK with Stroj?

Good to know… Good… To… Know…

Storj needs to make the policy clear.

[Roxor]

I suppose ‘the policy’ is interesting at an intellectual level. But my guess is most of the owners of the 30000+ nodes care about real-world implementation of rules. If I had Th3Vans hardware, and the way Storj satellites worked let me make money with them, I’d set up the same thing.

Storj makes money reselling capacity. If they ever wanted to have less capacity to resell… they have a variety of ways to start enforcing policies they’ve already written. No new policies required

[RecklessD]

@arrogantrabbit

• Using VPN to bypass this rule was therefore frowned upon.
• But using multiple subnets from ISP is OK with Stroj?

A VPN gives you the ability to fake being in a different location. This may be why its “frowned” upon.

[Pentium100]

Also, a VPN increases latency.
By the way, how would Storj reliably detect that multiple nodes are running on the same hardware if the node operator did not reveal such information?

@Roxor

If I had Th3Vans hardware, and the way Storj satellites worked let me make money with them, I’d set up the same thing.

For me the limit is the number of IPs I can get without putting my server in some data center. I know a few ISPs that would probably be willing to give me a few IPs, but they do not serve my home.

[Roxor]

@Pentium100

…the limit is the number of IPs I can get…

+1 !

When I first tried Storj, I thought raw space would be my limitation: but that was pretty easy to overcome.

Then I thought Internet-speed would be the issue… but that also kinda fixed itself as providers offered faster plans.

Then it seemed CPU/memory to run nodes would be the problem. But modern CPUs are crazy-fast and even consumer motherboards started coming with high RAM limits.

So now I’m realizing /24 IPs were the true bottleneck all along…

[arrogantrabbit]

@RecklessD

A VPN gives you the ability to fake being in a different location.

But so does owning multiple routes. There is no difference. The /24 was empirical “most residential users have one ip”, so this rules out them and their neighbours (!), yielding geo diversity.

@Pentium100

By the way, how would Storj reliably detect that multiple nodes are running on the same hardware if the node operator did not reveal such information?

100% reliably — they can’t. But to get a high confidence level it’s pretty trivial by observing correlations in timing let alone downtimes during long period of time.

It does not matter if they run on the same hardware, only that they are correlated.

@Pentium100

Also, a VPN increases latency.

Insignificant. My vpn adds 3ms. My cable connection at home is 15-50ms. So fiber+vpn is way under cable connection.

@Pentium100

For me the limit is the number of IPs

Exactly. My point is — storj is seemingly ok with concentration when th3van does it but not when arrogantrabbit does it. And there is nothing in the policy that would explain why.

[alpharabbit]

@Roxor

So now I’m realizing /24 IPs were the true bottleneck all along…

There is no bottleneck at all. The problem is way too much nodes fighting for the peanuts.

[arrogantrabbit]

If there are 100 nodes total each gets 1% of all data. But if I launch another 100 nodes in my cellar — now I get 50% of all data.

Peanut size does not matter, allocation does.

[Roxor]

If someone had 100 IPs… they’d probably be filling 1TB every day (as a guess). BUT… how much used-space would they need to fill to even pay for those IPs: because they’re probably renting them. Maybe 200TB?

Cool to think about these things… even if you’ll never do them…