Delegate Traffic based on node type

Hi all,

I’m new here so forgive me if this is not a new suggestion but thought I would throw it out there.
Quick background, Developer by trade with a bit of operations thrown in.

No doubt, we have a lot of different people and nodes connected. From hobbyists running a node on their laptop, to Homelab types and many in between. This effects what hardware and network each user has. Some with 500GB’s and an DSL/Cable connection. Some with 70TB’s and a gigabit+ connection with again, a large mixture of hardware and network capabilities. Some people have a limited broadband pipe with sub 20mbit but a lot of storage and would do well hosting data that doesn’t require regular Egress, while others can provide high bandwidth, low latency data retrieval.

As a developer, I use AWS S3 for high availability, but at the expense of higher storage costs. If I need cheap bulk storage, Backblaze B2. The key difference between the two is performance and cost. S3 is great for performance, B2 for cold storage. I would recommend, if you want to really get the interest of Devs/Ops types, the ability to choose between storage nodes and bandwidth nodes would be great.

Nodes with low bandwidth would charge less for storage but when egress is required, charge more. Such as restoring from backup. While high bandwidth nodes charge more for storage, less for egress, but they make up the cost due to receiving high priority traffic.

Implementing such a system would make Tardigrade more viable for customer facing systems while also keeping the idea of cheap distributed storage alive. Give the customers what they need while tailoring the backend to support the wide range of infrastructure available.

4 Likes

Hey @Cyber-Hoarder welcome and thank you for such a thoughtful first post :slight_smile:

tagging @brandon our product manager to hop into the discussion!

1 Like

ack, just remembered he may be on vacation at the moment, during te holiday. OP if you also wouldnt mind, it would be really great if you can submit this idea in the idea portal, which is ideas.storj.io

Hi CyberHoarder,

Welcome to the Storj Community! Great thoughts. The network is architected to naturally play into these sorts of scenarios.

My thesis has been that by combining B2 pricing with S3-level of availability, we will be able to efficiently tackle both markets/use-cases simultaneously. I believe that an advantage of Tardigrade is minimal, and simple pricing - ie there is no need to spend days sifting through various licensing schema – as you will be paying only for what you use (in terms of bandwidth and storage).

Compared to AWS/Google Cloud/Azure storage offerings. we are half the cost half the cost, 20% more performant, globally vs regionally available, open source, and encrypted. There is also no need to replicate data across availability zones to achieve redundancy or global availability (making it even cheaper than the Big Guys).

If you want infrequently-accessed storage, you will not be paying for unused bandwidth, and pricing/feature set is highly competitive with Backblaze and lower cost alternatives.

Additionally, it is important to note that B2 and others have high egress bandwidth charges. Some of these providers claim “unlimited” egress which really isn’t unlimited. You can only transfer the data out one time, and if you do it more than that they will just block your account. With Storj you can download the file as many times as you would like.

That being said, we do also have some notion of “storage classes” on our roadmap. Today, the idea is to leverage Reed Solomon (in conjunction with Audit and Repair processes) to optimize for performance for S3-like use case, while still having SLA-driven assurance of data durability. In the future, we hope to leverage node selection to tackle compliant storage (HIPPA, GDPR, SOCs, etc.). Node selection will also play a role in CDN-type use cases, and may make it even more efficient/economical to select for cold-storage.

5 Likes

@Cyber-Hoarder you make some great points. @keleffew response is spot on with our strategy for how we believe we can make Tardigrade more viable for all sorts of customers and use cases.

3 Likes

As this part is really important for business development in the EU region and Germany in particular is there some rough estimate when such a feature could become available?

I as SNO wold even egree to sign such agreement, if it realy give me more oportunities. System build this way that even if i want to brake it i cant.

Maybe one feasible solution would be to spin up a satellite that allows only SNOs with an IP from the EU region. This however could be circumvented by using 3rd party VPN.
I don’t know if there is a way to make 100% sure the data will not leave the EU-region. Probably not.

no need to regional restriction, just SNO that signed agreement and all other. when you sign agreement, you have to identify yourself with document, this will be chackpoint.

How does this help if the customer is not permitted to store data on Tardigrade if his data gets stored outside of the EU/Germany?

if people indetiry themself and indetify location, then storj know that thay can send this data to thime nodes an this is inside EU, if you need hold data insode German it not secure at all, one good blackout will chage data to not accessable.

Ok, I understand.
But location of node owner is not necessarily location of the node.
E.G. I could claim I am in Germany, but run a node in USA.
So I think some kind of geoblocking is mandatory and would be requested by customers, e.g. public or health sector.
But I like your idea too, maybe in config.yaml there could be an additional entry a SNO must state which regulations he is complying.

This is true. But total blackout in Germany is really very seldom. In fact I have never experienced one in my life. And of course, a customer who selects such an options knows normally should be aware of this specific risk.

you cant make it just like config, if you you need realy signd document, then you will get some additional key or something that will go to config.

in EU there is some sort of agreement about border free of work and lot of other things…

I don’t know. But I don’t think it is feasible to make written contracts with ca. 1500 nodes in Germany alone. The GDPR allows “other legally binding instruments” so there might be other ways like suggested: Declaration of compliance in config.yaml + geoblocking.

you cant write to config, that if you put yes here, you will need to folow 50+ rows of requaerments and all thish requaerment shold be writen there. or something like that.

No, but you can put the contract and the requirements on the Storj website to read and make the SNO confirm the contract in config.yaml. Something like sno.confirms.gdpr.compliance: true/false. And make it that the node won’t start if this field is not set.

I mean if you want, you could sign the contract on the page with your node id, receive a confirmation token for this id and put that in config.yaml as confirmation of gdpr compliance.

I guess there are many ways how to handle that digitally.