Announcing Community Satellite Pilot Program

Product Discussions
1

Hello Storj Community!

The final incomplete task from the v3 whitepaper is getting Satellites run by the community, and we’re ready to start that process!

Our end goal is to allow anyone to run their own Satellite if they are so inclined, for maintaining and managing their own data against the public storage node network. We want to make sure that community Satellites are run well and are good partners with our storage node community. We see this progressing in phases.

There are large complicated parts of the Satellite in terms of billing, payment, audit and repair, edge services etc., and we want to make sure that running these things is as easy of a process to do correctly as possible. Because we don’t want to bite off more than we can chew, we want to propose starting small off the bat, and only tackling a few things at a time.

Phase 1 - Unpaid community Satellites

These first community Satellites will not pay storage node operators. Storage node operators should only volunteer their storage nodes to this unpaid Community Satellite program if they want to help. This would be similar to those of you helping us with our QA Satellite (unpaid) or other testing projects. During this early phase, we expect that all of the data stored in nodes managed by community satellites should be test data only.

Phase 2 - Edge services

This phase will help Community Satellite operators set up the optional suite of edge services such as the hosted gateway, web UI file browser, and other advanced features.

Phase 3 - Paid Community Satellites

If all goes well in Phase 1, we will start working on improving the Satellite operator to Storage Node payment process. We will be iterative and we will attempt to improve the onboarding process. We will have much more to share here as we approach this. We also are looking to understand how to enable users to migrate/failover between satellites.

Phase 4 - Revenue generating Satellites

The final phase will be where Satellite operators can start collecting payment from their users as well.

Phase 1 Specifics

We have created a new Satellite list at this location: https://www.storj.io/unpaid-community-satellites, akin to the existing https://www.storj.io/dcs-satellites. We are going to start a test Community Satellite to add soon, but if you would like to join us and run your own Satellite as part of this program, we’d be happy to add you to this list.

It’s worth pointing out that we have a bunch of rough edges to dust off since we haven’t really pursued this heavily before. In particular, we intend to add a bunch of per-Satellite limiting (space usage, requests/sec, etc.) and management features to the storage node. We want storage node operators to be able to limit resource usage per Satellite or per Satellite trust list, and we don’t intend to add resource-consuming Satellites to this unpaid Community Satellite list until storage nodes have support for these management features.

Otherwise, we request that interested storage node operators who are volunteering for this unpaid pilot add this low-resource unpaid Community Satellite list to their trusted list on experimental storage nodes. You can do that by configuring your experimental storage node with the following config:

server.use-peer-ca-whitelist: false # becoming the default soon
storage2.trust.sources: https://www.storj.io/dcs-satellites,https://www.storj.io/unpaid-community-satellites

Calls to action:

  • As a storage node operator, please consider adding https://www.storj.io/unpaid-community-satellites to trusted satellite lists for any experimental storage nodes you have, instructions above (and you’ll want to leave https://www.storj.io/dcs-satellites as well.)
  • If you want to join us on the journey of running your own Satellite, please let us know (on this thread!) your Satellite’s hostname, DRPC port, and Node ID for inclusion in the unpaid-community-satellites list.

We’re most interested in feedback. Please let us know your thoughts.

18 Likes
2

What are the hardware requirements for running a satellite?

1 Like
3

It highly depends on the amount of users you expect, and it will also depend on the number of storage node operators who opt in to the initial phases of this program. We run Satellites on our development laptops regularly for integration tests, but of course, in those cases we don’t have the full network of 20k nodes. We will have to see if there are resource minimums for this program as we roll this out. For a small network of nodes, I imagine you could get away with a Raspberry Pi. The database (CockroachDB, and what it’s running on) is the main driver for performance.

5 Likes
4

Do satellites require large amounts of storage?

5

Please give a template to fill with dummy data as I am lost on hostname and DRPC port. Is this the same Node ID of my active storagenode ?

6

Do you think this is a good move?
I consider Storj an enterprise grade because the satellites are run by a competent group of people assuring this backend is in a good hands, is redundant, HA and geo-replicated, so no matter what, this service will always be available.
The same group of people who are running this software are also developing it, so in case of any issues they know their way and how to fix any potential problems.
Now you will introduce, I’m afraid, people with zero experience running anything HA to run something that will be branded I assume Storj as well, as some kind of resellers god knows with what kind of a business practices.
Can you imagine how bad of a PR can this cause? Someone running a satellite on a poorly connected, under powered and unreliable hardware, with no experience fixing any potential problems, no HA etc. and then loosing customer’s metadata?
I consider myself as someone working with HA (familiar and working with vPC, VRRP, BGP, CEPH, load balancers etc.) and running everything doubled or tripled just to make sure the things I’m running are available as much as possible (even though my current node uptimes would suggest to contrary), but I wouldn’t dare to run a satellite to risk I’ll loose customer’s metadata and thus all their data, unless I’m missing something and loosing a satellite won’t cause stored data to be lost.

3 Likes
7

Nice move, whole Bitcoin network works on top of something like “community nodes”.
What about uptime and bandwidth requirements?

1 Like
8

A couple of answers here:

It depends on your use case. Satellites require a bit of metadata to keep track of what storage nodes an object (or more accurately using Storj terms, a segment) is placed on. Satellites must only store this metadata per segment, so, if you have lots of data but just a few segments, no, not at all, the Satellite doesn’t use much space. Essentially, the Satellite stores just your file list (and what nodes the file list is on).

Yep, you can see examples on https://www.storj.io/dcs-satellites

We do! Consider this similar to Redhat Enterprise vs Fedora, or Wordpress.com vs Wordpress.org.

Storj DCS (and services offered on storj.io) will always be our highly available, SLA-backed cloud offering. We will be encouraging people to sign up for Storj DCS on our website. It’s how we make money!

What is available in github.com/storj/storj is not branded “Storj DCS”. Instead, it is branded “Storj OSP” (open source project). We firmly believe in the cause of decentralization, and we believe that letting others offer their own Storj OSP-powered services for themselves or others is the next step here.

We have been discussing what requirements a Satellite operator would have to meet to federate with Storj DCS, but rest assured they will be significantly higher than joining our community Satellite program. We remain just as eager as you to keep Storj DCS a high quality, high bar service.

Thanks! They will likely be more than running a storage node. Satellites that aren’t run well risk becoming “untrusted” by storage node operators. If the Satellite is unavailable, it could miss signed order reports from storage nodes, and thus not know how much to pay storage nodes. Earning and keeping trust with storage node operators is one of the major keys to having a well run Satellite, and to do that will require strong uptime.

4 Likes
9

Is there going to be sort of a trust system as well with this? Maybe a reputation system where if someone starts a satellite and someone invites a bunch of people onto it and then goes and stops running it all together what happens to all the peoples data?

2 Likes
10

Hello @jtolio,

I’m interested in joining the satellite pilot program, already looking into satellite installation.

Thanks a lot

Edit: I’m not able to find the docs to install a satellite, need some help.

11

Thank you for your hard work. This is an unexpected move.

However, I have a lot of concerns.

Creating a hurdle to entry:
There will be inexperienced people who just want to try it out without committing, and you can turn off your community satellite at any time without consequences. There should probably be some kind of barrier to entry, because anyone who wants to try it is making work for the Storj team.

You said you can run a satellite on your local PC for testing purposes. How about if the person who wants to run a satellite has to set it up locally first.

Vetting for the community satellite:
If someone wants to run a satellite, they are probably already running a storagenode themselves. How about if someone sets up a community satellite, he first has to use his own storagenode to store data. Then he can play with it and try to find people who want to use his community satellite. If they don’t, other storagenodes won’t even be bothered. After a few months, if the uptime is 100% and they can find people and prove to the community that they have customers and what storage volumes they expect, they can be approved by the community.

Before I approve a community satellite:
I want to get a lot of transparent information.
Who is operating the community satellite? Is it one person who might hit a tree and not respond for months until the server is shut down? And during that time, no payout is ever made?
Does the person know enough about networks and security?
What is their backup plan?
How many users do they want to reach in what time and how much storage do they expect.
What does a long-term business plan look like?
How much money is available in case customers don’t pay but still want to pay storagenodes?
Maybe something like an imprint of the person.

No payment?
What if they want to use other storage but don’t pay at the end of the month? People could exploit it and create a community satteliten to get PBs of storage for 1-2 months. If it is noticed that a community sattelite does not pay it has to be painstakingly removed again. On the storagenodes would then a lot of files lying around. Since the community sattelite performs the garbage collection, but it no longer exists to do so, the many files remain forever? Those exploiting it could just create a new community sattelite.

Price undercutting:
I’m also very worried that a community sattelite could eventually undercut the prices of the official sattelites, putting pressure on Storj to lower their prices. Maybe there should be some sort of fee for running a sattelite (fixed monthly cost + percentage of payments received). It can’t be that Storj develops the sattelite and a community sattelite only has the cost of the server it runs on, and if the community sattelite has a few paying customers it makes a profit.

Also: storage-node: Decomissioned satellites logs are not removed · Issue #4874 · storj/storj · GitHub is still open. If people start creating satellites and turn them off right away, everyone will get unexpected logs.

12

I want to get a lot of transparent information. Who is operating the community satellite?

Satellite operators need to be activelly transparent. It could be good to have a space in the forum dedicated to presentations.

For example, as I intend to join satellite operators, I can show some transparency myself.

Do I have the technical skills?
Yes, I’m a professional big data sysadmin from Spain, I deal with networking and security daily working for multinationals.

My strongest background is in development, more specifically in the videogame industry. Blasphemous was my biggest success while I did game development.

What is my motivation?
Since 2021 I’ve been focusing in obteining a DevOps profile. That is a big motivation for hosting, because doing it well gives a lot of credibility.

How will 100% uptime be achieved?
I will use a VPS for the satellite. It will have high availability if enabling it is possible, so uptime is 100% without doubt. My focus is in having a professional quality satellite.

13

I try to understand why Storj is creating a concurent service for free, that will compete in prices with it’s own service. And especialy now when it tryes to find a sustainable economics system. Storj team makes the software for comunity satellites and mantaines it, and they give it for free to anyone who wants to create a service that competes with Storj’s own service that pays the salaries for dev team…
What I’m missing? Is there a fee for running comunity satellites? This whould make sense somehow, but still…

2 Likes
14

Big customers want to operate their own satellites, and storagenodes want to partner with big customers. Obviously Storj deserves a fee.

5 Likes
15

I would say the opposite.
I personally know one Fortune 500 company that is, for over a year now, moving everything from their own datacenters to the cloud and it is for one major reason - to save headcounts. And I would say it is the norm now with the big companies. I can’t imagine such a company adding costs by hiring a bunch of people to take care of a satellite, a technology that would be very new to them, to develop procedures, DR plans, not mentioning convincing nodes to join them. Or to risk the employees would cause a data loss. Such company would also need some support contract with Storj, which would add costs as well. It is thus far easier for them just to start using the service as it is, or if capacity would be insufficient, asking Storj to scale up the existing ones.
The decision makers in these corporations would, in my opinion, take a huge risk to save very little by running their own satellites. And not to mention the anonymity they would loose by asking people to join their satellites.

2 Likes
16

A selling point of storj is precisely privacy and security. That is reinforced if companies spin up a satellite. Most companies are not 100% in the cloud, they use the hibrid cloud aproach.

This is even more true in Web3 companies, they would definitely host their own satellite.

Fortune 500 are generally more conservative, they will not be early adoptes but instead implement It when the technology is proven.

3 Likes
17

I have some concerns with this; maybe I don’t understand the tech behind it, but here it is:

  1. How much control will Storj have over the comunity sats? Can it set some limits over the prices? Can it impose other constraints? Or everyone is free to do how it pleases?
  2. What are the plans to combat bad actors?
    As I see it, the storj software is open source, so anyone can use the code and modify it, to run custom versions. One bad actor could create a satellite that sends all the customer files to a sigle data center and maybe even reconstruct the files, or use a fake encryption; the storagenodes whould be geting fake data just to appear that they store something, and so on. Like when FBI ran fake VPN services for cartels and terrorists, to get their data. How can Storj prevent that?
    This descentralised storage service is not trustless like blockchain technologies. You must trust the satellites operator for good behavior. And if he has nothing to loose, than it’s hard to trust him.
1 Like
18

But that would be more an issue for customers of that satellite, not really for SNO‘s. If they don’t get their payment, they can just leave the satellite and won’t loose anything except a little bit of HDD lifespan.

2 Likes
19

I would consider these community satellites as separate entities that each need to earn their own trust. Which makes a Storj hosted satellite list a little weird to me. Node operators should be able to individually vet community satellites and have an easy way to add and remove them.

This would require more tools to be made available to node operators:

  • A list of community satellites to select from
  • A reputation system where node operators could rate those satellites on several measures
  • Tools in the node software to add and remove satellites, including the required data clean up after removal.
  • Fixes to prevent errors in logs for nodes that no longer exist

I would normally be one of the first to sign up for this, but I don’t really know based on this post what I’m signing up for. What kind of satellites can we expect to be on this list. From the post it seems to be kind of a wild west. And if that’s the case, I really don’t recommend adding this list to an existing node. Especially if the above mentioned tools are not yet available. Can you please clarify what we can expect? And if it really is kind of a wild west playground, please advise SNOs to set up a separate node for these satellites, at least at this stage.

10 Likes
20

This is early so a lot will be built out to improve it over time.

I think node operators need some kind of filter to determine if any particular Sat can join or not. Earnings is one consideration. A threshold that if a Sat pays at or over a certain amount accept it, otherwise ignore it. (Big conversation around how that would work). There should also be some official nodes list from Storj, those that are rated high, and then new and poorly rated.

I will setup a node specifically to help with this and expand as needed. I may also setup a Sat for learning/assistance purposes.

6 Likes