Docker Image for Satellite

[martinr92]

Hi,

What is the recommended Docker Image for hosting a Satellite?
Found the following on DockerHub:
https://hub.docker.com/r/storjlabs/satellite
The storj-up is using the following:
img.dev.storj.io/storjup/storj

Just to be sure: I’m not talking about a simple test environment. I’m more interested in running a productive community satellite.

Thanks

[nerdatwork]

Announcing Community Satellite Pilot Program

Hello Storj Community! The final incomplete task from the v3 whitepaper is getting Satellites run by the community, and we’re ready to start that process! Our end goal is to allow anyone to run their own Satellite if they are so inclined, for maintaining and managing their own data against the public storage node network. We want to make sure that community Satellites are run well and are good partners with our storage node community. We see this progressing in phases. There are large complic…

[martinr92]

I know about this thread.
But couldn’t find the answer to my question there. Or did I miss something?

[Alexey]

There is no guide to run an own Community satellite, but you may make yourself familiar with the setup using two tools to run a local test network:

• Test network · storj/storj Wiki · GitHub
• GitHub - storj/up: Docker-compose files for running full Storj network locally

Then you may contact the team to add your satellite to the Community satellites list accordingly mentioned program for the satellite operators at Phase 1. We did not move further this phase yet.

There is no automatic rewarding, unless you figure out how to attract customers and implement your own integration with payment processors to accept payments from these customers and implement a rewarding of nodes joined to your satellite (you also need to attract Node Operators as well).
This is still actual:

Compensation for running a satellite

Hello @JMess035, Welcome to the forum! There is no guide how to run a Community satellite yet, but it’s open source software, so you can try to setup and run it. However, your satellite will not be a part of Storj DCS. You need to attract SNOs and customers, your satellite will not be advertised by default. Your compensation will be actually your profit from running such a business.

[martinr92]

@Alexey

There is no guide to run an own Community satellite

I know. Thats my main problem right now.
Reverse-engineering is a very hard work, so I need some help from the developers.

Maybe a few of my questions can be answered?

• What is the recommended database? storjup uses cockroach, the boostrap shell script uses postgresql. Since I have only experience with postgresql: Are there any storj-sattelite specific benefits to use cockroach instead? Any difference for mediainfo and master database?
• The storjup splits the satellite into 3 main parts: API, Core and Admin. Can you describe their main tasks for each in 1-2 sentences? Is it required, to split them up into separate container executions or can they run in one instance?
• It looks like there are dependencies between the 3 parts: Core and Admin needs the API. Is there also a dependency between the Core and Admin instance?
• STORJ_ORDERS_ENCRYPTION_KEYS: What is this encryption keys used for? Is there any generator? If not, what is the format: Alpha-Decimal, Hex, or something else?
• What is the preferred docker image to use? I know, that there exists two of them for more or less the same thing. But why? Which one is Storj.io using? The images on img.dev.storj.io sound more like for a testing / quick start. Is the registry on dockerhub the “serious” one?

What I mean with the original post: I don’t run just simply storjup or storj-sim. I need to understand the parts and its details to run them in a proper way. Otherwise, I can never move them properly into a K8s cluster environment.
After this works, I will join the Community Satellite program (to hopefully find some more nodes to do a wider test).

A few answers would help me and also other people to operate successfully a satellite.

[Alexey]

@martinr92

Are there any storj-sattelite specific benefits to use cockroach instead?

it’s optimized to use a cockroach database, it works better in a distributed mode and allows a horizontal scaling. In the one node setup PostgreSQL can be more convenient, however it’s up on you. For the production load the cockroach database is better.

@martinr92

What is the preferred docker image to use?

I think you may use any of them or build your own using a Makefile from the repository. Our CI regularly publish images to the docker registry, so you may use them too. The only difficulty is to use a matching versions of all components, thus images from img.dev.storj.io are more consistent, but they more for the testing/develop purpose.

[elek]

@martinr92

• What is the recommended database?

I (personally) would recommend using Cockroach as they are used in production almost everywhere. Cockroach is postgresql compatible, so easy to switch. Also has good web UI to spot slow SQL queries.

We have some cockroach specific optimization (mainly using AS OF SYSTEMTIME) and some codepath maybe slower with Postgres (where we had cockroach only optimization).

But both are tested with unit tests.

Note: we use Postgres for some of the queue-s, as Cockroach hasn’t the best performance for temporary queue-like data. But you can start with cockroach as you can switch any time later if you have higher loads.

@martinr92

• The storjup splits the satellite into 3 main parts: API, Core and Admin. Can you describe their main tasks for each in 1-2 sentences?

Check the following diagrams:

• https://raw.githubusercontent.com/storj/illustrated/main/docs/satellite-api.svg
• https://raw.githubusercontent.com/storj/illustrated/main/docs/core.svg

TLDR:

• API is the main endpoint. It has the UI and all the required RPC endpoints. You can upload/download with the API
• Core is for durability (audit process)
• Admin is just managing support requests (like updating limits, modifying projects …).
• You also need GC (to delete old files from storagenodes) and repair workers (+ rangedloop). Some of these can be run as part of core/other services but we run them as separated microservices.

@martinr92

• It looks like there are dependencies between the 3 parts: Core and Admin needs the API. Is there also a dependency between the Core and Admin instance?

There are no strict dependencies, AFAIK. Each of them depends on the database. But I can be wrong here (for example if some parts of the Admin uses the API, but I would be surprised)

@martinr92

• STORJ_ORDERS_ENCRYPTION_KEYS: What is this encryption keys used for? Is there any generator? If not, what is the format: Alpha-Decimal, Hex, or something else?

This is a key=value pair (to make it easier to use multiple keys or deprecate old one.

And they are given as hex. Use the same format what you can find on storj-up, but the second part (after =) should be a fully random number.

0100000000000000=0100000000000000000000000000000000000000000000000000000000000000

I am not aware of any generators, but I agree that we need one (if doesn’t exist)

@martinr92

• What is the preferred docker image to use? I know, that there exists two of them for more or less the same thing. But why? Which one is Storj.io using? The images on img.dev.storj.io sound more like for a testing / quick start. Is the registry on dockerhub the “serious” one?

We have two serios one in dockerhub:

• One is the result of our satellite builds: Docker
• We use this one in production, but we add our branding HTML + CSS set on top of these images (but that’s private, as it includes our branding stuff)

You can use the public images from Docker hub. Should work well.

But I don’t see any serious risk in using the img.dev.storj.io images. We use them for integration testing, so they should work. But they are not released with the release process, so usually they are not the latest one…

[elek]

@Alexey

here is no guide to run an own Community satellite,

One more comment:

Personally I would prefer to make it easier to run Satellites (either for a private storage cluster or for any Community Satellites).

Therefore I started a video series to explain what you need to run your own Satellites. (But for sake of the simplicity I don’t use containers there.)

Here is the link:

Also: I started this only recently, so I have only 2 videos, yet. Especially because I feel that I need to explain a few fundamentals to make it clear.

Next video in the playlist will cover the structure of Access Grants (at least according to the plan), and after that I am planning to continue with Satellite Admin.

And your questions gave me some additional motivation to continue this one, so I will prepare the next one soon.

Also: please let me know if you have additional questions, or if you have topics which should be covered by any new video.

(I hope it’s clear that I am writing here my personal opinions, these are not official docs, but as we are talking about technical details, there shouldn’t be any big differences…)

[martinr92]

@elek Thank you so much for all the details and information.
I will also check out the youtube videos. Thats great and helps a lot understanding the whole satellite concept.