[Tutorial] Run your own satellite (part 3) - Generate certificates

[kocoten1992]

Disclaimer, this is half a tutorial, half a journal, I create this guide as I go. But rest asure, what I did was meant to run a reliable satellite, for production environment not a toy.

Not sure if you remember, in the early day, as a SNO, the flow when you create a new node have this very first step, fill in your email to get a authorize token, but nowaday:

Step 4. Create an Identity - Storj Docs

Storage Node - Storj Docs

You won’t find that step anymore, you wonder why?

---

As far as I understand, currently, there are (only?) 2 core services in satellite require certificate: satellite and jobq. Jobq originally was a built-in service within satellite, but later it was rewrite to be a dedicate service run outside of satellite. Why? Mostly because of performance. What does it do?

Jobq is a microservice that holds and prioritizes the list of storage segments waiting to be repaired. When the satellite detects that a segment has lost too many pieces (e.g., because storage nodes went offline), it pushes a repair job into jobq. Repair workers then pop jobs from the queue, always getting the most critical segments first — those with the lowest health score. Jobs that have already been attempted recently go into a separate retry queue and are held back until enough time has passed.
-Claude

The point is this: since it a separated, standalone service and could communicate with satellite over the internet, 2 issues arrive:

1. Does communicate between jobq and satellite secure? Yes it is.
2. Could a random satellite use your jobq? Now we get the crux of the problem.

---

Let me first show you my script for generating certificate:

#!/bin/bash

if [ "$(id -u)" -ne 0 ]; then
    echo -e "\033[1;31mERROR: must be running as root, exit code 1\033[0m" >&2
    exit 1
fi

if [ "$SCRIPT_DIR" = "" ]; then
    SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
fi

if [ "$ID_DIR" = "" ]; then
    ID_DIR=$SCRIPT_DIR/id
fi

OS_ID=$(grep ^ID= /etc/os-release | cut -d= -f2 | tr -d '"')

if [[ ! -f "$SCRIPT_DIR/identity" ]]; then
    echo "identity binary file not exist in $SCRIPT_DIR, exit code 2"
    exit 2
fi

# 1. GEN ROOT CA
if [[ ! -d "$SCRIPT_DIR/root-ca" ]]; then
    echo "-- GENERATING ROOT CA --"
    mkdir -p "$SCRIPT_DIR/root-ca"
    $SCRIPT_DIR/identity certificate-authority create \
        --ca.cert-path="$SCRIPT_DIR/root-ca/root-ca.cert" \
        --ca.key-path="$SCRIPT_DIR/root-ca/root-ca.key"
else
    echo "-- ROOT CA ALREADY GENERATED -- SKIPPING --"
fi

# 2. GEN SATELLITE ID
if [[ ! -d "$ID_DIR/satellite" ]]; then
    echo "-- GENERATING SAT CERT --"
    $SCRIPT_DIR/identity create satellite \
        --config-dir="$SCRIPT_DIR" \
        --identity-dir="$ID_DIR" \
        --parent-cert-path="$SCRIPT_DIR/root-ca/root-ca.cert" \
        --parent-key-path="$SCRIPT_DIR/root-ca/root-ca.key"
else
    echo '-- SATELLITE ID DIR ALREADY GENERATED -- SKIPPING --'
fi

# 3. GEN JOBQ ID
if [[ ! -d "$ID_DIR/jobq" ]]; then
    echo "-- GENERATING JOBQ CERT --"
    $SCRIPT_DIR/identity create jobq \
        --config-dir="$SCRIPT_DIR" \
        --identity-dir="$ID_DIR" \
        --parent-cert-path="$SCRIPT_DIR/root-ca/root-ca.cert" \
        --parent-key-path="$SCRIPT_DIR/root-ca/root-ca.key"
else
    echo '-- JOBQ ID DIR ALREADY GENERATED -- SKIPPING --'
fi

# 4. GEN AUDITOR ID
if [[ ! -d "$ID_DIR/auditor" ]]; then
    echo "-- GENERATING AUDITOR CERT --"
    $SCRIPT_DIR/identity create auditor \
        --config-dir="$SCRIPT_DIR" \
        --identity-dir="$ID_DIR" \
        --parent-cert-path="$SCRIPT_DIR/root-ca/root-ca.cert" \
        --parent-key-path="$SCRIPT_DIR/root-ca/root-ca.key"
else
    echo '-- AUDITOR ID DIR ALREADY GENERATED -- SKIPPING --'
fi

The important part here is your root ca, if you create certificate this way, it create a chain of certificates (you CAN create satellite identity without a root ca).

---

So why is that SNO don’t need authorize token anymore?

I think because of this commit storagenode/peer: don't require CA whitelist any longer · storj/storj@1d63395 · GitHub, but I’m a bit confused, in that commit, they say they gonna eventually disable it on satellite entirely.

But then how do it stop a random satellite connect to a random jobq if jobq expose over the internet? As I understand, ca-whitelist is last battalion to prevent this situation – unless of course, jobq must be on the same infrastructure with satellite, no expose to the internet, but then we are locking ourself to a single vendor with a point of failure. Am I mis-understand something? Any developer explanation would be really appreciated.

For now my script is working as intended, see you on part 4.

Hi @Alexey, could you help me merge part 2 and 1, I’ll edit it after you merge, thank you very much!

---

IMPORTANT NOTICE

P/s: After writing tutorial after 10 parts, I’ve a new insight, maybe instead of using multiple keys, you can just use a single key pair (still need root-ca, so that 2) for all services, if you see future guide only use satellite service instead of dedicate named keys: api, admin, core, auditor, … You know why. See [Tutorial] Run your own satellite (part 3) - Generate certificates - #8 by kocoten1992

[Alexey]

I can merge threads but not posts, especially across threads. So, you would need to use a copy-paste of your posts to the main wiki page. I recommend to click Edit in the source post and copy the content then paste it to the destination post.
So, do you still want that I merge threads? Your Part 2 and Part 3 posts will appear as comments in the thread Part 1, original threads will be deleted, if there will be no other posts.
I might be able to merge the messages in the thread later, not sure about that, usually it works automatically as a hint for several consecutive messages from the same person.

However, you might also consider to make them like this:

[Tutorial] How to run your own satellite (part 1)

Hi there, I’ve spend a few days playing with satellite, it’s nowhere near ready but I think, while I’m at it if I share my experience, the more people can host satellite the better for storj ecosystem. One thing I notice, official storj implementation is not a fixed thing, it subject to change, so you need to stay on top of it, I learn a few things about storj direction when reading comments from Gerrit (storj’s review platform), if you read it - you are more prepared for the future. Another …

[Tutorial] Run your own satellite (part 2) - Set up cockroachdb

Hi guys, This is the second step in How to run your own satellite - first step is here. Currently at this time, the only distributed database supported are google spanner and cockroachdb (there are likely TiDB will be there in the future). At this point, I can only recommed cockroachdb. Manually set up cockroachdb cluster is a great pain, though I still recommend you do it once to understand how it work, but not a second time. I’ve write a script for you, it automate the entire process: Gen…

[Tutorial] Run your own satellite (part 3) - Generate certificates

Disclaimer, this is half a tutorial, half a journal, I create this guide as I go. But rest asure, what I did was meant to run a reliable satellite, for production environment not a toy. Not sure if you remember, in the early day, as a SNO, the flow when you create a new node have this very first step, fill in your email to get a authorize token, but nowaday: You won’t find that step anymore, you wonder why? As far as I understand, currently, there are (only?) 2 core services in satellite r…

[Alexey]

@kocoten1992

Am I mis-understand something?

Yes. We removed only requirement to sign the certificate with a special signing token, but you still need to whitelist a satellite’s certificate for jobq. Also, I’m not sure that this exact case would be considered for removing a requirement to use a signed certificate.

Copilot Summary

I’ll help you understand the authentication and authorization between a satellite and jobq. Let me search the Storj repository for information about this interaction.Let me search for more specific information about the authentication and authorization mechanisms, particularly around TLS options and peer CA whitelisting.Based on my analysis of the Storj codebase, here’s how authentication and authorization work between a satellite and jobq:

Authentication and Authorization Flow

1. TLS Certificate-Based Authentication

The communication between satellite and jobq is secured using TLS (Transport Layer Security) with X.509 certificates:

Server Side (jobq):

• The jobq server loads its identity from certificate files during startup (satellite.NewJobq())
• TLS options are created using tlsopts.NewOptions() which configures the server with its identity certificate and private key
• A revocation database is opened to check for revoked certificates

Client Side (satellite services):

• Satellite services (like rangedloop) also load their identity from certificates
• They create TLS options using the same tlsopts.NewOptions() function
• They use a DRPC dialer with those TLS options to connect to jobq

2. Peer CA Whitelist (Authorization)

Both server and client can enforce a peer CA whitelist:

// ApplyPeerCAWhitelist applies the peer CA whitelist to the TLS options
func ApplyPeerCAWhitelist(usePeerCAWhitelist bool, peerCAWhitelistPath string, tlsOpts *tlsopts.Options) (err error) {
    if usePeerCAWhitelist {
        whitelist := []byte(tlsopts.DefaultPeerCAWhitelist)
        if peerCAWhitelistPath != "" {
            whitelist, err = os.ReadFile(peerCAWhitelistPath)
        }
        tlsOpts.PeerCAWhitelist, err = pkcrypto.CertsFromPEM(whitelist)
        tlsOpts.VerificationFuncs.ServerAdd(peertls.VerifyCAWhitelist(tlsOpts.PeerCAWhitelist))
    }
    return nil
}

This mechanism:

• Loads a whitelist of authorized CAs (either default or from a custom file)
• Adds a verification function to validate that incoming connections come from whitelisted CAs
• Only allows connections from peers whose certificates were signed by whitelisted CAs

3. Connection Flow

1. Server startup (jobq process):

   • Loads identity certificate
   • Creates TLS options with CA whitelist
   • Creates a DRPC server on the configured listen address
   • Registers the JobqEndpoint with DRPC

2. Client connection (satellite rangedloop, balancer, etc.):

   • Loads identity certificate
   • Creates TLS options
   • Creates a DRPC dialer
   • Dials the jobq server’s NodeURL using the dialer
   • TLS handshake occurs:
     • Both sides present their certificates
     • Both sides verify each other’s certificate chain
     • If CA whitelist is enabled, peer certificates are validated against it

4. Communication Methods

Once authenticated, the client accesses jobq through the Client interface:

type Client struct {
    client pb.DRPCJobQueueClient  // wrapped DRPC client
    // ...
}

All RPC methods like Push(), Pop(), Inspect(), etc. are called through this authenticated connection.

Key Points

• No role-based access control: There’s no per-operation authorization. Once a peer is authenticated via TLS certificate, it has full access to all jobq operations.
• Certificate identity verification: The jobq server uses the identity embedded in the client’s certificate during the TLS handshake to authenticate the connection.
• Mutual TLS: Both client and server authenticate each other; it’s not one-way.
• Revocation support: The system supports certificate revocation checks via the revocation database.

However, we can remove the signing requirement even for this case, but we will need to add their self-signed certificates directly to the whitelist.
So, no need to modify a jobq’s config for any new satellite with a signed certificate, for unsigned - you need to modify a config of jobq to allow this satellite.
In case of storagenode the signature was just a gate on an early stages of the network, now it’s not needed.

[kocoten1992]

@Alexey

However, we can remove the signing requirement even for this case, but we will need to add their self-signed certificates directly to the whitelist.
So, no need to modify a jobq’s config for any new satellite with a signed certificate, for unsigned - you need to modify a config of jobq to allow this satellite.

I see, if that the case, then there is no need for root-ca anymore? I’ll adjust my script if this feature is going to be built!

[Alexey]

At the moment it’s used, so no need to change anything yet.

[kocoten1992]

Okay, I just think about it, not sure if it true, I feel a slight push into microservice, so right now there are 2 services (sat and jobq), but after a while, if it become say 5 services.

Then each service communicate with each other, then we need to add 4 self signed cert to each service, if it become 10, then would have to add 9 to each service, this wouldn’t scale is it?

[Alexey]

The satellite consist of several micro services, which will be run separately, you will figure that out eventually, or you can take a look on GitHub - storj/up: Docker-compose files for running full Storj network locally · GitHub

@kocoten1992

if it become 10, then would have to add 9 to each service, this wouldn’t scale is it?

perhaps, if jobq would serve several satellites, but I believe it’s configured to run one instance per satellite, so scaling is not a problem.

[kocoten1992]

Hi @Alexey,

I thought more about this, is there any advantage for having more than 1 key pair? I mean why can’t I use 1 key pair for all services: api, jobq, auditor, repair, etc… how do you guy do in production?

[Alexey]

I don’t have prod configuration information, it’s internal.
But you may have more than 1 key pair for gating or distributing services, for example you may allow the Community to participate and host some of the services for you.
I asked AI, and it has some ideas: Google Search, why 1 keys pair is a bad idea.

[kocoten1992]

@Alexey

you may allow the Community to participate and host some of the services for you.

This is the real advantage. For other thing, AI just spouting random nonsense:

Zero-Downtime Rotation: Active and legacy key pairs can be kept in the configuration simultaneously so existing storage node communication isn't disrupted while rolling out a new master certificate.

There are technique to rotate key even root-ca without downtime, I will write about it in the future, for anyone want it now, it called certificate bundling (copy and paste several cert together - both old and new when transition).

So, I’ll update the guide, to use just 1 key, if you need more for any purpose, you can always use different keys later.

[Alexey]

I more concerned about the difficulty to scale and the possibility to gain a high level access to all components with leaking only one key.

[kocoten1992]

I think even leaking just 1 key is already game over, but I’ll keep the old instruction for anyone want multiple keys.