[Tutorial] Run your own satellite (part 4) - Building storj binaries

Developer Discussions
,
1

In StorJ culture: each directory in cmd can be built into an executable binary. For our purpose, we only need to care about 3 main server side projects: storj/storj, storj/gateway-st, storj/edge (also called gateway-mt – multi-tenant).

Q: What the different between single and multi tenant?

First, let’s establish some jargon:

gateway: a bridge between you and the network, allowing you using HTTP to interact with your data (instead of using uplink library to interact directly with the network).

tenant mean: end user (not satellite).

Q: What should I use? gateway-st or edge?

gateway-st: can connect from 1 user to 1 satellite.

edge: can connect from multiple users to multiple satellites.

Personally, I don’t see any point in using single tenant, unless of course, you are the only user in your own deployment, I’m not going to focus on that.

Turn out, single tenant is extremely important, because data is decrypted right on gateway, so it can read anything, multi-tenant mean you have to trust the gateway not to read your key and your data.

To build all the binary, I’ve a script for you, put this script under the root dir of storj/storj and storj/edge, it will loop through ./cmd/ and build all binaries it can find and place it under ./bin/

#!/usr/bin/env bash
set -euo pipefail

ROOT="./cmd"
BIN_DIR="./bin"

mkdir -p "$BIN_DIR"

find "$ROOT" -type d | while read -r dir; do
  if find "$dir" -maxdepth 1 -type f -name '*.go' | grep -q . &&
     grep -q '^package main$' "$dir"/*.go 2>/dev/null; then

    name="$(basename "$dir")"

    echo "Building $name from $dir"
    go build -trimpath -buildvcs=false -ldflags="-s -w" -o "$BIN_DIR/$name" "./${dir#./}"
  fi
done

# for gateway-st - simply go to it root dir and run
go build -trimpath -buildvcs=false -ldflags="-s -w" -o gateway-st .

List of binaries as of today:

# storj/storj
certificates                   okprog
connect-test                   om-license
convert-node-id                piecestore-benchmark
crashcollect                   placement-test
diskstat                       satellite
failprog                       segment-verify
filewalker-benchmark           spanner-key-parser
identity                       storagenode
is-valid-sj1-blob              storagenode-benchmark
jobq                           storagenode-updater
jobqtool                       storj-admin
metabase-listing-performance   storj-sim
metabase-minimize-listing-csv  tag-signer
metabase-verify                uplink
metric-receiver                verify-graceful-exit-receipt
migrate-encryption-master-key  versioncontrol
migrate-public-ids             write-hashtbl
multinode

# storj/edge
authservice        certmagic-admin  linksharing  simplegateway
authservice-admin  gateway-mt       mcp-server

# storj/gateway-st
gateway-st

Ideally, you put that in a jenkins job, but I’m not going to show you how to do it here (unless of course if the demand is high, please comment below).

Updated

See [Tutorial] Run your own satellite (part 4) - Building storj binaries - #13 by kocoten1992

See you in part 5.

2

Your first post was converted to wiki. Add content there. Why do you keep spamming new ones, a teaspoon at a time?

3

I plan to create a commercial sate in the future. I think that a movement - a lot of satellite - is better than I’m alone, trying to inspire other to host satellite. A new post will have some dopamine kick to refresh motivation. An old post is like a library, only the determined would go there.

4

So you are using storj forum as a marketing platform to send spam for the benefit of the future competing company. Amazing.

We don’t want dopamine junkies running satellites. We only want determined ones to do so. You are solving a wrong problem.

Content of your posts is not even enough for small drive-by comment, alone whole top level post. Stop this. Storj forum is not your personal Twitter account.

5

Storj is providing and maintaining all needed parts as docker images. Why not just using them? :thinking:

6

Oh… let me check that out first and re-adjust this :zipper_mouth_face:

7

Also there is a

8

Hi @alpharabbit, I research a bit for this route, it might be viable. TBH, I’m not a fan of k8s in the early day since it focus solely on orchestrate on a single cloud vendor, orchestration between multi-cloud was not officially supported, think you need to hack it via labels, it create a vendor lock situation, that why I’m avoiding it.

But it seem community have some project that tackle this exact problem, karmada for example, so I might need to revisit this route in the future. For now, both for educational purpose and for understand closely what each component do in storj, let do it by hand - still production ready I assure you. I will research about multi cloud with k8s and write a guide for it later.

9

How to use storj-up with latest satellite version? @Alexey I am missing jobq there… :thinking:

10

You may build specific versions based on a gerrit change, or on GitHub ref.

storj-up build remote --help

or point to your local binaries: GitHub - storj/up: Docker-compose files for running full Storj network locally · GitHub

11

I stand corrected about gateway-st, gateway can read your key and your data, so it quite big ask to trust a random gateway, fixed.

12

Yes, only Gateway-MT doesn’t require trust, because it also works with an auth service, where your access grant is stored encrypted by your access key. So the satellite/edge operator still doesn’t have access to content, unless you share your access key with them. If your S3 creds are public, like in linkshare, then anyone who have your linkshare URL can download the available data.

13

Update:

The old way of build satellite ./cmd/satellite still work, but should be replace with this:

go build -trimpath -buildvcs=false -ldflags="-s -w" -o bin/satellite-modular ./satellite/satellite

I can only guess the reason: some of the service in satellite use sort of cronjob, the keyword to search for in source code is sync2.NewCycle, it blur the line between developer and traditional operator.

If you run the old binary ./cmd/satellite, and if there are more than 1 instance of satellite running at the same time, both satellites will trigger sync2.NewCycle, some trigger are harmless but some might cause trouble, to have more fine grained control - the best way forward should be using satellite-modular.

Thanks @elek for pointing this out!

14

It’s independent problem. Both old and new (modular) binaries should have only one “core” services running. That supposed to be a singleton. Yes, there are multiple batch processes, and only one is expected to be running.

But from api satellite api you can run as much as you need. That’s the microservice which receives the RPC calls…