[Tutorial] Run your own satellite (part 12) - Satellite repair

Developer Discussions
,
1

What Is Repair Service?

Repair service pull injured segments out of the jobq service, attempts to fix them, and then decides whether to permanently remove the entry (success) or put it back (fail) to be retried later.

How Does Repair Works?

This section is AI-assisted (Claude) and may not 100% accurate, still good enough to get the general idea.

Phase 1 — Piece Classification (piecesCheck)

Before touching any data, the repairer asks the overlay (node DB) about every node holding a piece of the segment and classifies them into buckets:

  • Healthy — online, in the right placement, not clumped
  • Retrievable — can be downloaded (superset of Healthy; includes unhealthy-but-online pieces)
  • Missing — node is gone or disqualified
  • Clumped — multiple pieces on the same /24 subnet (declumping feature)
  • OutOfPlacement — piece is on a node violating geo-placement rules
  • ForcingRepair — clumped or out-of-placement pieces that are causing the repair trigger
  • Exiting — node is gracefully exiting

Phase 2 — Early Exit Decisions

The repairer bails out early in several cases without touching any data:

  • Irreparable: Retrievable.Count() < RequiredShares — not enough pieces exist anywhere to reconstruct; logged as irreparable, left in the queue for later retry.
  • Already healthy enough: Healthy.Count() > RepairShares — if only ForcingRepair pieces exist, it may just drop those pieces from the metadata without any download/upload at all.
  • Segment deleted or expired — immediate clean exit.

Phase 3 — Download (GET_REPAIR)

The satellite creates GET_REPAIR order limits which simultaneously contacts multiple storage nodes and downloads pieces in parallel. A few details worth noting:

  • It uses an erasure coding (Reed-Solomon) reader — it needs at least RequiredShares pieces to reconstruct the full segment, but may attempt more in parallel.
  • There’s a long-tail mechanism (DownloadLongTail config) — it can issue extra concurrent download requests beyond the minimum needed, to avoid being blocked by slow nodes.
  • After the download, it checks if the segment was modified during repair (race condition guard). If it was altered or deleted mid-repair, the job is silently dropped. Eg:
    A user deletes the file — segment is gone entirely → drop the job.
    A user overwrites/re-uploads the file — the segment’s piece list changes (new nodes, new piece numbers) → drop the job).

Phase 4 — Reconstruction to Tempfile

The decoded segment is written to a temporary file on disk (unless --repairer.in-memory-repair=true). However, note that on modern debian/ubuntu, tmpDir is /tmp is tmpfs which is already on RAM.

Phase 5 — Upload to New Nodes (PUT_REPAIR)

The repairer asks the overlay for new nodes (not already holding a piece), creates PUT_REPAIR order limits which re-encodes and uploads new pieces to those new nodes in parallel.

Phase 6 — Metadata Update

After upload, the satellite updates the segment’s piece table in metabase:

  • Adds the newly uploaded pieces (new node IDs)
  • Removes unhealthy/replaced pieces, failed pieces, and re-uploaded ones
  • Sets RepairedAt timestamp

Phase 7 — Reputation Update (optional)

If --repairer.reputation-update-enabled=true, nodes that served pieces successfully during the GET phase get audit successes recorded, nodes that failed get audit failures. Disabled by default.

Minimum repair command

./satellite-modular repair \
--identity-dir ./id/satellite/ \
--server.extensions.revocation false \
--database-options.url "cockroach://my_role:123456@rendezvous.example.com:26257/my_database?sslmode=verify-full&sslrootcert=ca.crt" \
--metainfo.database-url "cockroach://my_role:123456@rendezvous.example.com:26257/my_metainfo_database?sslmode=verify-full&sslrootcert=ca.crt" \
--orders.encryption-keys 88a2ab18c619535b=e9bbe3c658d595d5c40dba994152abc07109cdd4425fec627194bdec32c43333 # not a real key
--jobq.tls.use-peer-ca-whitelist false \
--jobq.tls.extensions.revocation false \
--jobq.server-node-url "12kK5Y15fU1b5YADDQiBj2Dd5WsF1LARVWHpW6nXRcusV3wiRZD@127.0.0.1:15781"

--jobq.server-node-url is the address of your jobq plus base58 string, to find that out, use this command:

./identity certificate-authority id --ca.cert-path=./id/jobq/ca.cert
# or ./id/satellite/ca.cert if you use single key for all services

The is a flag --repairer.max-repair (default: 5), for segment repairs run concurrently within a single process, and you can also run multiple ./satellite-modular repair processes simultaneously against the same job queue.

Checkout more config from ./satellite-modular repair -h | grep repair.

And see you on part 13.

[Tutorial] How to run your own satellite (part 1)
[Tutorial] Run your own satellite (part 10) - Introducing mud and ranged-loop