[Tutorial] Run your own satellite (part 22) - Monitoring

Developer Discussions
,
1

Today, we will set up our first monitoring, the objective is simple: if log split out ERROR, your phone should lit up.

In monitoring, we have push/pull model: logs can be pushed to log server, or log server will scrape from endpoints. Each have it pro and con, for my case, I’ll use push.

Log/metric/trace

In my last guide [Tutorial] Run your own satellite (part 20) - Introduction to Prometheus, I recommend VictoriaMetrics, but as thing progress, I discover something new.

In the monitoring world, there are log/metric/trace (and more). If you use each server for a single type of task - infra engineer is the one who have to absorb the grunt work.

If only there was a single binary for agent and a single binary for server :eyes:.

Go shopping

Fortunately for us, the monitoring ecosystem is quite vibrant, see Vendors | OpenTelemetry.

I’ve done the homework for you, this is my list of criteria:

  • open source (and have UI)
  • use compiled language (memory efficiency)
  • support many thing on 1 server - log/metric/trace/etc
  • simple deployment model (reduce operational overhead)

After criteria filter, the list shrink significantly, a few more rounds and I think I’ll go with OpenObserve (hope I don’t regret it).

Installation tip and trick

You might want some env for your installation:

ZO_TELEMETRY=false
ZO_SYNTHETICS_ENABLED=true
ZO_COMPACT_DATA_RETENTION_DAYS=14

Also, you could enable json log for your satellite - structured logging is easier for machine:

# config.yaml
log:
    encoding: json

First alert

For now, what I want is just a notification when ERROR log pop up in satellite log, you can ask AI to clone this repo and write the query for you.

Watchout pitfall: remember to exclude openobserve service out of the query, otherwise the query itself become the source of the alert (in journald) and it keep ping-pong each other :).

For example:

SELECT count(*) AS errors
FROM "journald"
WHERE body_message LIKE '%"L":"ERROR"%'
  AND body__systemd_unit NOT LIKE 'openobserve%'

Since setup monitoring, how (un?)fortunate, I did not encounter any ERROR since.

Thanks for reading!

2 Likes
[Tutorial] How to run your own satellite (part 1)