I made my docker-based nodes log to journald. Why? In theory, journald has some nice features, like automating rotation, compression, collecting all logs in a single place, potentially sending them to a dedicated machine. Here’s a description of my attempt of using journald with storage nodes. Posting it in hope someone will find it useful.
It was actually quite simple to configure. I needed to add some magical parameters to the docker run command:
docker run --log-driver=journald --log-opt tag='{{.Name}}' … -e RUN_PARAMS="--log.output stdout" …
--log-driver tells docker to redirect stdout and stderr of the application to journald. Whatever comes from stderr is considered a warning. By default the storage node binary logs to stderr, so to avoid logging everything as warnings, I added -e RUN_PARAMS="--log.output stdout". Also, to make it easy to distinguish logs from different nodes, --log-opt tag='{{.Name}}' tags each log entry with the container’s name—useful if you have multiple nodes on the same machine and want the node name to be visible in journalctl -f.
I also set the following in /etc/systemd/journald.conf:
[Journal]
Storage=persistent
SystemMaxUse=100G
SystemKeepFree=5G
SystemMaxFileSize=250M
SystemMaxFiles=1000000
…just to make sure that each single journalctl file has a convenient (for me) size and I’m not losing any data unless I really really want to.
You can then follow logs of a single storage node by:
journalctl --cat -f CONTAINER_NAME=storagenode42
You can follow logs of all nodes (assuming all their containers’ names start with storagenode) by:
journalctl -f CONTAINER_NAME=storagenode*
Or just see all system logs together, both storage nodes and other services, by:
journalctl -f
It turns out though, journald logs take a lot of space; on my system it was about 5× the amount of plaintext files, this even despite that I enabled compression in the journald’s configuration file. I solved it by compressing rotated files periodically with:
xz /var/log/journal/*/*@*.journal
(all files there with @ in their name contain log entries that were already rotated).
Moreover, for some reason journald amplified I/O a lot; it was strange, but iotop reported that journald performs more I/O than a storage node. It’s apparently “normal” for journald, it has not been optimized for the amount of log entries that come out of storage nodes.
Due to these two reasons, after few weeks I decided to go back to plain files.