Watchtower advanced configuration

Odmin · June 28, 2019, 5:40am

Hi all,

I would like share advanced configuration of watchtower for SNO:

docker run -d \
    --restart=always \
    --name watchtower \
    -v /var/run/docker.sock:/var/run/docker.sock \
    -v /etc/ssl/certs:/etc/ssl/certs \
    -e WATCHTOWER_NOTIFICATIONS=email \
    -e WATCHTOWER_NOTIFICATION_EMAIL_FROM=Storj_v3_alpha@storj.odmin.example \
    -e WATCHTOWER_NOTIFICATION_EMAIL_TO=odmin@storj.odmin.example \
    -e WATCHTOWER_NOTIFICATION_EMAIL_SERVER=smtp.odmin.example \
    storjlabs/watchtower --debug --stop-timeout 300

This configuration will send email for you when update is comming and add to email some information from log.

smtp.odmin.example - it my local smtp server without autentification, if your smtp require autentification, please add addition options:

 --notification-email-server-user
 --notification-email-server-password

Also you can see all options using this command:

docker run --rm storjlabs/watchtower --help

here is sample output:

docker run --rm storjlabs/watchtower --help
NAME:
   watchtower - Automatically update running Docker containers

USAGE:
   watchtower [global options] command [command options] [arguments...]

VERSION:
   0.3.1 - 98f916f29a041ae045b1e7a7e5d05e6202c766f4 - 2019-04-04T20:55:23Z

COMMANDS:
     help, h  Shows a list of commands or help for one command

GLOBAL OPTIONS:
   --host value, -H value                       daemon socket to connect to (default: "unix:///var/run/docker.sock") [$DOCKER_HOST]
   --interval value, -i value                   poll interval (in seconds) (default: 300) [$WATCHTOWER_POLL_INTERVAL]
   --schedule value, -s value                   the cron expression which defines when to update [$WATCHTOWER_SCHEDULE]
   --no-pull                                    do not pull new images [$WATCHTOWER_NO_PULL]
   --no-restart                                 do not restart containers [$WATCHTOWER_NO_RESTART]
   --cleanup                                    remove old images after updating [$WATCHTOWER_CLEANUP]
   --tlsverify                                  use TLS and verify the remote [$DOCKER_TLS_VERIFY]
   --stop-timeout value                         timeout before container is forcefully stopped (default: 10s) [$WATCHTOWER_TIMEOUT]
   --label-enable                               watch containers where the com.centurylinklabs.watchtower.enable label is true [$WATCHTOWER_LABEL_ENABLE]
   --debug                                      enable debug mode with verbose logging
   --notifications value                        notification types to send (valid: email, slack, msteams) [$WATCHTOWER_NOTIFICATIONS]
   --notifications-level value                  The log level used for sending notifications. Possible values: "panic", "fatal", "error", "warn", "info" or "debug" (default: "info") [$WATCHTOWER_NOTIFICATIONS_LEVEL]
   --notification-email-from value              Address to send notification e-mails from [$WATCHTOWER_NOTIFICATION_EMAIL_FROM]
   --notification-email-to value                Address to send notification e-mails to [$WATCHTOWER_NOTIFICATION_EMAIL_TO]
   --notification-email-server value            SMTP server to send notification e-mails through [$WATCHTOWER_NOTIFICATION_EMAIL_SERVER]
   --notification-email-server-port value       SMTP server port to send notification e-mails through (default: 25) [$WATCHTOWER_NOTIFICATION_EMAIL_SERVER_PORT]
   --notification-email-server-tls-skip-verify  Controls whether watchtower verifies the SMTP server's certificate chain and host name. If set, TLS accepts any certificate presented by the server and any host name in that certificate. In this mode, TLS is susceptible to man-in-the-middle attacks. This should be used only for testing. [$WATCHTOWER_NOTIFICATION_EMAIL_SERVER_TLS_SKIP_VERIFY]
   --notification-email-server-user value       SMTP server user for sending notifications [$WATCHTOWER_NOTIFICATION_EMAIL_SERVER_USER]
   --notification-email-server-password value   SMTP server password for sending notifications [$WATCHTOWER_NOTIFICATION_EMAIL_SERVER_PASSWORD]
   --notification-slack-hook-url value          The Slack Hook URL to send notifications to [$WATCHTOWER_NOTIFICATION_SLACK_HOOK_URL]
   --notification-slack-identifier value        A string which will be used to identify the messages coming from this watchtower instance. Default if omitted is "watchtower" (default: "watchtower") [$WATCHTOWER_NOTIFICATION_SLACK_IDENTIFIER]
   --notification-msteams-hook value            The MSTeams WebHook URL to send notifications to [$WATCHTOWER_NOTIFICATION_MSTEAMS_HOOK_URL]
   --notification-msteams-data                  The MSTeams notifier will try to extract log entry fields as MSTeams message facts [$WATCHTOWER_NOTIFICATION_MSTEAMS_USE_LOG_DATA]
   --help, -h                                   show help
   --version, -v                                print the version

Here is sample text of email:

2019-06-25 16:24:37 (info): Found new storjlabs/storagenode:alpha image (sha256:904c30b6704b493c5fd5d4625e25d2cac0b27689c48b22cc741e93a73cd36cc7)
2019-06-25 16:24:39 (info): Stopping /storagenode (49f91eb2b2faed09944e2ee2d1c8633fa2387cdad9aa601591052d0a4bdb86cd) with SIGTERM
2019-06-25 16:25:00 (info): Creating /storagenode

jocelyn · June 28, 2019, 10:03pm

hey @Odmin this is great thank you!!

Odmin · June 29, 2019, 5:54am

You are welcome!
I just share my good experience with community

orion · July 1, 2019, 7:46pm

Thanks @Odmin ! this is good stuff

Odmin · September 11, 2019, 5:59pm

Today we have update for watchtower, I would like update advanced configuration too:

Now we have watchtower from containrrr

docker run --rm storjlabs/watchtower --help

here is sample output:

Watchtower automatically updates running Docker containers whenever a new image is released.
More information available at https://github.com/containrrr/watchtower/.

Usage:
  watchtower [flags]

Flags:
  -c, --cleanup                                     remove previously used images after updating
  -d, --debug                                       enable debug mode with verbose logging
  -h, --help                                        help for watchtower
  -H, --host string                                 daemon socket to connect to (default "unix:///var/run/docker.sock")
  -S, --include-stopped                             Will also include created and exited containers
  -i, --interval int                                poll interval (in seconds) (default 300)
  -e, --label-enable                                watch containers where the com.centurylinklabs.watchtower.enable label is true
  -m, --monitor-only                                Will only monitor for new images, not update the containers
      --no-pull                                     do not pull any new images
      --no-restart                                  do not restart any containers
      --notification-email-from string              Address to send notification emails from
      --notification-email-server string            SMTP server to send notification emails through
      --notification-email-server-password string   SMTP server password for sending notifications
      --notification-email-server-port int          SMTP server port to send notification emails through (default 25)
      --notification-email-server-tls-skip-verify
                                                    Controls whether watchtower verifies the SMTP server's certificate chain and host name.
                                                    Should only be used for testing.

      --notification-email-server-user string       SMTP server user for sending notifications
      --notification-email-to string                Address to send notification emails to
      --notification-msteams-data                   The MSTeams notifier will try to extract log entry fields as MSTeams message facts
      --notification-msteams-hook string            The MSTeams WebHook URL to send notifications to
      --notification-slack-channel string           A string which overrides the webhook's default channel. Example: #my-custom-channel
      --notification-slack-hook-url string          The Slack Hook URL to send notifications to
      --notification-slack-icon-emoji string        An emoji code string to use in place of the default icon
      --notification-slack-icon-url string          An icon image URL string to use in place of the default icon
      --notification-slack-identifier string        A string which will be used to identify the messages coming from this watchtower instance (default "watchtower")
  -n, --notifications strings                        notification types to send (valid: email, slack, msteams
      --notifications-level string                  The log level used for sending notifications. Possible values: panic, fatal, error, warn, info or debug (default "info")
  -R, --run-once                                    Run once now and exit
  -s, --schedule string                             the cron expression which defines when to update
  -t, --stop-timeout duration                       timeout before a container is forcefully stopped (default 10s)
  -v, --tlsverify                                   use TLS and verify the remote

Updated run string:

docker run -d \
    --restart=always \
    --name watchtower \
    -v /var/run/docker.sock:/var/run/docker.sock \
    -v /etc/ssl/certs:/etc/ssl/certs \
    -e WATCHTOWER_NOTIFICATIONS=email \
    -e WATCHTOWER_NOTIFICATION_EMAIL_FROM=Storj_v3_alpha@storj.odmin.example \
    -e WATCHTOWER_NOTIFICATION_EMAIL_TO=odmin@storj.odmin.example \
    -e WATCHTOWER_NOTIFICATION_EMAIL_SERVER=smtp.odmin.example \
    -e WATCHTOWER_CLEANUP \
    storjlabs/watchtower --debug --interval 21600 --stop-timeout 300s

Odmin · October 9, 2020, 2:27pm

Here is updated configuration to the current moment:

docker run -d \
    --dns 1.1.1.1 \
    --restart=always \
    --name watchtower \
    -v /var/run/docker.sock:/var/run/docker.sock \
    -v /etc/ssl/certs:/etc/ssl/certs \
    -e WATCHTOWER_NOTIFICATIONS=email \
    -e WATCHTOWER_NOTIFICATION_EMAIL_FROM=Storj_v3_node@odmin.example.local \
    -e WATCHTOWER_NOTIFICATION_EMAIL_TO=odmin@odmin.example.local\
    -e WATCHTOWER_NOTIFICATION_EMAIL_SERVER=example.local \
    -e WATCHTOWER_NOTIFICATION_EMAIL_SERVER_TLS_SKIP_VERIFY \
    containrrr/watchtower --debug --interval 21600 --stop-timeout 300s --cleanup

peem · October 9, 2020, 2:37pm

containrrr/watchtower
or
storjlabs/watchtower

???

Odmin · October 9, 2020, 2:39pm

I discribed it here

peem · October 9, 2020, 2:43pm

Will your configuration be able to update two nodes on the same machine? because mine did not cope (it closed both nodes but lifted one) and manually updated …

jeremyfritzen · October 9, 2020, 2:47pm

Based on what I see, the docker run command stated by @Odmin should update all containers on the current machine.

Odmin · October 9, 2020, 2:47pm

This configuration updating all docker containers, if you would like the update just two:

docker run -d \
    --dns 1.1.1.1 \
    --restart=always \
    --name watchtower \
    -v /var/run/docker.sock:/var/run/docker.sock \
    -v /etc/ssl/certs:/etc/ssl/certs \
    -e WATCHTOWER_NOTIFICATIONS=email \
    -e WATCHTOWER_NOTIFICATION_EMAIL_FROM=Storj_v3_node@odmin.example.local \
    -e WATCHTOWER_NOTIFICATION_EMAIL_TO=odmin@odmin.example.local\
    -e WATCHTOWER_NOTIFICATION_EMAIL_SERVER=example.local \
    -e WATCHTOWER_NOTIFICATION_EMAIL_SERVER_TLS_SKIP_VERIFY \
    containrrr/watchtower storagenode watchtower --debug --interval 21600 --stop-timeout 300s --cleanup

Alexey · October 10, 2020, 10:01pm

The containrrr/watchtower do not follow the rolling update procedure and thus not recommended to use.

Odmin · October 11, 2020, 6:31am

Could you please describe more detail the “rolling update procedure”?

Alexey · October 11, 2020, 11:10am

It should not update the storagenode container immediately after receiving an information that a new version is available.
This is default configuration for the storjlabs/watchtower
It would be much better to follow the cursor on https://version.storj.io, however, even for storjlabs/watchtower it’s not the case.
So we implemented an own storagenode-updater to follow the cursor on https://version.storj.io
The Linux version almost done.

DocMAX · May 3, 2023, 4:32pm

Why do i get?:

time=“2023-05-03T16:30:23Z” level=error msg=“Failed to send shoutrrr notification” error=“error enabling StartTLS: x509: certificate is valid for , not smtp.intra” index=0 notify=no service=smtp

I set
WATCHTOWER_NOTIFICATION_EMAIL_SERVER_TLS_SKIP_VERIFY: yes
in my docker compose

Alexey · May 4, 2023, 3:24am

Hello @DocMAX,
Welcome to the forum!

Looks like your SMTP doesn’t support StartTLS or it’s absent. It’s better to ask for help regarding watchtower in the origin (we uses fork of Watchtower).

In general, the format of providing values for variables in the docker-compose you may see there: Ways to set environment variables in Compose | Docker Documentation

For the storagenode software the notification from the watchtower less useful, because now watchtower only updates a base image (not frequently), the node itself is updated inside the container with the storagenode-updater service during the rollout.