Watchtower and Docker Compose

I use docker-compose for everything on my server and I added Storj and Watchtower to it, my node is still on v1.1.1, is there something i have configured incorrectly?

        storagenode:
        image: storjlabs/storagenode:beta
        restart: unless-stopped
        ports:
            - "28967:28967"
            - "14002:14002"
        environment:
            - "WALLET="
            - "EMAIL="
            - "ADDRESS="
            - "STORAGE=5TB"
        volumes:
            - "/mnt/storj/identity:/app/identity"
            - "/mnt/storj/storage:/app/config"
    watchtower:
        image: storjlabs/watchtower
        restart: unless-stopped
        command: "storagenode watchtower --stop-timeout 300s --interval 21600"
        volumes:
            - "/var/run/docker.sock:/var/run/docker.sock"

Storj is pushing rolling updates so docker nodes will be updated slowly. If your watchtower is running smoothly then it will update automatically. Be patient!

how can I tell if my watchtower is running properly? The logs are empty it seems

If you are using this server for anything else, I would strongly advise against using watchtower as the watchtower container effectively has root access to the entire host system. If you subscribe to the principle of least privilege, watchtower should only be used on systems that do nothing but run Storj nodes.

Until the beta tag is updated on dockerhub, there’s nothing that watchtower can do to even begin to update nodes.

Can you elaborate on that ?

The storjlabs/storagenode:beta image is published on Docker Hub. Storj publishes many interim containers likely due to an automated CI/CD process, but ultimately only attaches the beta tag to the release containers that SNOs are to pull to their machines. So only when the image is pushed to Docker Hub can a SNO actually get the latest beta tagged release.

Storj also publishes their own fork of the containrrr/watchtower image that updates Docker containers on a host. Watchtower functions to automatically check Docker Hub for the latest images for running containers on a set interval. The upstream image results in a watchtower container that defaults to checking every 300 seconds, or 5 minutes. This can be modified by setting the --interval switch in the watchtower command.

The storjlabs/watchtower fork of the upstream image makes the key modification to set the interval internally to a random number between 12 and 72 hours. Therefore, depending on what the random interval time is generated for a given SNO’s system, it may take up to 72 hours for watchtower to check Docker Hub for an updated beta image after it’s been pushed there by the Storj team.

1 Like

Starting this release Storj is pushing rolling updates which means it might take longer than 72 hours for docker images to be deployed.

What is the mechanism for pushing the rolling updates if not via watchtower? They can only leverage the software that is already deployed for Docker-based SNOs.

The 72h starts after the docker image is published, but it may take longer for the docker image to be published. Updates roll out roughly like this.
image
Though some times the phased rollouts of binary nodes takes a little longer or shorter depending on the needs. The docker image isn’t always published on day 8. But usually around that time.

1 Like

Thanks for the calendar @BrightSilence. My main message is that people worried about their watchtower installations should rest easy as long as the Docker image is not yet updated on Docker Hub. If we’re in the blue zone on the calendar and people are not getting updated within 72 hours, then they should start questioning whether watchtower isn’t working correctly. Checking the beta tag on Docker Hub allows one to be sure whether the image is available or not.

2 Likes

Please, replace it ASAP to the long syntax

    volumes:
      - type: bind
        source: /mnt/storj/identity
        target: /app/identity
      - type: bind
        source: /mnt/storj/storage
        target: /app/config

The difference is the same as -v (your version) and --mount (the must have version):

If you use -v or --volume to bind-mount a file or directory that does not yet exist on the Docker host, -v creates the endpoint for you. It is always created as a directory.
If you use --mount to bind-mount a file or directory that does not yet exist on the Docker host, Docker does not automatically create it for you, but generates an error.

For the storagenode the first behavior is the way to disqualification if your drive doesn’t mounted for any reason, because the data will be stored inside the container and will be removed with docker-compose down

3 Likes

All of my docker containers remain on the same drive, if it’s unmounted nothing would work anyway, but I will update it next time I get around to updating my compose. Does the watchtower setup look correct, and will it touch any container that isn’t the storagenode?

You will experiment with names of containers though, because docker-compose modify them.

Good catch @Alexey. @lholliger, include the container_name key in your file. See the below excerpt from my docker-compose.yml:

  storagenode:
    container_name: storagenode
    hostname: storagenode
    image: storjlabs/storagenode:beta
    ports:
    - "28967:28967"
    - "14002:14002"
    ...

Do the same for watchtower. Otherwise the containers will get a fun but inconsistent names every time to run docker-compose up -d and your watchtower container won’t be able to update them.

This is wrong. The template for compose container names is ${DIR}_${SERVICE}_${NUM}:

  • $DIR is the name of the directory containing the compose file.
  • $SERVICE is the name of the service as configured in the compose file.
  • $NUM is the container number, which will be 1 unless you’ve configured compose to start multiple containers per service.

So in this case, it would be ${DIR}_storagenode_1 (we don’t know what $DIR is, but OP does).

1 Like

Good call, you’re right. I’ve gotten into the habit of naming my containers, so I was mixing this up with running docker run without the --name argument.

1 Like

It looks like that watchtower wont work. The node version is still 1.5.2.

#docker_compose.yml
version: "3.3"
services:
  storagenode:
    image: storjlabs/storagenode:latest
    volumes:
      - type: bind
        source: "/home/storjio/.local/share/storj/identity/storagenode"
        target: /app/identity
      - type: bind
        source: "/home/storjio/storagenode_files"
        target: /app/config
    ports:
      - 28968:28967
      - 14002:14002
    deploy:
      restart_policy:
        condition: always
    environment:
      - WALLET=0xe5380DBa756Ea63b2bc98769bEB614750DaE0089
      - EMAIL=marlon.otto@outlook.de
      - ADDRESS=storjio.ghostip.tk:28968
      - STORAGE=530GB
  watchtower:
    image: storjlabs/watchtower
    restart: always
    container_name: watchtower
    command: --interval 21600 --cleanup true --stop-timeout 300s --cleanup
    volumes:
    - /var/run/docker.sock:/var/run/docker.sock

Scroll up to this post

Did anybody managed to get proper update interval either via WATCHTOWER_SCHEDULE (–schedule) or WATCHTOWER_POLL_INTERVAL (–interval)?

Doesn’t matter which one i define, after starting storjlabs/watchtower, the log shows start time which is doesn’t align with any of that. “Starting Watchtower and scheduling first run: 2021-05-13 07:12:42 -0400 EDT”.

Restarts of container is another random time value. ENV var or command, “schedule” or “interval”, no difference.