Watchtower still does not work for me, what's wrong?

thebadcat · July 26, 2020, 6:29pm

You will get log info for watchtower if you start it with the debug flag (the 2nd -d):

docker run -d --restart=always --name watchtower -v /var/run/docker.sock:/var/run/docker.sock storjlabs/watchtower storagenode watchtower --stop-timeout 300s --interval 21600 -d

Yes, I also originally thought the single -d was sufficient but was later told that the first one was actually a docker flag and not part of the watchtower invocation.

You can then do:

docker logs -t watchtower

to see the info (not very verbose actually)

nerdatwork · July 26, 2020, 7:05pm

This version was pushed only for Windows GUI nodes and not for docker so your node is correct to use v1.6.4

mrkeyboardcommando · July 26, 2020, 7:58pm

ok, thats why it cannot find a update.

Wouldn’t watchtower still be logging without using the -d flag?
It was previously, and because of the log items i removed the container and went to see for a fix.

And that 2nd -d is also not present in the official documentation…

Alexey · July 27, 2020, 6:49am

It will log without -d flag an info and error level messages.

BrightSilence · July 27, 2020, 9:18am

Yes, but that only includes log messages when an update is found, not when it checks for updates. The default logging is pretty minimal, so I prefer to use that flag to have a little bit more.

baker · July 27, 2020, 2:24pm

For what it’s worth, I am using watchtower on a rock64 without issues. I am using the exact command from the storj documentation, without experimental and without -platform aarch64. I would hazard a guess that we are using different OS though. I am using the ayufan Debian Stretch image, which is admittedly bit outdated at this point.

$ uname -srm
Linux 4.4.202-1237-rockchip-ayufan-gfd4492386213 aarch64

$ docker --version
Docker version 19.03.8, build afacb8b

mrkeyboardcommando · July 27, 2020, 2:50pm

i do use the ayufan image…

Linux 4.4.126-rockchip-ayufan-239 aarch64
Docker version 19.03.8, build afacb8b

This node should be decommissioned… as it isn’t supported anymore.
That’s why i am setting up a second node on x86-64 hardware… maybe the rock64 will return someday

baker · July 27, 2020, 3:14pm

I have been meaning to switch my rock64 over to the latest Armbian Buster release since it is regularly updated (currently kernel v5.4.43) and now officially supports the rock64 (I don’t believe it did when I first got the board).

Alexey · August 4, 2020, 5:12am

18 posts were split to a new topic: Iptables on a new pine64 system

thebadcat · August 21, 2020, 9:24pm

FINALLY!!! I noticed today on my dashboard that it said my node had an UPTIME of 30h58m. Knowing that this didn’t look correct I logged in and checked watchtower and it looks like it finally is finding and updating my node:

2020-08-20T14:21:52.768505332Z time=“2020-08-20T14:21:52Z” level=debug msg=“No authentication credentials found for storjlabs/storagenode:latest”

2020-08-20T14:22:00.685977010Z time=“2020-08-20T14:22:00Z” level=info msg=“Found new storjlabs/storagenode:latest image (sha256:ebdf5bd79234128d2d3fb4183a17ef2a03a77d948eea11c00e47a652188fe030)”

2020-08-20T14:22:00.686126223Z time=“2020-08-20T14:22:00Z” level=debug msg=“Pulling storjlabs/watchtower:latest for /watchtower”

2020-08-20T14:22:00.686158266Z time=“2020-08-20T14:22:00Z” level=debug msg=“No credentials for storjlabs in /config.json”

2020-08-20T14:22:00.686271978Z time=“2020-08-20T14:22:00Z” level=debug msg="Got auth value: "

2020-08-20T14:22:00.686299270Z time=“2020-08-20T14:22:00Z” level=debug msg=“Got image name: storjlabs/watchtower:latest”

2020-08-20T14:22:00.686323813Z time=“2020-08-20T14:22:00Z” level=debug msg=“No authentication credentials found for storjlabs/watchtower:latest”

2020-08-20T14:22:01.448650215Z time=“2020-08-20T14:22:01Z” level=debug msg=“No new images found for /watchtower”

2020-08-20T14:22:01.448784885Z time=“2020-08-20T14:22:01Z” level=info msg=“Stopping /storagenode (73e2a25d41cdf852f3d3445471ad7b4e6a50f2a77227861f4f285bad46edd0cd) with SIGTERM”

2020-08-20T14:22:07.362952899Z time=“2020-08-20T14:22:07Z” level=debug msg=“Removing container 73e2a25d41cdf852f3d3445471ad7b4e6a50f2a77227861f4f285bad46edd0cd”

2020-08-20T14:22:07.476193008Z time=“2020-08-20T14:22:07Z” level=info msg=“Creating /storagenode”

2020-08-20T14:22:07.786226745Z time=“2020-08-20T14:22:07Z” level=debug msg=“Starting container /storagenode (925c1cacca082a84e08ef33a677213fe3302207ee4ad69c2c0af1c652d65dc55)”

2020-08-20T14:22:09.523425708Z time=“2020-08-20T14:22:09Z” level=debug msg=“Scheduled next run: 2020-08-22 23:59:51 +0000 UTC”

donald.m.motsinger · October 9, 2020, 12:21pm

Watchtower runs on my main server without problems since the beginning. To troubleshoot an unrelated problem I moved 3 nodes to another PC. Watchtower doesn’t work there. First I had the wrong names of the containers in my watchtower (d’oh). After correcting it, stopping and removing the watchtower container, It still doesn’t update.

docker logs watchtower gives me

time="2020-10-08T22:06:49Z" level=info msg="Waiting for running update to be finished..."

and these nodes are still on 1.12.3. My docker run command is

docker run -d --restart=unless-stopped --name watchtower -v /var/run/docker.sock:/var/run/docker.sock storjlabs/watchtower storagenode3 storagenode4 storagenode5 watchtower --stop-timeout 300s --cleanup

which is the same as on the other server, only with different container names.

peem · October 9, 2020, 12:43pm

with
--cleanup
I have the storjlabs/watchtower working, but not updating …

in the official documentation is:
docker run -d --restart=always --name watchtower -v /var/run/docker.sock:/var/run/docker.sock storjlabs/watchtower storagenode watchtower --stop-timeout 300s --interval 21600

there is no --cleanup

donald.m.motsinger · October 9, 2020, 2:17pm

I got the --cleanup from here

and it’s also documented here

https://containrrr.dev/watchtower/arguments/

I’ll try without it now and will report back. It works on the other server though without problems.

Odmin · October 9, 2020, 2:22pm

Please hold on, I will update it now to the current version of the configuration.

Odmin · October 9, 2020, 2:34pm

I post the updated configuration, I use the original watchtower from containrrr/watchtower because storjlabs/watchtower is not updated more than half of the year. I prefer to use the original with fixed bugs.

donald.m.motsinger · October 10, 2020, 10:42am

Just reporting back that watchtower updated all 3 nodes on that PC last night after removing the --cleanup option. It works with --cleanup on the other server though. Strange…

Odmin · October 10, 2020, 12:40pm

Please look into watchtower log docker logs -t watchtower
I use this configuration (with --cleanup) for a very long time and have no issue.

Edited:
PS.
Here and example of debug log when update is found:

--cleanup did a good job.

donald.m.motsinger · October 10, 2020, 1:48pm

As I wrote, --cleanup works on my server, but not that PC.

with --cleanup:

time="2020-10-08T22:06:49Z" level=info msg="Waiting for running update to be finished..."

without --cleanup:

    time="2020-10-10T02:30:07Z" level=info msg="Found new storjlabs/storagenode:latest image (sha256:ae1314be9e571adb052a8ad71999acd59bb308e660747df3e455d29197273081)"
    time="2020-10-10T02:30:08Z" level=info msg="Found new storjlabs/storagenode:latest image (sha256:ae1314be9e571adb052a8ad71999acd59bb308e660747df3e455d29197273081)"
    time="2020-10-10T02:30:08Z" level=info msg="Stopping /storagenode3 (22aade1421aa56ebc7599dee37c0519898057f0b4b76508ae33cc787e974cffa) with SIGTERM"
    time="2020-10-10T02:30:09Z" level=info msg="Stopping /storagenode4 (ed8e10b58706ceba937fe912938555d8812c9a364bf723a4e8f1d1ab211c874a) with SIGTERM"
    time="2020-10-10T02:30:10Z" level=info msg="Stopping /storagenode5 (f7291806ee52b7918c4db4011be1a80189730641d173ddcfbc96af253c149af3) with SIGTERM"
    time="2020-10-10T02:30:11Z" level=info msg="Creating /storagenode5"
    time="2020-10-10T02:30:12Z" level=info msg="Creating /storagenode4"
    time="2020-10-10T02:30:13Z" level=info msg="Creating /storagenode3"

Odmin · October 10, 2020, 5:41pm

Very interesting… try it docker image ls then docker image prune and rerun watchtower with --cleanup

donald.m.motsinger · October 13, 2020, 2:36pm

I guess it wasn’t the --cleanup which causes the problem. Now watchtower is stuck again

~# docker logs watchtower
time="2020-10-11T13:50:44Z" level=info msg="Waiting for running update to be finished..."

No more log entries since 2 days ago. I found this issue on Github, which sounds similar to my problem

I had to restart this PC several times in the last few days due to another problem and I didn’t stop watchtower before. I’ll investigate further in this direction…