Watchtower still does not work for me, what's wrong?

I am no expert, but here is what I would do assuming that you need to replace the watchtower with a new instance:

  1. Stop the watchtower: docker stop watchtower
  2. Remove the watchtower: docker rm watchtower
  3. Run the watchtower: (where storagenode is your node name)
    docker run -d --restart=always --name watchtower -v /var/run/docker.sock:/var/run/docker.sock storjlabs/watchtower storagenode watchtower --stop-timeout 300s --interval 21600

@stuberman

Thx, for attempting to help. However, I’ve tried that many times with the same results. Anyway, I’ve retried again but this time I also added the –run-once flag so it will try to update immediately and the result is the same as before as seen in the watchtower logs:

storj@pine64 : ~$ docker logs -t watchtower

2020-07-05T02:22:44.828130831Z time="2020-07-05T02:22:44Z" level=info msg="Running a one time update."

2020-07-05T02:22:44.828382880Z time="2020-07-05T02:22:44Z" level=debug msg="Checking containers for updated images"

2020-07-05T02:22:44.828417839Z time="2020-07-05T02:22:44Z" level=debug msg="Retrieving running containers"

2020-07-05T02:22:44.848491689Z time="2020-07-05T02:22:44Z" level=debug msg="Pulling storjlabs/storagenode:latest for /storagenode"

2020-07-05T02:22:44.854495744Z time="2020-07-05T02:22:44Z" level=debug msg="No credentials for storjlabs in /config.json"

2020-07-05T02:22:44.854588705Z time="2020-07-05T02:22:44Z" level=debug msg="Got auth value: "

2020-07-05T02:22:44.854614331Z time="2020-07-05T02:22:44Z" level=debug msg="Got image name: storjlabs/storagenode:latest"

2020-07-05T02:22:44.854637873Z time="2020-07-05T02:22:44Z" level=debug msg="No authentication credentials found for storjlabs/storagenode:latest"

2020-07-05T02:22:45.704945124Z time="2020-07-05T02:22:45Z" level=debug msg="No new images found for /storagenode"

2020-07-05T02:22:48.062868865Z time="2020-07-05T02:22:48Z" level=info msg="Running a one time update."

2020-07-05T02:22:48.063088871Z time="2020-07-05T02:22:48Z" level=debug msg="Checking containers for updated images"

2020-07-05T02:22:48.063310211Z time="2020-07-05T02:22:48Z" level=debug msg="Retrieving running containers"

2020-07-05T02:22:48.091711894Z time="2020-07-05T02:22:48Z" level=debug msg="Pulling storjlabs/storagenode:latest for /storagenode"

2020-07-05T02:22:48.103093984Z time="2020-07-05T02:22:48Z" level=debug msg="No credentials for storjlabs in /config.json"

2020-07-05T02:22:48.103198821Z time="2020-07-05T02:22:48Z" level=debug msg="Got auth value: "

2020-07-05T02:22:48.103223363Z time="2020-07-05T02:22:48Z" level=debug msg="Got image name: storjlabs/storagenode:latest"

2020-07-05T02:22:48.103246905Z time="2020-07-05T02:22:48Z" level=debug msg="No authentication credentials found for storjlabs/storagenode:latest"
1 Like

Sorry I could not help you out.

I just looked in my docker config.json files and there is no string ‘storjlabs’…

Can you show your watchtower command ?

This is it:

docker run -d --restart=always --name watchtower -v /var/run/docker.sock:/var/run/docker.sock storjlabs/watchtower storagenode watchtower --debug --stop-timeout 300s --interval 21600

I think I’ve seen this error before a long time ago. It only happened for someone who had logged into docker hub in their client. If this applies to you, try logging out first and trying again.

1 Like

@BrightSilence

I am not logged into the docker.

So, does anyone else have any other ideas regarding this problem?

Someone has to know the meaning of these log messages:

2020-07-05T02:30:40.653052961Z time="2020-07-05T02:30:40Z" level=debug msg="Sleeping for a seconds to ensure the docker api client has been properly initialized."
2020-07-05T02:30:41.653975805Z time="2020-07-05T02:30:41Z" level=debug msg="Retrieving running containers"
2020-07-05T02:30:41.703132922Z time="2020-07-05T02:30:41Z" level=debug msg="There are no additional watchtower containers"
2020-07-05T02:30:41.703266843Z time="2020-07-05T02:30:41Z" level=debug msg="Starting Watchtower and scheduling first run: 2020-07-06 20:19:46 +0000 UTC m=+150545.377839561"
2020-07-06T20:19:46.228138964Z time="2020-07-06T20:19:46Z" level=debug msg="Checking containers for updated images"
2020-07-06T20:19:46.353213857Z time="2020-07-06T20:19:46Z" level=debug msg="Retrieving running containers"
2020-07-06T20:19:46.582928799Z time="2020-07-06T20:19:46Z" level=debug msg="Pulling storjlabs/watchtower:latest for /watchtower"
2020-07-06T20:19:46.599640486Z time="2020-07-06T20:19:46Z" level=debug msg="No credentials for storjlabs in /config.json"
2020-07-06T20:19:46.599913203Z time="2020-07-06T20:19:46Z" level=debug msg="Got auth value: "
2020-07-06T20:19:46.599967288Z time="2020-07-06T20:19:46Z" level=debug msg="Got image name: storjlabs/watchtower:latest"
2020-07-06T20:19:46.599994497Z time="2020-07-06T20:19:46Z" level=debug msg="No authentication credentials found for storjlabs/watchtower:latest"
2020-07-06T20:19:49.179111194Z time="2020-07-06T20:19:49Z" level=debug msg="No new images found for /watchtower"
2020-07-06T20:19:49.179373869Z time="2020-07-06T20:19:49Z" level=debug msg="Pulling storjlabs/storagenode:latest for /storagenode"
2020-07-06T20:19:49.179567083Z time="2020-07-06T20:19:49Z" level=debug msg="No credentials for storjlabs in /config.json"
2020-07-06T20:19:49.179637544Z time="2020-07-06T20:19:49Z" level=debug msg="Got auth value: "
2020-07-06T20:19:49.179699421Z time="2020-07-06T20:19:49Z" level=debug msg="Got image name: storjlabs/storagenode:latest"
2020-07-06T20:19:49.179759964Z time="2020-07-06T20:19:49Z" level=debug msg="No authentication credentials found for storjlabs/storagenode:latest"
2020-07-06T20:19:52.234134539Z time="2020-07-06T20:19:52Z" level=debug msg="No new images found for /storagenode"
2020-07-06T20:19:52.235227031Z time="2020-07-06T20:19:52Z" level=debug msg="Scheduled next run: 2020-07-08 14:08:51 +0000 UTC"

What credentials is being looked for in the /config.json file and is that why the storagenode is not being updated by the watchtower?

Do you have this file ever?

@Alexey

Ha Ha… no, I just realized the config file is config.yaml. So, what is config.json?

This is config file in the docker installation: https://docs.docker.com/engine/reference/commandline/cli/#configuration-files
If you have any login information, the watchtower will fail to use a public hub without authorization: https://github.com/containrrr/watchtower/issues/451

So, please, try to do:

docker logout

Then stop and remove the watchtower, pull it again and start it again.

@Alexey

Been busy so just getting back to this. I’ve stopped and removed watchtower. I even removed the image (docker rmi) and this is a list of my images:

storj@pine64:~$ docker images
REPOSITORY              TAG                 IMAGE ID            CREATED             SIZE
storjlabs/storagenode   latest              3d9e434f32d0        2 weeks ago         25MB
storjlabs/watchtower    latest              b550a479ea32        8 months ago        13.7MB

Are this IMAGE IDs correct? It says the watchtower image was created 8 mos. ago, is that correct? No matter what I have done I still see the same result… No updates being done and the following messages:

2020-07-13T04:05:12.795280889Z time="2020-07-13T04:05:12Z" level=debug msg="Checking containers for updated images"
2020-07-13T04:05:12.795619736Z time="2020-07-13T04:05:12Z" level=debug msg="Retrieving running containers"
2020-07-13T04:05:12.824539837Z time="2020-07-13T04:05:12Z" level=debug msg="Pulling storjlabs/storagenode:latest for /storagenode"
2020-07-13T04:05:12.832332811Z time="2020-07-13T04:05:12Z" level=debug msg="No credentials for storjlabs in /config.json"
2020-07-13T04:05:12.832421731Z time="2020-07-13T04:05:12Z" level=debug msg="Got auth value: "
2020-07-13T04:05:12.832448399Z time="2020-07-13T04:05:12Z" level=debug msg="Got image name: storjlabs/storagenode:latest"
2020-07-13T04:05:12.832474608Z time="2020-07-13T04:05:12Z" level=debug msg="No authentication credentials found for storjlabs/storagenode:latest"
2020-07-13T04:05:13.541413398Z time="2020-07-13T04:05:13Z" level=debug msg="No new images found for /storagenode"

Should I just give up at this point and we all admit that. at least for me, watchdog is a piece of excrement and it should just be flushed. This is getting way too frustrating.

BTW, when I do a docker logout , as suggested, this is what I get:

storj@pine64 : ~ $ docker logout

Not logged in to https://index.docker.io/v1/

Can you show output of

docker ps -a

@nerdatwork

storj@pine64:~$ docker ps -a`

CONTAINER ID        IMAGE                          COMMAND                  CREATED             STATUS              PORTS                                                                        NAMES
fee59cdd9d53        storjlabs/watchtower           "/watchtower storage…"   24 minutes ago      Up 24 minutes                                                                                    watchtower
f0881c59d33f        storjlabs/storagenode:latest   "/entrypoint"            8 days ago          Up 10 hours         0.0.0.0:7777->7777/tcp, 0.0.0.0:14002->14002/tcp, 0.0.0.0:28967->28967/tcp   storagenode

Let me just add this again before I am once again asked for this information:

The official documentation does not have the –platform arch64 specified in the pull. When I try to pull without it I see the following message:

storj@pine64 : ~ $ docker pull storjlabs/watchtower
Using default tag: latest
latest: Pulling from storjlabs/watchtower
no matching manifest for unknown in the manifest list entries

To get the pull to work I have to add the extra platform parameter as such:

storj@pine64 : ~ $ docker pull storjlabs/watchtower --platform aarch64
Using default tag: latest
latest: Pulling from storjlabs/watchtower
Digest: sha256:31c0c558ec8adad5cec6cb86320c58018289ca22dafa019c31cd5d6f65f03655
Status: Image is up to date for storjlabs/watchtower:latest

You will also notice that the :latest suffix has been as the default tag. Again, earlier in this was told I should be using the :beta. I have tried the :beta tag but that does not work:

storj@pine64 : ~ $ docker pull storjlabs/watchtower:beta
Error response from daemon: manifest for storjlabs/watchtower:beta not found

storj@pine64 : ~ $ docker pull storjlabs/watchtower:beta --platform aarch64
Error response from daemon: manifest for storjlabs/watchtower:beta not found

Anyway, just wanted to provide this information before someone decides to ask me again about why I am using the latest tag and the aarch64 platform option.

I know you said you’re not logged in. But I’ve only ever seen this error for people who were. Maybe just to be sure try running docker logout.

The latest tag is the correct one now. That has changed recently. So you can ignore my earlier comment.

As I posted earlier … this is what I get when I do docker logout

1 Like

Sorry, didn’t scroll up to see you already ran it.

Having some spare time so I decided to try to reproduce your problem.
uname -a
Linux raspberrypi3 5.3.0-1018-raspi2 #20~18.04.1-Ubuntu SMP Wed Feb 5 04:56:34 UTC 2020 aarch64 aarch64 aarch64 GNU/Linux

1
docker pull storjlabs/storagenode:latest --platform aarch64
“–platform” is only supported on a Docker daemon with experimental features enabled
docker --version
Docker version 18.09.7

2 (auth entered but invalid on docker hub):
docker pull storjlabs/watchtower:latest
Error response from daemon: Get https://registry-1.docker.io/v2/storjlabs/watchtower/manifests/latest: unauthorized: incorrect username or password

3
docker logout
Removing login credentials for https://index.docker.io/v1/
docker pull storjlabs/watchtower:latest
latest: Pulling from storjlabs/watchtower
3a807ee5a0f8: Pull complete
18bdfcdb74a4: Pull complete
047a83254312: Pull complete
Digest: sha256:31c0c558ec8adad5cec6cb86320c58018289ca22dafa019c31cd5d6f65f03655
Status: Downloaded newer image for storjlabs/watchtower:latest

4 Corrupt auth
docker pull storjlabs/watchtower:latest
WARNING: Error loading config file: /root/.docker/config.json: illegal base64 data at input byte 0
latest: Pulling from storjlabs/watchtower
3a807ee5a0f8: Pull complete
18bdfcdb74a4: Pull complete
047a83254312: Pull complete
Digest: sha256:31c0c558ec8adad5cec6cb86320c58018289ca22dafa019c31cd5d6f65f03655
Status: Downloaded newer image for storjlabs/watchtower:latest

Nothing I do looks like your result!

It looks like your docker is looking for an auth that it doesn’t have in .docker/config.json. Did you ever have a login for docker hub that you used here?

I think you need to investigate why you need the “platform” switch as a normal setup doesn’t need it.

Hope this helps in some way :wink:

2 Likes

@andrew2.hart

The docker on this system has only ever been used for storj so if there was any login done for any reason it was not done by me and was done by storj. I have already tried running docker logout (as seen above) which reported there were no login. This is the content of my .docker/config.json file:

storj@pine64 : ~ $ cat .docker/config.json
{

"auths": {},
"HttpHeaders": {
    "User-Agent": "Docker-Client/18.09.3 (linux)"
}

}

Should there be something in the auths field?

As you can see from my previous post, unlike yours, the –platform parameter is necessary in my case. I have no idea why it is not in your case.

This is quite frustrating…