Two watchtower containers running

Node Operators troubleshooting
1

Hi There,

My storage node has been running great since August '19. I check on it once every week or so and it just keeps humming along. Watchtower has been updating the nodes without fail.

Today I had to reboot my node (moving it). I checked and I have two containers with Watchtower. I searched the forum but didn’t find a thread that looked relevant.

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
363a4e3b3037 storjlabs/storagenode:beta “/entrypoint” 12 days ago Up 38 minutes
8aa59eea5e59 storjlabs/watchtower:latest “/watchtower storage…” 4 weeks ago Up 44 minutes
eee9e0d212d1 bf9c7c4c4647 “/watchtower storage…” 5 months ago Up 44 minutes

So I’m wondering if I need to nuke one of them and if so which one?
Thanks for any help

2

same. just saw it on mine too. 4 weeks ago something happened with watchtower:

docker ps -a
CONTAINER ID        IMAGE                         COMMAND                  CREATED             STATUS              PORTS                      NAMES
024f75446642        storjlabs/storagenode:beta    "/entrypoint"            12 days ago         Up 12 days          0.0.0.0:28967->28967/tcp   storagenode
e19d4183e3b9        storjlabs/watchtower:latest   "/watchtower storage…"   4 weeks ago         Up 4 weeks                                     watchtower
3df90f1c8bba        66a01cc8b87f                  "/watchtower storage…"   5 months ago        Up 5 months                                    VuaOHFyCTIgepcQbQsMTYOCKAVjusNsI
1 Like
3

old watchtower created a new watchtower container on 2020-02-19T22:19:35Z:

docker logs VuaOHFyCTIgepcQbQsMTYOCKAVjusNsI
time="2019-10-09T16:19:30Z" level=info msg="Found new storjlabs/storagenode:beta image (sha256:0f08381929f57eb15dc48e47d28aca8cc7907935c0df0509a837ecd4a507b782)"
time="2019-10-09T16:19:30Z" level=info msg="Stopping /storagenode (8677a6be53cda06701823465c542bff3932b9d0add1b9651535f39e2ae23cc5a) with SIGTERM"
time="2019-10-09T16:19:38Z" level=info msg="Creating /storagenode"
time="2019-10-13T22:19:23Z" level=info msg="Found new storjlabs/storagenode:beta image (sha256:53de16e5bba45f3981fe980bb7d2909f07634b5a8a2036b093d5e3930cb938f2)"
time="2019-10-13T22:19:30Z" level=info msg="Stopping /storagenode (01d4246aed222af43c139bf14c0ad5ebc96cef8b8cabf0f2c26d135e246ecdb1) with SIGTERM"
time="2019-10-13T22:19:36Z" level=info msg="Creating /storagenode"
time="2019-10-16T22:19:21Z" level=info msg="Found new storjlabs/storagenode:beta image (sha256:6cbb207adaba7f1519d25db645449a7e5f1104d0b683ba6ab1af67bf1b8affb3)"
time="2019-10-16T22:19:23Z" level=info msg="Stopping /storagenode (427a1567ef1903d7ed7a48ab2459eaa8edd46d4304b5e3bc77f4b1493b98405c) with SIGTERM"
time="2019-10-16T22:19:36Z" level=info msg="Creating /storagenode"
time="2019-10-21T16:19:19Z" level=info msg="Found new storjlabs/storagenode:beta image (sha256:177cb6979839c60feee22a5b7b1a2a42ebaa7f9bc9e1e38bd9df064695086178)"
time="2019-10-21T16:19:25Z" level=info msg="Stopping /storagenode (357d0a003bd8f5610e1c9a234d261a91560701acea12f051e27942dad41bc933) with SIGTERM"
time="2019-10-21T16:19:32Z" level=info msg="Creating /storagenode"
time="2019-10-28T16:19:21Z" level=info msg="Found new storjlabs/storagenode:beta image (sha256:58b5d9dff814f9da2f8de2e6b10c8c430a6dda2845f88534fbdce699cdbd0f97)"
time="2019-10-28T16:19:23Z" level=info msg="Stopping /storagenode (7d75e492404309551b4b895d5b838fce420ba97d06aa88a27a9cd473c923c2fa) with SIGTERM"
time="2019-10-28T16:19:26Z" level=info msg="Creating /storagenode"
time="2019-11-07T22:19:22Z" level=info msg="Found new storjlabs/storagenode:beta image (sha256:5720335ea79bc71b0df47d5a3635a3ebfd4b6cfa23cd4bb2714fea5d45b82039)"
time="2019-11-07T22:19:29Z" level=info msg="Stopping /storagenode (2954845bff7fe0661f714013632748bd686309dd8b8d950b153fcf9aed4bb841) with SIGTERM"
time="2019-11-07T22:19:32Z" level=info msg="Creating /storagenode"
time="2019-11-20T22:19:21Z" level=info msg="Found new storjlabs/storagenode:beta image (sha256:6b8542c78084bad2a728534e6e8a1a1f2846d8732444e5cc61b16a07fee228dd)"
time="2019-11-20T22:19:28Z" level=info msg="Stopping /storagenode (78af6ecd522b4bf52c2b1636fd41c488271303a91de1f2e76985f48588ee4c60) with SIGTERM"
time="2019-11-20T22:19:33Z" level=info msg="Creating /storagenode"
time="2019-12-11T22:19:23Z" level=info msg="Found new storjlabs/storagenode:beta image (sha256:028b6f8a7253a197d2a46bf4583b7a6e2508f6d18408caa350b96afe0789ee3c)"
time="2019-12-11T22:19:24Z" level=info msg="Stopping /storagenode (f231fa57364590911038cbf67a1819bfd27d52f72be4a0bfa99b9122079dbbf8) with SIGTERM"
time="2019-12-11T22:19:30Z" level=info msg="Creating /storagenode"
time="2019-12-18T22:19:22Z" level=info msg="Found new storjlabs/storagenode:beta image (sha256:265e696b18b968b60e647504e1e3b79063e37e47babf2487b8da4c32edf18498)"
time="2019-12-18T22:19:28Z" level=info msg="Stopping /storagenode (c921cb64bcafec9b559004c4705e3d191eb0370c81e94324c591a1395be03c84) with SIGTERM"
time="2019-12-18T22:19:33Z" level=info msg="Creating /storagenode"
time="2020-01-01T16:19:28Z" level=info msg="Found new storjlabs/storagenode:beta image (sha256:fae06a84e7ba253abbd5f0597db83daa62b9e6a4806bf7df5a28b76fc3fac053)"
time="2020-01-01T16:19:35Z" level=info msg="Stopping /storagenode (607dd9e095e774e1f3b006252d2d736930569183c0e3aab15bdb830d12aefc57) with SIGTERM"
time="2020-01-01T16:19:41Z" level=info msg="Creating /storagenode"
time="2020-01-08T04:19:23Z" level=info msg="Unable to update container /storagenode. Proceeding to next."
time="2020-01-08T04:19:33Z" level=info msg="Unable to update container /watchtower. Proceeding to next."
Failed to send notification email:  dial tcp: lookup smtp.gmail.com on 8.8.8.8:53: read udp 172.17.0.3:42105->8.8.8.8:53: i/o timeout
time="2020-01-09T22:19:36Z" level=info msg="Found new storjlabs/storagenode:beta image (sha256:5a015f5071093df0c3bb557d483cf3ba14300adc3487f8789fae6228259c44ba)"
time="2020-01-09T22:19:44Z" level=info msg="Stopping /storagenode (ff557827177301078f1bda5853ba0eb9c5538d686758e810e589b8b4ee32b1bb) with SIGTERM"
time="2020-01-09T22:19:53Z" level=info msg="Creating /storagenode"
time="2020-01-23T22:19:37Z" level=info msg="Found new storjlabs/storagenode:beta image (sha256:6d7a3cb013baf083b4c0035c777a179fa70dc94c26ac8b7f6e5ba362e4950794)"
time="2020-01-23T22:19:39Z" level=info msg="Stopping /storagenode (8af7a1b05dcae8990ca0b2a226aa9c05442e9aa41737c7be9361561223f694b0) with SIGTERM"
time="2020-01-23T22:19:45Z" level=info msg="Creating /storagenode"
time="2020-01-30T04:19:30Z" level=info msg="Found new storjlabs/storagenode:beta image (sha256:9b1973c3147fd990fc26bb3ae1c9170ced17a313ce091bba7734ec2b1dae7ca2)"
time="2020-01-30T04:19:32Z" level=info msg="Stopping /storagenode (2481623a59ce45747539a4816f3f921c1939ef711c5394b0bac1208c2535e6da) with SIGTERM"
time="2020-01-30T04:19:37Z" level=info msg="Creating /storagenode"
time="2020-01-30T22:19:37Z" level=info msg="Found new storjlabs/storagenode:beta image (sha256:6853e774b127fe85b7741d64fca17f9b373e6654435fc323d3d0a24d001749a1)"
time="2020-01-30T22:19:44Z" level=info msg="Stopping /storagenode (c5bec5af44164757ca45f19c5e238bf79e0ebfbc8b36e41cd58f7661989d6ee1) with SIGTERM"
time="2020-01-30T22:19:51Z" level=info msg="Creating /storagenode"
time="2020-02-19T16:19:35Z" level=info msg="Found new storjlabs/storagenode:beta image (sha256:07d3705be30e007335b579827da9e652c83e990dc561577f5e32f95774227010)"
time="2020-02-19T16:19:42Z" level=info msg="Stopping /storagenode (1e2234dc319422b8b0789ae75a662ae89f1d448178011d8f45509c7b9df93da1) with SIGTERM"
time="2020-02-19T16:19:51Z" level=info msg="Creating /storagenode"
time="2020-02-19T22:19:35Z" level=info msg="Found new storjlabs/watchtower:latest image (sha256:97f431810a13b08011792f04d0a01a30a65a26ec9d5c470f55cd7d68c4c4f6c6)"
time="2020-02-19T22:19:35Z" level=info msg="Creating /watchtower"
time="2020-03-10T22:19:36Z" level=info msg="Found new storjlabs/storagenode:beta image (sha256:9a38b6e84751fdf83b7a60ce2f5cbb8b8ba71691be6309cbb1e54eb56249b0ec)"
time="2020-03-10T22:19:36Z" level=info msg="Stopping /storagenode (ea53b328f26790b14e47a9d3d36a7d9bd6d387b506804e55f41be0984f266903) with SIGTERM"
time="2020-03-10T22:19:43Z" level=info msg="Creating /storagenode"
1 Like
4

When rebooting, docker must have started all current containers.

I believe the current image ID is 6af6621e20c1

  1. Stop the old watchtower:

docker stop eee9e0d212d1

  1. Remove the old watchtower container

docker rm eee9e0d212d1

  1. Remove the old watchtower image

docker image prune

  1. Verify watchtower image ID:

docker images

  1. Verify new watchtower still running:

docker ps

5

this isnt so much about what the fix is, thats trivial enough, but rather about the old watchtower image spinning up new watchtower containers when theres both new storagenode and watchtower images.
ive done no reboots of services and/or cold reboots.

6

This sort of thing seems to happen from time to time here:

As to why… I don’t have an answer.

7

for as long as its on here as info, we should be good