Watchtower spawning copies of itself

[tyakimov]

Yo, I also had the same issue on ARM. I believe the issue is present on all systems having scanned a number of post regarding rogule watchtower containers spinning up.

In one of the earlier threads @BrightSilence pointed out that we should use the --cleanup flag with watchtower to suppers the explosion. That works, but with quirks also

I was running with the --clean flag on watchtower before last update from 12-14 hrs ago on storagenode and what happened is that watchtower carried out the upgrade but then it stopped itself.

So, it does the reverse with that flag, prevents extra copies but kills itself. xd.

For reference, this is how I’m running watchtower atm, so its an ez solution, keep an eye for the next upgrade e-mail, then take care to start it up, it literally wiped its own container clean, could not find it in docker ps -a:

docker run -d \
--restart always \
--name watchtower \
-v /var/run/docker.sock:/var/run/docker.sock \
-v /etc/ssl/certs:/etc/ssl/certs \
-e WATCHTOWER_TIMEOUT=300s \
-e WATCHTOWER_NOTIFICATIONS=email \
-e WATCHTOWER_NOTIFICATION_EMAIL_FROM=waldo \
-e WATCHTOWER_NOTIFICATION_EMAIL_TO=waldo \
-e WATCHTOWER_NOTIFICATION_EMAIL_SERVER=smtp.gmail.com \
-e WATCHTOWER_NOTIFICATION_EMAIL_SERVER_USER=waldo \
-e WATCHTOWER_NOTIFICATION_EMAIL_SERVER_PASSWORD=waldo \
storjlabs/watchtower:latest \
storagenode watchtower --stop-timeout 300s --interval 21600 --cleanup
                                                                 ^magic sauce

[voltage]

@tyakimov thank you, i’ve tried magic “–cleanup” - did not helped.

[voltage]

@littleskunk have no idea how to fix it. Non of solutions, that worked to other - does not work in my case

Yesterday i was trying until 2am trying to restart the docker, but could not…

[tyakimov]

aw, too bad, so perhaps there are platform-specific symptoms as well then. would not know what the watchtower version I was running 12hrs ago was to give more debug info… right now tho, i’m on:
root@storj:~# docker image history storjlabs/watchtower:latest
IMAGE CREATED CREATED BY SIZE COMMENT
1a50696a5aae 45 hours ago /bin/sh -c #(nop) ENTRYPOINT ["/watchtower"] 0B
45 hours ago /bin/sh -c #(nop) COPY file:beb92e145ecb3ddb… 11.4MB
2 days ago /bin/sh -c #(nop) COPY dir:c28957be915afa145… 1.17MB
2 days ago /bin/sh -c #(nop) COPY file:091d2a65366d205f… 233kB
2 days ago /bin/sh -c #(nop) LABEL io.storj.watchtower… 0B

edit: welp, looking at this i must have been on the same one as now.

[Odmin]

Guys, I think the key magic was broken on watchtower side (old release) and fixed on current watchtower release.

You can add --debug and see this:

So, watchtower should check for another running watchtower instances after upgrade himself.

[tyakimov]

can confirm this is already present in watchtower 1a50696a5aae 45 hours ago. I will remove my --cleanup flag and see next update xd.

[tyakimov]

oh, i member now. i had a bomb of watchtower mails 1-2 weeks ago from watchtower failing to upgrade itself. So then i did the upgrade manually. Perhaps the original poster was in the same situation as me, but didn’t have watchtower mail notifications and never noticed the build-up of watchtower instances, all of which then detected the yet newer version 45hrs and and doughnuts hit the fan.

[sorry2xs]

good morning I woke up and my windows node updated over night to 0.22.1 the cli dashboard no longer displays bootstrap info is that normal, the Mac OS node has not updated yet

[tyakimov]

hi, yas, that’s ok last contract static now, neighborhood gone.

[BrightSilence]

@tyakimov

In one of the earlier threads @BrightSilence pointed out that we should use the --cleanup flag with watchtower to suppers the explosion. That works, but with quirks also

That wasn’t me and it also wouldn’t work since this only cleans up old images, NOT containers. The container duplication is a bug in watchtower.

[anon27637763]

@voltage

SiaCoin, FileCoin, MaidSAFE

SC nodes require buy in first… you need to fund your operator wallet with SC… which needs to be purchased at an exchange, most likely Poloniex. Once you’ve funded your wallet, you can advertise your node on their network and receive contracts for your resources.

Filecoin is not yet fully functional, as far as I know.

I don’t know much about MaidSAFE…

There’s also SWARM. That’s what I was looking at running myself, as it utilizes the ENS architecture and has a decent shot at replacing much of the Internet infrastructure… and how everything communicates. However, I think the minimum buy-in was $2500 for a qualified node operator.

Storj is a happy medium at the moment. It’s not too drastic a change from “How things are run right now” … there is no immediate buy-in to run a node and get started playing with the tech on a live network… and it’s not too difficult to setup from a technical standpoint by comparison with many other projects starting up on decentralization.

If you want to read a little bit about some of the various projects, I found this hackernoon article to be quite informative.

[cdhowie]

@voltage

I seen lot’s of people, including my self, loosing nodes because of docker.
Most of the problems i had - was with docker. And now - docker again.
I can’t understand anymore is this project for fun or somebody is trying to make a real thing? If it’s a joke, then it’s waisted too many time. If this is not a joke, why the hell Storj is still using “docker”?

I don’t think anyone is having issues with Docker itself, they’re having issues with the images that run in Docker containers. This is like saying that you’re having “never ending problems with Windows” because Chrome is misbehaving.

Docker just runs containers from images. Defective images are not a problem with Docker any more than bugs in Chrome indicate a problem with Windows.

Docker is used in production by huge companies to manage applications that run on entire fleets of servers. My company uses it quite successfully.

I’m not saying Docker is perfect, but I haven’t seen anything crop up yet on these forums that’s an issue with Docker itself. They’re always a watchtower bug, a storagenode bug, or operator error (misconfiguration, incorrect commands, etc.).

[tankmann]

Same here on Synology. Will revert back to original command.

[kevink]

This has become a whining thread for some and is heavily off-topic now. Let’s stay focused here.
You can always create a new thread to air your problems about disqualification, frustration with docker, etc…

[donald.m.motsinger]

@voltage

“docker ps | grep watchtower | awk ‘{print “docker rm --force”, $1}’ | bash
awk: cmd. line:1: ‘{print
awk: cmd. line:1: ^ invalid char ‘?’ in expression”

I’m surprised that someone, who “runs servers in 8 different geographical locations around the country”, can’t understand clear error messages. Your single and double quotes are wrong (“invalid char”) and Alexey pointed that out to you in the very next post, yet you still keep ranting about it here…

[Alexey]

Please, do not respond on this scream of soul, the author of the screaming comment created a separate thread again Watchtower goes crazy! All MEM & CPU in use (ubuntu) - #2 by nerdatwork and solved the problem by uninstalling the docker and installing it back.
This would be my second suggestion, if the one-line-killer doesn’t help. Unfortunately the author was frustrated and his mind was closed.

The only thing I would mention that uninstalling should be with purge.
There is another workaround - find out and remove all watchtower containers from the filesystem:

Watchtower spawning copies of itself

I also had two nodes totally taken out by this. The nodes were so overrun by watchtower instances that they stopped responding. Docker also crashes so running docker commands resulted in failure to open the docker unit socket. I was able to recover them manually but it took a long time and my uptime rep is pretty much destroyed for the time being. I also had to yank power a few times so I’m hoping I don’t have corrupted data at this point… To resolve, I had to: Reboot the node cd to my /…

, but it would be longer than just reinstall.

[cdhowie]

@Alexey

There is another workaround - find out and remove all watchtower containers from the filesystem

@Alexey

but it would be longer than just reinstall

Couldn’t one just run this? Much easier than a reinstall.

docker ps --format '{{.ID}}|{{.Command}}' | grep /watchtower | cut -d\| -f1 | xargs docker rm -f

[Alexey]

Yes, it is or script from the solution. IF the docker is still responding.

[cameron]

Yes, as part of the replacement of kademlia, the bootstrap node has been removed

[selsrog]

I too was hit by watchtower spawning.

CPU 100%. Tried countless things but eventually restored an earlier snapshot.

Guess my reputation has taken a serious beating by this… (also due to the fact that I noticed it quiet late, as both SNOboard and UptimeRobot indicated my node was running just fine…).