Watchtower goes crazy! All MEM & CPU in use (ubuntu)

When tried to kill:


docker ps | grep watchtower | awk ‘{print “docker rm --force”, $1}’ | bash
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

docker ps -a | awk ‘{ print $1,$2 }’ | grep watchtower:latest | awk ‘{print $1 }’ | xargs -I {} docker stop {}
awk: awk: cmd. line:cmd. line:1: 1: ‘{print
‘{awk:
cmd. line:awk: 1: cmd. line:^ invalid char ‘?’ in expression1:
^ invalid char ‘?’ in expression
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?


sudo service docker status
● docker.service - Docker Application Container Engine
Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
Active: deactivating (stop-sigterm)
Docs: https://docs.docker.com
Main PID: 1026 (dockerd)
Tasks: 414
CGroup: /system.slice/docker.service
├─ 1026 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
├─15068 /usr/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 28967 -container-ip 172.17.0.134 -container-port 28967
└─15082 /usr/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 14002 -container-ip 172.17.0.134 -container-port 14002

Oct 04 08:37:51 raid6 dockerd[1026]: time=“2019-10-04T08:37:51.968777592+03:00” level=warning msg="dca890e5c7ffcfd317122de1b38b0072f64845dd79ddbe00573630
Oct 04 08:37:52 raid6 dockerd[1026]: time=“2019-10-04T08:37:52.112293308+03:00” level=info msg=“ignoring event” module=libcontainerd namespace=moby topic
Oct 04 08:37:52 raid6 dockerd[1026]: time=“2019-10-04T08:37:52.942009569+03:00” level=info msg=“ignoring event” module=libcontainerd namespace=moby topic
Oct 04 08:37:55 raid6 dockerd[1026]: time=“2019-10-04T08:37:55.245621775+03:00” level=warning msg="338423d19aed2809d98eeecac7d2bad5d101695a9408f705638b05
Oct 04 08:37:56 raid6 dockerd[1026]: time=“2019-10-04T08:37:56.138126347+03:00” level=warning msg="2e4bc42c49ae973578114da0e7a3eddd7db4f9babbc9a181347e9d
Oct 04 08:37:57 raid6 dockerd[1026]: time=“2019-10-04T08:37:57.341281501+03:00” level=info msg=“ignoring event” module=libcontainerd namespace=moby topic
Oct 04 08:37:58 raid6 dockerd[1026]: time=“2019-10-04T08:37:58.109719172+03:00” level=warning msg="cd13462cde421860742aff034c8f01b33798266f1f78b1bdf1cce7
Oct 04 08:38:05 raid6 dockerd[1026]: time=“2019-10-04T08:38:05.706664839+03:00” level=warning msg="3998f0186c0b4cc80916a7a9f04ff917578d28b8bb196edf72948f
Oct 04 08:38:13 raid6 dockerd[1026]: time=“2019-10-04T08:38:13.979896568+03:00” level=info msg=“ignoring event” module=libcontainerd namespace=moby topic
Oct 04 08:38:21 raid6 dockerd[1026]: time=“2019-10-04T08:38:21.504988159+03:00” level=info msg=“ignoring event” module=libcontainerd namespace=moby topic


sudo service docker start
Job for docker.service canceled.


Restart of the machine does not helped as well.

Any ideas?

You need to search your issue before replicating the same post. Your previous post was merged here:

Help Storj in keeping this forum clean of repetitive posts. :slight_smile:

2 Likes

You need to replace all the " ` " marks with a “single quote” in the command, Something in the forum messes up “single quote mark” and throws an error when you run it.

1 Like

@S0litiare thank you, i’ve notice this and fixed quote, but it does not help as the docker is not running:

docker ps -a | awk ‘{ print $1,$2 }’ | grep watchtower:latest | awk ‘{print $1 }’ | xargs -I {} docker stop {}
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

Tried all possible combinations:

sudo service docker start <-- this command freezes forever (or until cmd+c) nothing happening
sudo service docker stop <-- this command freezes, forever (or until cmd+c) with error:
Job for docker.socket failed.
See “systemctl status docker.socket” and “journalctl -xe” for details.


systemctl status docker.socket
● docker.socket - Docker Socket for the API
Loaded: loaded (/lib/systemd/system/docker.socket; enabled; vendor preset: enabled)
Active: inactive (dead) since Fri 2019-10-04 11:18:58 EEST; 2h 30min ago
Listen: /var/run/docker.sock (Stream)

Oct 04 00:58:55 raid6 systemd[1]: Starting Docker Socket for the API.
Oct 04 00:58:55 raid6 systemd[1]: Listening on Docker Socket for the API.
Oct 04 11:18:58 raid6 systemd[1]: Closed Docker Socket for the API.
Oct 04 13:45:48 raid6 systemd[1]: docker.socket: Socket service docker.service already active, refusing.
Oct 04 13:45:48 raid6 systemd[1]: Failed to listen on Docker Socket for the API.
Oct 04 13:47:05 raid6 systemd[1]: docker.socket: Socket service docker.service already active, refusing.
Oct 04 13:47:05 raid6 systemd[1]: Failed to listen on Docker Socket for the API.
Oct 04 13:49:18 raid6 systemd[1]: docker.socket: Socket service docker.service already active, refusing.
Oct 04 13:49:18 raid6 systemd[1]: Failed to listen on Docker Socket for the API.

Yesterday i’ve seen tens of thausands of docker watchtower procesess… restart of the server does not help.

If you reboot the server, can you get “docker ps” to run?

Docker stucked.
I can’t start it, i can’t stop it, i can’t uninstall it…

Will try in evening, but today is not that evening, where i would like to spend with docker…

sudo apt-get autoremove -y --purge docker-engine docker docker.io docker-ce
Reading package lists… Done
Building dependency tree
Reading state information… Done
Package ‘docker-engine’ is not installed, so not removed
Package ‘docker’ is not installed, so not removed
Package ‘docker.io’ is not installed, so not removed
The following packages will be REMOVED:
docker-ce*
0 upgraded, 0 newly installed, 1 to remove and 3 not upgraded.
1 not fully installed or removed.
After this operation, 109 MB disk space will be freed.
dpkg: error processing package docker-ce (–remove):
package is in a very bad inconsistent state; you should
reinstall it before attempting a removal
Errors were encountered while processing:
docker-ce
E: Sub-process /usr/bin/dpkg returned an error code (1)

First reboot the machine.

Once the machine is rebooted run docker ps and see if that gives you anything. Since you tried to uninstall and it failed it might throw up an error. That’s OK.

Just reinstall docker :
sudo apt-get install docker-engine docker docker.io docker-ce

Then reboot the machine once more.

Then run : docker ps and see what that gives you.

Thank you, but did not worked out:

sudo apt-get install docker-engine docker docker.io docker-ce
Reading package lists… Done
Building dependency tree
Reading state information… Done
Package docker-engine is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source
However the following packages replace it:
docker-ce

E: Package ‘docker-engine’ has no installation candidate

trying other ways…:

sudo apt-get install docker-ce docker-ce-cli containerd.io
Reading package lists… Done
Building dependency tree
Reading state information… Done
containerd.io is already the newest version (1.2.6-3).
containerd.io set to manually installed.
docker-ce-cli is already the newest version (5:19.03.2~3-0~ubuntu-bionic).
docker-ce-cli set to manually installed.
Recommended packages:
aufs-tools cgroupfs-mount | cgroup-lite pigz
The following packages will be upgraded:
docker-ce
1 upgraded, 0 newly installed, 0 to remove and 3 not upgraded.
1 not fully installed or removed.
Need to get 0 B/22.7 MB of archives.
After this operation, 91.1 kB of additional disk space will be used.
Do you want to continue? [Y/n] y
(Reading database … 148179 files and directories currently installed.)
Preparing to unpack …/docker-ce_5%3a19.03.2~3-0~ubuntu-bionic_amd64.deb …
Unpacking docker-ce (5:19.03.2~3-0~ubuntu-bionic) over (5:19.03.1~3-0~ubuntu-bionic) …
Setting up docker-ce (5:19.03.2~3-0~ubuntu-bionic) …
A dependency job for docker.service failed. See ‘journalctl -xe’ for details.
invoke-rc.d: initscript docker, action “start” failed.
● docker.service - Docker Application Container Engine
Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
Active: inactive (dead)
Docs: https://docs.docker.com

Oct 04 16:02:25 raid6 systemd[1]: Dependency failed for Docker Application Container Engine.
Oct 04 16:02:25 raid6 systemd[1]: docker.service: Job docker.service/start failed with result ‘dependency’.
Oct 04 18:58:50 raid6 systemd[1]: Dependency failed for Docker Application Container Engine.
Oct 04 18:58:50 raid6 systemd[1]: docker.service: Job docker.service/start failed with result ‘dependency’.
dpkg: error processing package docker-ce (–configure):
installed docker-ce package post-installation script subprocess returned error exit status 1
Processing triggers for systemd (237-3ubuntu10.29) …
Processing triggers for ureadahead (0.100.0-21) …
ureadahead will be reprofiled on next reboot
Errors were encountered while processing:
docker-ce
E: Sub-process /usr/bin/dpkg returned an error code (1)

piece by piece removed all docker and all it’s things arround.
Reinstalled,
Node started, but already 2min. OFFLINE, watching log’s…

This node finished.
Have another with similar problem… :expressionless:
will have to drive tomorrow to the place, as over VPN only ping responds, ssh takes ages to connect and ages to run any of the commands, because of the same problem… probably…