Weird behavior: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

Flow · February 24, 2020, 2:21am

Hi everyone, new user but was on rocketchat before…
I have been running my node since may’19 I believe, everything went fine.
Now I have no idea how to fix my issue.
I run my node on Debian 9.9.0, I am not very much linux savvy but I try.
I have Watchtower installed and use ssh to check the dashboard.
When I came back today I noticed that my ssh dashboard was terminated, as it would do when there was an update from Watchtower. However it was not it. I tried to start the dashboard and it throws `root@flux:/var/lib/docker/overlay2# docker exec -it storagenode /app/dashboard.sh
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

So top shows that the dockerd, containerd and storagenode are running.
I tried to reboot and got a timeout on trying to unmount the whole bunch of overlay2/…/merged folders.
I had never seen those before. After reboot the same daemons are running but I still get the Cannot connect to Docker error. It seems that for some reason some folders/mount points would be messed up. Look at everything in df -h :

overlay          35G   26G  7.6G  78% /var/lib/docker/overlay2/22971e9dbc2adb453fc278d027500904608a53e4e77bad8bb3a37b21adb9ef98/merged
overlay          35G   26G  7.6G  78% /var/lib/docker/overlay2/24a0641cd8707c01a4bbb7252640186ab3ec0ac0f647d95d92555642faa92e47/merged
overlay          35G   26G  7.6G  78% /var/lib/docker/overlay2/21fb9c1a9b73a5368f87a9ce21b91407b479e8bdb7f84de23e2bbf9ae1a13115/merged
overlay          35G   26G  7.6G  78% /var/lib/docker/overlay2/f9e3444713604491ecfa9c5048feb33dffe8edf0c811b72c0541c8a8ba040b24/merged
overlay          35G   26G  7.6G  78% /var/lib/docker/overlay2/d6fce5a133b792ff0dd0bde89867dcd140953f1a0da85b1784819a5a993402ce/merged
overlay          35G   26G  7.6G  78% /var/lib/docker/overlay2/9b6a3bbf0d88a6c05c2752dde9a6121845abaa6b12534990d8c94eec76d73078/merged
overlay          35G   26G  7.6G  78% /var/lib/docker/overlay2/16f8d74e9c89ac5789c0f70534d60c192b5b0de6b5e70411f192e75dd7e28ae5/merged
overlay          35G   26G  7.6G  78% /var/lib/docker/overlay2/7f1c9571038d3364763cdf2d6eec82dd9a59d1ef3eaf7fc0b1d1aad1f86ef9a6/merged
overlay          35G   26G  7.6G  78% /var/lib/docker/overlay2/cfbbeae0f607ddf4e0533bb71f0b1297cd3db3e46f84b4297fb3b0b5c003f9f6/merged
overlay          35G   26G  7.6G  78% /var/lib/docker/overlay2/6c476e7d470be8784d90e345b141457c6c4740bd68b1b630302cab2df6ad190a/merged
overlay          35G   26G  7.6G  78% /var/lib/docker/overlay2/aedc507df031f70e2fedfa1048abc67b3742a9e7f3f4a7c84f7f0493ee4bbc23/merged
overlay          35G   26G  7.6G  78% /var/lib/docker/overlay2/fced116e1a8bbaf8045d8f890f00347c197714d5451d83c74b8ac25070b9ba38/merged
overlay          35G   26G  7.6G  78% /var/lib/docker/overlay2/e73f86f8f45392ffba5bbe059e560f982f8c9edacc53045893fd33a9bfc518ce/merged
overlay          35G   26G  7.6G  78% /var/lib/docker/overlay2/21bf1ebb0aaca0b17ab738f34ec11452fc724bf4731589a4a416101b363b970a/merged
overlay          35G   26G  7.6G  78% /var/lib/docker/overlay2/a61d769916a2fc305ce16a0cbba6370206d687b92b6602e71a265026891e8c26/merged
overlay          35G   26G  7.6G  78% /var/lib/docker/overlay2/50381dfc3fdbcd0de001a18f20ab0edadd6cdfaaf36a8db44f982549f3c43beb/merged
overlay          35G   26G  7.6G  78% /var/lib/docker/overlay2/e64db6e3cc52c20a72fbd803221cc8841d57b1bc0325ecce6da015934c7fa7e1/merged
shm              64M     0   64M   0% /var/lib/docker/containers/49318e9ed43bf6e3b57e5575d52e5ee635f072a6add64af7233fc91cae396749/mounts/shm
shm              64M     0   64M   0% /var/lib/docker/containers/9f1843d115f2c8dc8b8e4155e24498adbde4cd68250c418f908cdc30b5ffc943/mounts/shm
shm              64M     0   64M   0% /var/lib/docker/containers/a69a4d5ed0e928837523cd15dcf1eccf44caca6f6c03d4114446f6a1a96b8c41/mounts/shm
shm              64M     0   64M   0% /var/lib/docker/containers/ac1b4f396a304dbf415bee7545cadb7a68cb8ba7943b8f23b0ec91a7fff4a3f4/mounts/shm
shm              64M     0   64M   0% /var/lib/docker/containers/c8953d47c04302e64d300a639970e2e3bdabe90f9646e0a10f681d2a0af51ffd/mounts/shm
shm              64M     0   64M   0% /var/lib/docker/containers/d272082f5fb7806195c48e20e693b8b7046f4b4d3f307ab8a13d5e0a1c1156a5/mounts/shm
shm              64M     0   64M   0% /var/lib/docker/containers/6fb673dfe203ddb79421f0763cc8a7492545808c2a2938ccde66fb85f4549c99/mounts/shm
shm              64M     0   64M   0% /var/lib/docker/containers/2b8154f5d1e563c28452d024b01f35884752e396a76633e928075abb60c2762d/mounts/shm
shm              64M     0   64M   0% /var/lib/docker/containers/34fd25b9c6e44461aaafc6d7b4fcaa64e3c5d70da16bf2cf66bac06b8e07789c/mounts/shm
shm              64M     0   64M   0% /var/lib/docker/containers/ed4274e1a30263fb23bb0f2111740b871149da044296058920bf4af1155f631e/mounts/shm
shm              64M     0   64M   0% /var/lib/docker/containers/6d92fc27b7dcc5d8565b6cb60ac87b9ec86fcc7d1efa25faff7fa781ae5f9246/mounts/shm
shm              64M     0   64M   0% /var/lib/docker/containers/fbe001588940eb7fddaa3a94a1d055d40839294dae6e9d144aa77ac1840d1cdc/mounts/shm
shm              64M     0   64M   0% /var/lib/docker/containers/b4be0d9fbba6d8fcc29fdefd2a751c0582a75e5ab63c0059782685723d16a7f8/mounts/shm
shm              64M     0   64M   0% /var/lib/docker/containers/b5f56240031931e9473d28c0d49562f8331b0e409773acfa2320d7a59cf169e9/mounts/shm
shm              64M     0   64M   0% /var/lib/docker/containers/66e40e73704b15e30a45ce213e45aa8a366458b43c8ee5434954abd239e187df/mounts/shm
shm              64M     0   64M   0% /var/lib/docker/containers/5424b2606fd0e28c9856a3e983099855f269717c0cd447ca21a50698e8856207/mounts/shm
shm              64M     0   64M   0% /var/lib/docker/containers/40b54d46969c068a032e76c2c695d9b3c62afad6927ad707d2c4f3d38de975c2/mounts/shm
shm              64M     0   64M   0% /var/lib/docker/containers/db5f1e90c38c53bba192cf3414c647dfe6f16df2859cd28a2f49d76dfdc3798f/mounts/shm
shm              64M     0   64M   0% /var/lib/docker/containers/f503815f1730de60b70fcd616f665c40ed0910e984e5208779816a94806b8940/mounts/shm
shm              64M     0   64M   0% /var/lib/docker/containers/58f242f8dbd53b28bccd57996a871aef9186089346f6f6c4c845cc7646fa4db0/mounts/shm
shm              64M     0   64M   0% /var/lib/docker/containers/01f15e27969fcdba59539123a6be5604907c041d9d4abc1cb08d7ee7ce19f3e2/mounts/shm

overlay          35G   26G  7.6G  78% /var/lib/docker/overlay2/3c8cb4e4caa1f736dc98ee0ce7aefeb91fc3d850a22698c4afa84aaae94cd0d7/merged
'overlay          35G   26G  7.6G  78% /var/lib/docker/overlay2/678ddb39ea8686d718b54832375dba76dc183218eb5fccf5223da5df9a2835c8/merged
shm              64M     0   64M   0% /var/lib/docker/containers/00fb5c4d0dfb064aeac89dac3d788304654cca073eb9d76706d2aec2d47cb466/mounts/shm
overlay          35G   26G  7.6G  78% /var/lib/docker/overlay2/d3b4055d37dc24d2409e6d9da54d7cc5576f8185a38a3dbb1c5cd1ab476c5438/merged
shm              64M     0   64M   0% /var/lib/docker/containers/56a9ffade6199aee3f2bbaf3f3c8101a911051ecfd8d289b3d47823405337547/mounts/shm

I hope the formatting will come out right… I had to remove alot so I could post but there’s 900 of those.
So whenever I restart the computer now it gets stuck on some containers trying to unmount them, like I said theres about 900 of those and I wonder if this is normal behavior?
I could not upgrade docker through apt upgrade I had to kill the storagenode, dockerd and containerd processes running otherwise it would just remain stuck. Here is the output of dpkg :

Job for docker.service canceled.
invoke-rc.d: initscript docker, action "start" failed.
● docker.service - Docker Application Container Engine
   Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
   Active: deactivating (stop-sigterm)
     Docs: https://docs.docker.com
 Main PID: 25742 (dockerd)
    Tasks: 759
   Memory: 285.3M
      CPU: 27min 54.031s
   CGroup: /system.slice/docker.service
           └─25742 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock

Feb 23 21:12:06 flux dockerd[25742]: time="2020-02-23T21:12:06.084498455-05:00" level=info msg="ignoring event" module=libcontain…askDelete"
Feb 23 21:12:07 flux dockerd[25742]: time="2020-02-23T21:12:07.357226020-05:00" level=info msg="ignoring event" module=libcontain…askDelete"
Feb 23 21:12:07 flux dockerd[25742]: time="2020-02-23T21:12:07.602432900-05:00" level=error msg="stream copy error: read /proc/se…dy closed"
Feb 23 21:12:07 flux dockerd[25742]: time="2020-02-23T21:12:07.602513829-05:00" level=error msg="stream copy error: read /proc/se…dy closed"
Feb 23 21:12:07 flux dockerd[25742]: time="2020-02-23T21:12:07.602594562-05:00" level=info msg="blockingPicker: the picked transp…odule=grpc
Feb 23 21:12:07 flux dockerd[25742]: time="2020-02-23T21:12:07.602813537-05:00" level=error msg="failed to get event" error="rpc …space=moby
Feb 23 21:12:07 flux dockerd[25742]: time="2020-02-23T21:12:07.603582879-05:00" level=error msg="stream copy error: reading from …osed fifo"
Feb 23 21:12:07 flux dockerd[25742]: time="2020-02-23T21:12:07.604241019-05:00" level=error msg="stream copy error: reading from …osed fifo"
Feb 23 21:12:07 flux dockerd[25742]: time="2020-02-23T21:12:07.604270810-05:00" level=error msg="failed to get event" error="rpc …ugins.moby
Feb 23 21:12:07 flux dockerd[25742]: time="2020-02-23T21:12:07.647995507-05:00" level=info msg="Processing signal 'terminated'"
Hint: Some lines were ellipsized, use -l to show in full.
dpkg: error processing package docker-ce (--configure):
 subprocess installed post-installation script returned error exit status 1

I did let it try about 30 minutes before killing the pid’s.

I think that storagenode is working but I have no way of knowing as I cannot launch the dashboard in ssh nor in a browser.

I need help peeps! Don’t want to loose my node!

nerdatwork · February 24, 2020, 3:04am

Do you use -v or -- mount in your docker run command ?

Flow · February 24, 2020, 3:41am

docker run -d --restart unless-stopped -p 28967:28967     -e WALLET=""     -e EMAIL=""     -e ADDRESS=":28967"     -e BANDWIDTH="11TB"     -e STORAGE="10TB"     -v "/home/flow/.local/share/storj/identity/storagenode":/app/identity     -v "/mnt/16TB/Storj":/app/config     --name storagenode storjlabs/storagenode:alpha

This is my run command minus wallet, email.
I do use -v for /mnt…

I still have storagenode running as a process, does this mean that it is running?
How can I verify if I can’t run any docker commands?

nerdatwork · February 24, 2020, 4:00am

That -v is the issue here. Docker run command has since been updated.

docker run -d --restart unless-stopped -p 28967:28967 -p 127.0.0.1:14002:14002 -e WALLET=“0xXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX” -e EMAIL="user@example.com" -e ADDRESS=“domain.ddns.net:28967” -e BANDWIDTH=“20TB” -e STORAGE=“2TB” --mount type=bind,source=“”,destination=/app/identity --mount type=bind,source=“”,destination=/app/config --name storagenode storjlabs/storagenode:beta

Reference:

Flow · February 24, 2020, 4:03am

Oh ok, I will change that, but this would fix the folders/mount issue I guess?

Flow · February 24, 2020, 4:05am

How would I prevent it from starting on boot?

nerdatwork · February 24, 2020, 4:06am

What do you want to prevent ?

Flow · February 24, 2020, 4:08am

Well right now from a fresh reboot I cannot enter any docker commands…

nerdatwork · February 24, 2020, 4:10am

You should immediately stop your node, remove the container and update the docker run command. You are risking your data by using -v option.

docker stop -t 300 storagenode
docker rm storagenode
enter your updated docker run command with --mount option

Flow · February 24, 2020, 4:13am

Yes but I cannot run ANY docker commands!
I figured if I prevent docker to start automatically I could update it and it’s run command.

Flow · February 24, 2020, 4:18am

See, the daemons and containers are working, even watchtower, but I cannot enter docker commands, the answer is always :

Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

Tasks: 469 total,   2 running, 467 sleeping,   0 stopped,   0 zombie
%Cpu(s): 30.6 us, 14.7 sy,  0.0 ni, 42.3 id, 12.2 wa,  0.0 hi,  0.2 si,  0.0 st
KiB Mem : 16466128 total,  5812148 free,  3360252 used,  7293728 buff/cache
KiB Swap:  1007612 total,  1007612 free,        0 used. 12579848 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
  525 root      20   0  806524 268400  14060 R  89.1  1.6  51:37.80 NetworkManager
    1 root      20   0  146212  14128   5320 S  40.9  0.1  40:33.64 systemd
 6688 flow      20   0   71332  12704   5608 S  38.0  0.1  36:48.14 systemd
  694 lightdm   20   0   71288  12680   5592 S  37.0  0.1  36:43.41 systemd
  727 root      20   0 7694092 146312  46056 S  12.2  0.9  14:50.02 dockerd
  560 root      20   0 5652708  67040  28732 S   4.6  0.4   5:14.24 containerd
 4342 root      20   0  140504  43992  19532 S   1.0  0.3   4:56.37 storagenode
  306 root      20   0   78976  21780  21236 S   2.0  0.1   2:36.51 systemd-journal
  518 message+  20   0   45488   4352   3492 S   1.7  0.0   1:56.62 dbus-daemon
  335 root      20   0   46820   4440   2852 S   1.3  0.0   1:22.49 systemd-udevd
  513 root      20   0   46420   4852   4276 S   1.0  0.0   0:56.47 systemd-logind
 8436 flow      20   0   45460   4364   3172 R   1.0  0.0   0:43.87 top
13539 root      20   0 1172604 895888  10528 S   0.3  5.4   0:34.79 packagekitd
  515 root      20   0  250116   4100   2584 S   0.0  0.0   0:30.55 rsyslogd
  514 avahi     20   0   48160   4400   3104 S   0.3  0.0   0:27.85 avahi-daemon
  198 root      20   0       0      0      0 D   0.7  0.0   0:16.57 kworker/u16:8
  230 root       0 -20       0      0      0 S   0.0  0.0   0:16.45 kworker/6:1H
  278 root      20   0       0      0      0 S   0.3  0.0   0:15.04 jbd2/sdc3-8
    7 root      20   0       0      0      0 S   0.3  0.0   0:13.07 rcu_sched
  527 root      20   0  422608   9016   7228 S   0.3  0.1   0:11.54 ModemManager
 8718 root      20   0       0      0      0 S   0.0  0.0   0:09.34 kworker/u16:0
  203 root      20   0       0      0      0 S   0.0  0.0   0:05.84 kworker/3:2
25156 root      20   0       0      0      0 S   0.0  0.0   0:05.31 kworker/6:4
20961 root      20   0       0      0      0 S   0.0  0.0   0:05.23 kworker/1:0
16705 root      20   0       0      0      0 S   0.3  0.0   0:05.05 kworker/u16:1
   46 root      20   0       0      0      0 S   0.0  0.0   0:04.98 ksoftirqd/6
14532 root      20   0       0      0      0 S   0.0  0.0   0:04.49 kworker/4:1
28419 root      20   0       0      0      0 S   0.3  0.0   0:03.39 kworker/0:2
 3704 root      20   0       0      0      0 S   0.0  0.0   0:03.38 kworker/u16:2
12420 root      20   0       0      0      0 S   0.0  0.0   0:03.26 kworker/4:2
  699 lightdm   20   0  639056  74792  48896 S   0.0  0.5   0:02.95 lightdm-gtk-gre
   16 root      20   0       0      0      0 S   0.0  0.0   0:02.93 ksoftirqd/1
16811 root      20   0       0      0      0 S   0.0  0.0   0:02.85 kworker/1:3
31170 root      20   0       0      0      0 S   0.0  0.0   0:02.15 kworker/7:1
    3 root      20   0       0      0      0 S   0.0  0.0   0:02.05 ksoftirqd/0
   22 root      20   0       0      0      0 S   0.0  0.0   0:01.96 ksoftirqd/2
   34 root      20   0       0      0      0 S   0.0  0.0   0:01.93 ksoftirqd/4
24784 root      20   0       0      0      0 S   0.0  0.0   0:01.86 kworker/5:1
14004 root      20   0       0      0      0 S   0.0  0.0   0:01.84 kworker/6:0
  577 root      20   0  389140  53160  35636 S   0.0  0.3   0:01.72 Xorg
   28 root      20   0       0      0      0 S   0.0  0.0   0:01.64 ksoftirqd/3
25183 root      20   0       0      0      0 S   0.3  0.0   0:01.60 kworker/7:2
   40 root      20   0       0      0      0 S   0.0  0.0   0:01.58 ksoftirqd/5
   52 root      20   0       0      0      0 S   0.0  0.0   0:01.54 ksoftirqd/7
23532 root      20   0       0      0      0 S   0.3  0.0   0:01.47 kworker/5:3
17444 root      20   0       0      0      0 S   0.0  0.0   0:01.26 kworker/3:3
29659 root      20   0       0      0      0 S   0.0  0.0   0:01.25 kworker/7:4
  549 vnstat    20   0    7336    896    824 S   0.0  0.0   0:01.23 vnstatd
13428 root      20   0       0      0      0 S   0.0  0.0   0:01.21 kworker/0:3
13625 root      20   0       0      0      0 S   0.3  0.0   0:00.99 kworker/2:4
  237 root       0 -20       0      0      0 S   0.0  0.0   0:00.91 kworker/5:1H
17206 root      20   0       0      0      0 S   0.0  0.0   0:00.90 kworker/2:1
 7003 root      20   0       0      0      0 S   0.0  0.0   0:00.70 kworker/5:4
 7062 flow      20   0  101448   4940   3900 S   0.0  0.0   0:00.70 sshd
 3302 root      20   0       0      0      0 S   0.3  0.0   0:00.61 kworker/4:3
  232 root       0 -20       0      0      0 S   0.0  0.0   0:00.51 kworker/0:1H
19050 root      20   0       0      0      0 S   0.0  0.0   0:00.48 kworker/3:1
  231 root       0 -20       0      0      0 S   0.0  0.0   0:00.47 kworker/7:1H

nerdatwork · February 24, 2020, 4:47am

Try reinstalling docker.

Flow · February 24, 2020, 5:02am

I am trying to update it it remains at 0% until I kill storagenode,containerd and then dockerd
But it fails because it needs containerd or dockerd to be running, at least when I kill everything then apt-get upgrade to update docker-ce, dockerd will show up in top along with containerd.

I can’t seem to update but maybe I could prevent it from launching? then upgrade?
Could I uninstall it without loosing data?

nerdatwork · February 24, 2020, 5:19am

Yes, node’s data is stored outside docker so removing or upgrade docker does not affect data. In your case it might affect data since you kept using -v flag. When docker isn’t able to locate mounted drive it starts downloading data in overlay inside docker. When you restart docker/pc this data is lost which results in failed audits.

peem · February 24, 2020, 6:55am

Do you have disk space? which shows for example:
df -h /dev/sda1

It seems to me that the docker does not start due to lack of disk space …

cdhowie · February 24, 2020, 5:55pm

Based on what he pasted earlier, there is ~20% free space on the disk that docker is using for container storage, but 0% free space in the shared memory volume (usually /dev/shm). This might be the problem – 64MB capacity for the shm volume seems very low. How much RAM does this system have?

Flow · February 26, 2020, 5:46am

Alright, so I uninstalled docker, rm -rf the /var/lib/docker folder
Rebooted reinstalled, adjust a few double quotes that were not right and my dashboard shows online!
Uptime and audits shows **98.8%**99.1% hopefully I didnt loose my node!

It now has 16h uptime.

@cdhowie Yes the 20% is used space and the /dev/shm has 0% used space.
System memory is 16gb as shown in the top command screenshot above.

I was able to use the new run command with --mount instead of -v, now does this in itself contains the problem in that it shouldn’t happen again?

I can’t even post before 24h has passed, went over my daily limit for a new old user… I wish admins to see this and maybe help to remediate this as in an emergency situation where I needed support I could not have a discussion with anyone after 22 thread replies, 2 thread created and 2 messages sent to someone, those are the limits for new users. I get it, we have to protect the forum from spamming but when this hinders the support one can get, it may have to be reviewed, just my opinion!