All of a sudden the node was no longer reachable.
The log showed:
dockerd: runtime: out of memory: cannot allocate 1395810304-byte block (1248329728 in use) dockerd: fatal error: out of memory dockerd: runtime stack: dockerd: runtime.throw(0x22e619c, 0xd) etc.
It seems that this triggered Docker to restart itself:
systemd: docker.service: Failed with result 'exit-code'. systemd: docker.service: Service RestartSec=2s expired, scheduling restart. systemd: docker.service: Scheduled restart job, restart counter is at 1. systemd: Stopped Docker Application Container Engine. systemd: Starting Docker Application Container Engine...
So the system almost recovered itself. But it did not work. The node still was not reachable and did not work. I manually stopped and restarted, the container, no avail. I had to remove the container and start it then.
Any ideas how to catch and resolve such a failure and restart the container automatically?