So I hope this doesn’t devolve into a finger pointing match between docker on mac and storj, but over the last couple weeks my computer has been locking up. I’ve been running a storagenode for over a year in this configuration with uptimes for a running container in the weeks range – now I can’t get thru a day.
In trying to troubleshoot this I saw that the memory used by the mac’s mount handler com.docker.osxfs grows quickly over time and once the computer runs out of swap space it kinda freezes up.
I tried downgrading docker from current (2.3.0.3) to the previous minor line (2.2.0.5), but didn’t help.
I don’t really want to revert backwards on the docker version of storj/storagenode as I’m sure that hasn’t been tested and would likely hose my configuration. Running 1.6.4
I was wondering if anybody else is seeing this issue running on a mac? Also, don’t tell me run docker in linux in a vm please. The current workaround is to restart docker twice a day to keep things running.
Here are some numbers to back things up with a timeline and some data points on memory use. I also notices that the overlay VM is growing slowly using df
in the container, but not sure that is related since a restart shouldn’t change the underlying filesystem contents of the same container instance.
7/22/20 4:22PM CT
- restarted docker for mac
- com.docker.osxfs: 4.2 MB
7/22/20 4:24PM CT
- started storjnode docker image 1.6.4
- storj dashboard and docker logs -f also running
- com.docker.osxfs: 51.8 MB
7/22/20 7:33pm CT (3h 10m)
- com.docker.osxfs: 3.89 GB
7/22/20 8:40pm CT (4h 17m)
- com.docker.osxfs: 6.10 GB
8:47pm CT, checking internal disk already 1.1G used
$ docker exec -ti storagenode df /
Filesystem 1K-blocks Used Available Use% Mounted on
overlay 61255492 1174936 56939232 2% /
8:54pm CT
- stopped dashboard and docker logs -f
- com.docker.osxfs: 6.61 GB
8:56pm CT
- com.docker.osxfs: 6.66 GB
$ docker exec -ti storagenode df /
Filesystem 1K-blocks Used Available Use% Mounted on
overlay 61255492 1175020 56939148 2% /
8:57pm CT
- com.docker.osxfs: 6.72 GB
- restart docker
8:57pm CT
- com.docker.osxfs: 6.39 MB
- continues to grow
- started storj-dashboard
install docker 2.2.0.5
9:08pm CT
- start storjnode, dashboard, and tail logs
- com.docker.osxfs: 51.7 MB
9:24pm CT (16m)
- com.docker.osxfs: 493.0 MB
9:42AM CT (12h 34m)
- com.docker.osxfs: 10.03 GB (11.36GB swap on computer used)
- reinstalling latest (2.3.0.3)
9:44am CT (45s)
- com.docker.osxfs: 71.5 MB (1.94GB swap on computer used)
12:07pm (2h 23m)
- com.docker.osxfs: 1.03GB (1.50GB swap on computer used)