So I hope this doesn’t devolve into a finger pointing match between docker on mac and storj, but over the last couple weeks my computer has been locking up. I’ve been running a storagenode for over a year in this configuration with uptimes for a running container in the weeks range – now I can’t get thru a day.
In trying to troubleshoot this I saw that the memory used by the mac’s mount handler com.docker.osxfs grows quickly over time and once the computer runs out of swap space it kinda freezes up.
I tried downgrading docker from current (220.127.116.11) to the previous minor line (18.104.22.168), but didn’t help.
I don’t really want to revert backwards on the docker version of storj/storagenode as I’m sure that hasn’t been tested and would likely hose my configuration. Running
I was wondering if anybody else is seeing this issue running on a mac? Also, don’t tell me run docker in linux in a vm please. The current workaround is to restart docker twice a day to keep things running.
Here are some numbers to back things up with a timeline and some data points on memory use. I also notices that the overlay VM is growing slowly using
df in the container, but not sure that is related since a restart shouldn’t change the underlying filesystem contents of the same container instance.
7/22/20 4:22PM CT - restarted docker for mac - com.docker.osxfs: 4.2 MB 7/22/20 4:24PM CT - started storjnode docker image 1.6.4 - storj dashboard and docker logs -f also running - com.docker.osxfs: 51.8 MB 7/22/20 7:33pm CT (3h 10m) - com.docker.osxfs: 3.89 GB 7/22/20 8:40pm CT (4h 17m) - com.docker.osxfs: 6.10 GB 8:47pm CT, checking internal disk already 1.1G used $ docker exec -ti storagenode df / Filesystem 1K-blocks Used Available Use% Mounted on overlay 61255492 1174936 56939232 2% / 8:54pm CT - stopped dashboard and docker logs -f - com.docker.osxfs: 6.61 GB 8:56pm CT - com.docker.osxfs: 6.66 GB $ docker exec -ti storagenode df / Filesystem 1K-blocks Used Available Use% Mounted on overlay 61255492 1175020 56939148 2% / 8:57pm CT - com.docker.osxfs: 6.72 GB - restart docker 8:57pm CT - com.docker.osxfs: 6.39 MB - continues to grow - started storj-dashboard install docker 22.214.171.124 9:08pm CT - start storjnode, dashboard, and tail logs - com.docker.osxfs: 51.7 MB 9:24pm CT (16m) - com.docker.osxfs: 493.0 MB 9:42AM CT (12h 34m) - com.docker.osxfs: 10.03 GB (11.36GB swap on computer used) - reinstalling latest (126.96.36.199) 9:44am CT (45s) - com.docker.osxfs: 71.5 MB (1.94GB swap on computer used) 12:07pm (2h 23m) - com.docker.osxfs: 1.03GB (1.50GB swap on computer used)