I’ve been running 5 nodes and everything goes well so far.
Last night, without changing anything in the config, all my nodes went down.
I tried to restart them this morning but they keep restarting after 1 minute or so.
I see this error when launching them: storagenode3 | Error: The directory named as part of the path /var/log/supervisor/supervisord.log does not exist storagenode3 | For help, use /usr/bin/supervisord -h
What Can I do ?
They’re down for about 9 hours now
Are you sure that’s the cause? As far as I know, the node doesn’t save the log file inside the container to /var/log, unless you specified that location in the config.yaml file. Nothing in the snippet you posted indicates that the local log file is mapped to the host.
I don’t use docker compose, but what I did was create a new mount in my docker run command and modified config.yaml to point at that location. In my case:
Then please do not use system location to bind your folders from host in this case.
The @baker 's suggestion is more correct.
However, if you want to use a system path, then you actually can bind only log file
Thats what I figured out.
But I dont understand why it doesnt work. Since the bind option mounts the host folder in the container, why is the container not able to store the files directly on the host machine (including system files stored on /var/log)?
Thats what I thought then indeed.
But is there a simple way to rotate logs or avoid them to grow indefinitely? I thought of a cron but don’t know what would happen if the log file is deleted (or flushed?) When the container is running.
Yes, the simplest way is to configure logrotate to manage your storagenode logs. It should already be running on your system. I recommend placing a logrotate config specifically for your node(s) in the /etc/logrotate.d folder. For example, I have a file /etc/logrotate.d/storagenode (it can have any name). The file contents are:
There are many options, so pick what is right for you. In my case this rotates two sets of logs (hence the wildcards), the logs are owned by my non-root user/group so I had to specify, and then I specify the rotate options. The config file has to be owned by root:root, so you’ll have to chown the config file. You can do dry-runs by running sudo logrotate -d /etc/logrotate.d/storagenode