All my nodes keep restarting since last night: "/var/log/supervisor/supervisord.log does not exist"

Node Operators troubleshooting
1

Hi!

I’ve been running 5 nodes and everything goes well so far.
Last night, without changing anything in the config, all my nodes went down.

I tried to restart them this morning but they keep restarting after 1 minute or so.
I see this error when launching them:
storagenode3 | Error: The directory named as part of the path /var/log/supervisor/supervisord.log does not exist
storagenode3 | For help, use /usr/bin/supervisord -h

What Can I do ? :frowning:
They’re down for about 9 hours now :frowning:

Thanks a lot

2

Ok, I think I got it.
It seems that my bind option in docker run is the root cause.

Here is part of my docker-compose file:

      - type: bind
        source: /home/pi/storj_logs
        target: /var/log

I think I shouldn’t map things like that. Done like this, it maps local log file to my host log file. So, it’s normal it couldn’t find other log files (such as supervisord.log).

Probably a recent update uncovers my faulty docker-compose configuration.

Can someone tell me how I should map log files to my host?

Thanks a lot!

3

Are you sure that’s the cause? As far as I know, the node doesn’t save the log file inside the container to /var/log, unless you specified that location in the config.yaml file. Nothing in the snippet you posted indicates that the local log file is mapped to the host.

I don’t use docker compose, but what I did was create a new mount in my docker run command and modified config.yaml to point at that location. In my case:

docker run...
...
--mount type=bind,source="/path/to/log/directory",destination=/app/logs \
...

Then in config.yaml

...
log.output: "/app/logs/storj1/storagenode1.log"
...

You are the second person I’ve seen with this issue:

4

Thanks for your answer :slight_smile:

Yes it does. This part is defined in the docker-compose file.
As far as I know, it does the same thing as specifying the following in “docker run” command:

--mount type=bind,source="/path/to/log/directory",destination="/path/in/the/container/"

It confirm it works since I was able to see logs on my host machine by doing this.

Anyway, I removed this bind mount for the moment in order to make my Storj containers work. Its ok for now.
I’ll fix this with the correct bind mount after workday.

[Edit] Sorry @baker , you are right. I made a mistake when copy-pasting the bind mount option in my first post. I fixed it.

1 Like
5

Oh and by the way, that’s what I did: I added a “-- log.output” option in the docker-compose to specify the log output file.

6

Then please do not use system location to bind your folders from host in this case.
The @baker 's suggestion is more correct.
However, if you want to use a system path, then you actually can bind only log file

      - type: bind
        source: /home/pi/storj_logs/storagenode.log
        target: /var/log/storagenode.log
...
      command:
        - "--log.output=/var/log/storagenode.log"

Or use a subfolder:

      - type: bind
        source: /home/pi/storj_logs/
        target: /var/log/storagenode/
...
      command:
        - "--log.output=/var/log/storagenode/storagenode.log"

But the simplest way is to store logs on the disk with data, then you do not need to bind something additional,

      command:
        - "--log.output=/app/config/storagenode.log"

In this case your logs will be in data location.

7

Thanks.

Thats what I figured out.
But I dont understand why it doesnt work. Since the bind option mounts the host folder in the container, why is the container not able to store the files directly on the host machine (including system files stored on /var/log)?

Thats what I thought then indeed.
But is there a simple way to rotate logs or avoid them to grow indefinitely? I thought of a cron but don’t know what would happen if the log file is deleted (or flushed?) When the container is running.

8

Yes, the simplest way is to configure logrotate to manage your storagenode logs. It should already be running on your system. I recommend placing a logrotate config specifically for your node(s) in the /etc/logrotate.d folder. For example, I have a file /etc/logrotate.d/storagenode (it can have any name). The file contents are:

/path/to/logs/storj*/storagenode*.log {
        su username username
        weekly
        maxsize 200M
        rotate 52
        dateext
        dateformat -%Y%m%d
        compress
        delaycompress
        missingok
        notifempty
        copytruncate
}

There are many options, so pick what is right for you. In my case this rotates two sets of logs (hence the wildcards), the logs are owned by my non-root user/group so I had to specify, and then I specify the rotate options. The config file has to be owned by root:root, so you’ll have to chown the config file. You can do dry-runs by running sudo logrotate -d /etc/logrotate.d/storagenode

More information on logrotate here:

2 Likes
9

Oh nice!
Thank you so much.
I was about to reinvent the wheel :person_facepalming:

1 Like
10

Because this is how the binding is working in the docker - it replaces content in the container with content from the host.