Logrotate type of behavior

Node Operators FAQ
1

So I am very much used to configure logrotate for various logs in Linux.

What is the default behavior with storj - docker logs (as seen in “docker logs --follow storagenode”)? Will it ever be rotated or will it grow infinitely until restart of the node?

2

As far as I can tell, It will grow indefinitely.
I moved the log location to an external folder and then rotated them.

3

Does it not seem to be problem then, if one can not leave node running without having to periodically restart it?

4

if you have enough room on the host OS you can just leave it.

I tend to restart my OS and docker at least once a month (for updates and housekeeping) so my logs were never huge in the first place. But I moved the log location out of the docker so that they could be saved between restarts in case their was issues.

5

@binary I just left the log in docker and then made this for exporting it, it might as noted take a little bit of performance, but i haven’t seen more than i could barely measure it…
and then one can use docker logs commands like --follow or --tail

haven’t setup any logrotate yet, but you could if you wanted deletion features and such, i just made a compressed folder for them and left it to pile up for future me to worry about lol

it might be a bit rough around the edges pretty new in linux so…

6
  • It won’t fill indefinitely, but rather only until the next time a container upgrade is done by watchtower. That will deleted the old container and thus the old logging
  • You can always start your container with the docker run sub-flag --log-opt max-size=50m, which will result in compressed logging which then gets deleted with container deletes

I personally run /var/lib/docker off the same disk as the storagenode file storage location, therefor roll with --log-opt max-size=50m.

7

You can use logrotate for the storage node logs as well. Here is my config /etc/logrotate.d/storagenode

/home/storagenode/storagenode1.log
/home/storagenode/storagenode2.log
/home/storagenode/storagenode3.log
/home/storagenode/storagenode4.log
{
        weekly
        missingok
        rotate 4
        copytruncate
        nocompress
        notifempty
        create 640 storagenode storagenode
        errors <my email address>
}

This config will truncate the existing logfile. The storage node will keep writing to the logfile. I am not compressing my log files at the moment. If you want to safe some space you should do that.

3 Likes
8

Don’t get me wrong :slight_smile: I do logrotate on my storj log file, I don’t leave it in docker but use log.output to write to file instead, and then rotate that. I was just wondering what happens for those who use default scenario

9

all inside the docker image, and then when the version is updated it’s gone.

10

I work with a default scenario and logs still there after version update. I have 6 months logs old

11

At this point I would run the storagenode with your system’s service manager, without docker.

There is no benefit in using docker for storagenode: all utilities involved are self-contained monolithic executables. There is no need for dependency isolation.

And then you can configure log rotate (btw storagenode still does not honor HUP signal, so you might need to run it under a daemon and log to stdout)

12

docker just a convinient way to deliver and run the software without too much technical experience, in Linux services in particular (and podman in your case).

13

Oh, definitely, not denying this; it’s a useful OS-agnostic software distribution system. But once the user wants to make some customizations (such as implement log rotation), at some point it becomes easier to just run storagenode natively, than to keep fighting choices made in the container.

For example, they could implement log rotate right in the local container instance, but that would have been wiped out on the next update, and this is against the whole idea of how containers are supposed to work. They could create their own container (from scratch, or derived from storj-es), but this is just “installing natively” with extra steps, unnecessary dependencies, and maintenance requirements.

1 Like
14

not needed to reinvent a wheel:

The excerpt example:

--log-opt max-size=50m \
--log-opt max-file=10 \

If you configured redirect to the file, then you can use a regular logrotate with options like

if you would like to use logrotate, but do not want to run it on the host, you may use containers only.

docker container is a easiest way to start quick after re-flash the SD card in case of raspberry Pi

15

I’ve tried to use logrotate on the host, but storagenode does not handle HUP signal — logrotate/newsyslog send it to signal the app to release log handles and start a new log. Instead, storagenode dies in response to it.

So, I’ve workaround it by sending storagenode logs to stdout, and running it under daemon utility, which does process HUP correctly.

Docker’s log rotate is reinvention of the wheel — each system already has its own log rotating facility, so why did docker feel the need to to reinvent the wheel implement yet another one internally is beyond me. (But that does not affect me personally — I don’t use docker. I’m wondering though whether docker handles HUP correctly in this case… i would not hold my breath though)

16

I’m agree that docker uses an own variant of log rotation. However, it’s aligned with their containers and node did not die on request to rotate the log :slight_smile:
But my mention of the wheel was about your suggestion to implement an own (third!) way to rotate logs inside the container, when it’s not needed at all due to two other options - either use a docker’s one or use a logrotate in case of logs outside docker.

1 Like
17

Good point! Too many logrotates, I got confused :slight_smile:

2 Likes