Best method to redirect logs to independant storage in Linux

I am looking for advice on the best method for redirecting the docker logs to a USB storage device. I would like to redirect my logs to a file, however I do not want to redirect them to the HDD that is being used for storage. I have thought of a couple of ways to accomplish this. My primary goal is to have a setup that does not write the logs to the same drive as the storage drive, and also will not cause the node to fail if the USB storage used for logs fails. These all assume I have mounted a USB storage device to /mnt/usbkey and created a folder called logs.

Option 1: Redirect directly to mount point (using --mount type=bind)
In this scenario, I would add a line to my run command:

--mount type=bind,source="/mnt/usbkey/logs",destination=/app/logs

and then add the following to the config.yaml file:

log.output: "/app/logs/storagenode.log"

The potential problem I see here is if the USB storage I am using for my logs fails, the storage node won’t start. I would prefer that the node continue working in the event the USB storage fails. If I am missing logs until I notice the error, I would prefer that to downtime.

Option 2: Redirect directly to mount point (using -v)
Same as Option 1 except mounting the USB storage location with the -v syntax:

-v /mnt/usbkey/logs/:/app/logs

I think this would mean if the USB storage is unavailable, docker will just start writing logs to it’s own internal volume. Those would of course be destroyed the next time the container is recreated. This is the option I am leaning towards, but perhaps there is some reason I do not know of that would make this a bad idea.

Option 3: Redirect to symlink on node’s USB HDD
In this case I would create a symlink on my storage drive at /mnt/storj1/v3alpha/logs that points to /mnt/usbkey/logs and add the line log.output: "/app/config/logs/storagenode.log" to my config.yaml file. This is the option I am least certain about, as I am not super familiar with how docker or Linux would deal with the USB storage being unavailable. Perhaps this is no better than either option above. Would this create extra I/O load on the USB HDD even though it is actually writing to a different device? This probably adds unneeded complication without much or any benefit, but figured I would add it to the list.

Some extra info: I want to have the USB storage formatted as FAT32 so it’s easier to access the files on any OS. I have already dived into the FAT32/linux mount permissions issue and have sorted that out (I think).

My apologies for the long read. I am hoping someone with more familiarity with docker/linux will have some insights as to the pros/cons related to these methods.

There’s a fourth option…

  1. Redirect the log as per Storj instructions and then use logrotate to copy the log file to your USB drive at some pre-determined time or when the log file grows to a particular size.

This option does not require reconfiguration of the node beyond normal settings. It also will not cause the node to not start if your USB drive is not connected. If the USB drive is missing, logrotate will fail and send you a system-wide unix mail about the failure.

Here’s an example of Option 4

2 Likes

Thank you for the suggestion. Although this would get the logs onto the external USB disk without much extra effort, it still means the logs are being written to and read from the HDD used for storage. The node is running on a rock64 with a USB3.0 drive, so I am trying to avoid any extra I/O on the storage drive. I am planning on processing the logs with logrotate, but preferably on the USB storage directly.