Best method to redirect logs to independant storage in Linux

I am looking for advice on the best method for redirecting the docker logs to a USB storage device. I would like to redirect my logs to a file, however I do not want to redirect them to the HDD that is being used for storage. I have thought of a couple of ways to accomplish this. My primary goal is to have a setup that does not write the logs to the same drive as the storage drive, and also will not cause the node to fail if the USB storage used for logs fails. These all assume I have mounted a USB storage device to /mnt/usbkey and created a folder called logs.

Option 1: Redirect directly to mount point (using --mount type=bind)
In this scenario, I would add a line to my run command:

--mount type=bind,source="/mnt/usbkey/logs",destination=/app/logs

and then add the following to the config.yaml file:

log.output: "/app/logs/storagenode.log"

The potential problem I see here is if the USB storage I am using for my logs fails, the storage node won’t start. I would prefer that the node continue working in the event the USB storage fails. If I am missing logs until I notice the error, I would prefer that to downtime.

Option 2: Redirect directly to mount point (using -v)
Same as Option 1 except mounting the USB storage location with the -v syntax:

-v /mnt/usbkey/logs/:/app/logs

I think this would mean if the USB storage is unavailable, docker will just start writing logs to it’s own internal volume. Those would of course be destroyed the next time the container is recreated. This is the option I am leaning towards, but perhaps there is some reason I do not know of that would make this a bad idea.

Option 3: Redirect to symlink on node’s USB HDD
In this case I would create a symlink on my storage drive at /mnt/storj1/v3alpha/logs that points to /mnt/usbkey/logs and add the line log.output: "/app/config/logs/storagenode.log" to my config.yaml file. This is the option I am least certain about, as I am not super familiar with how docker or Linux would deal with the USB storage being unavailable. Perhaps this is no better than either option above. Would this create extra I/O load on the USB HDD even though it is actually writing to a different device? This probably adds unneeded complication without much or any benefit, but figured I would add it to the list.

Some extra info: I want to have the USB storage formatted as FAT32 so it’s easier to access the files on any OS. I have already dived into the FAT32/linux mount permissions issue and have sorted that out (I think).

My apologies for the long read. I am hoping someone with more familiarity with docker/linux will have some insights as to the pros/cons related to these methods.

There’s a fourth option…

  1. Redirect the log as per Storj instructions and then use logrotate to copy the log file to your USB drive at some pre-determined time or when the log file grows to a particular size.

This option does not require reconfiguration of the node beyond normal settings. It also will not cause the node to not start if your USB drive is not connected. If the USB drive is missing, logrotate will fail and send you a system-wide unix mail about the failure.

Here’s an example of Option 4

2 Likes

Thank you for the suggestion. Although this would get the logs onto the external USB disk without much extra effort, it still means the logs are being written to and read from the HDD used for storage. The node is running on a rock64 with a USB3.0 drive, so I am trying to avoid any extra I/O on the storage drive. I am planning on processing the logs with logrotate, but preferably on the USB storage directly.

Hi @baker , I was looking into my logs recently and found out, that 100m x 3 is just a day, as traffic or let’s say logging has become very huge. Currently I am storing the logs on the SD card on my RPi4B, but I assume this will not help in prolonging its lifetime… That’s why I thought about moving it to an extra HDD or so. I think first I’ll go with the option 1, because it’s easy to implement. Is there anything you’ve found out in the last 2 years and can strongly recommend? Thx in advance.

Hi Bivvo,

So far I have only had a few problems, all of my own making. I ended up using my option 2 above, with the -v option. In practice, the -v option hasn’t been helpful or problematic, and I would probably recommend --mount type=bind. The main issue with my setup is that with the USB drive mounted as FAT32, and if there is an unclean shutdown and there are then errors on the file system, the mount goes into read-only mode and won’t be re-mounted as R/W unless I fix the file system manually. This would not likely be an issue with a file system like ext4. On the rare occasion where this has happened it can sometimes cause the node to not start properly and the docker container gets stuck in a restart loop as it can’t write to the output file.

Other than self imposed file system issues, I have not had any problems with the method of redirecting logs and managing them with logrotate directly on the USB thumb drive. I run two nodes now and I simply added a sub-directory on the USB thumb drive for each node and pointed each node to it’s relevant directory:

--mount type=bind,source="/mnt/logs-usb",destination=/app/logs \

Then in config.yaml
log.output: "/app/logs/storj1/storagenode1.log" for node1, maps to /mnt/logs-usb/storj1

log.output: "/app/logs/storj2/storagenode2.log" for node2 maps to /mnt/logs-usb/storj2
…etc

2 Likes

Great to hear, so that should not be an issue for me (ext4).

Have chosen the same directory with different file names. Works so far.

Thank you @baker .

1 Like

hi @baker , I want to redirect my logs in an USB hdd on my server that run 3 nodes (UBUNTU 22.04 server)

i’ve read " Best method to redirect logs to independant storage in Linux"

i did understand what you said :
I run two nodes now and I simply added a sub-directory on the USB thumb drive for each node and pointed each node to it’s relevant directory:

--mount type=bind,source="/mnt/logs-usb",destination=/app/logs \

Then in config.yaml
log.output: "/app/logs/storj1/storagenode1.log" for node1, maps to /mnt/logs-usb/storj1

log.output: "/app/logs/storj2/storagenode2.log" for node2 maps to /mnt/logs-usb/storj2
…etc

but could you explain how to logrotate with this config ?

thank you

best regards

Philippe456 - France :grinning: )

May I ask why you want to do this?

I’m just placing this link here Configure logging drivers | Docker Documentation

Hi phillipe,

This is easy to do with logrotate since you can use wildcards to define your paths and files. I placed a logrotate config file in /etc/logrotate.d/ called storagenode . It is a plain text file. My logrotate config file is as follows. Don’t include comments in your config file (there is an uncommented version below).

/mnt/logs-usb/storj*/storagenode*.log {
#the wildcard (*) characters tell logrotate to look in any subfolder that
#begins with storj, and rotate any log that starts with storagenode
#and has a .log extension
        su user group #run as specific user and group, this may not be necessary for you
        weekly #rotates logs weekly
        maxsize 200M #rotate logs if log file is greater than 200 MB even if it has been less than a week
        rotate 52 #keep 52 previous log files
        dateext #append date to rotated log file extensions
        dateformat -%Y%m%d #use this date format for rotated logs
        compress #compress rotated logs
        delaycompress #leave the most recent rotate log uncompressed until the next cycle
        missingok #do not error out if log file is missing
        notifempty #do not rotate empty log files
        copytruncate #copies rotated log and truncates original
}
Uncommented version
/mnt/logs-usb/storj*/storagenode*.log {
        su user group
        weekly
        maxsize 200M
        rotate 52
        dateext
        dateformat -%Y%m%d
        compress
        delaycompress
        missingok
        notifempty
        copytruncate
}

You can get more information about different log rotate options here
https://www.man7.org/linux/man-pages/man8/logrotate.8.html

2 Likes

Hello @nyancodex ,
Yes after réflexion, maybe logrotate is not useful if i put the logs outside the nodes… in a USB HDD.
WhatsApp so you think about .
Thx

Thank you @baker
I Will have Ă  look at the config file and link.
Want to understand how it works.:grinning:
Best regards

1 Like

logrotate is working with text files (logs), if it has a read-write-delete access, it will work independently of logs location.