Log rotating on FreeBSD

[Fogel]

Hi there,

Been running Storagenode on FreeBSD for quite a while, I’ve set --log.output to “/var/log/storagenode.log” and told newsyslog to rotate the logs daily.

/var/log/storagenode.log storj:storj 644 30 * @T00 JC /mnt/storj/storagenode_child.pid

This however restarts, or rather kills the node on every rotate since setting no signal-value in newsyslog sends a SIGHUP to the process found in the pid-file specified. I’m running daemon which automatically restarts storagenode as soon as it exits.

I would like to know if there is a way to send another signal to storagenode for it to re-open the log file, or if you could implement it if it’s not available today so that I don’t have to restart the node every night while rotating the logs.

For example, nginx supports this by sending the signal “SIGUSR1”.

[SGC]

i didn’t like the redirection of logs… so i came up with this
not really very familiar with BSD but the fundamental mechanics of it should work regardless, even if you might need to used of different commands.

the base concept is using date variables to select the log name and simply append them from docker, but in theory if you aren’t using docker then you should be able to just name the log using a variable name and it would rotate them like say daily, i like daily because it matches up with the dashboard graphs and such in case i wanted to use them for that…

here is a link to it on the forum.

@SGC

Cron command for live appending logs, with date stamp and without redirecting logs from the storagenode container.

Wanted to keep this simple, which i didn’t, but now it works kinda great…
This will keep a 1min delayed live log saved, without having to redirect your docker log files and thus you can run all scripts by default or run stuff like docker logs storagenode --follow

how to make it work is explained in detail, below and everything is basically copy paste ready.
Enjoy… and i hope some else finds this useful.

it has been suggested that this might require a bit of extra resources to run, but i’ve not seen much change in the tools i use for monitoring the system performance, tho YMMW

[Fogel]

Thanks for the reply, I’d like to keep it as standard as possible without extra scripts working around solutions that, in my opinion, should exist in the application (at least at some point)

I’m using Zabbix to follow the storagenode-log and draw up graphs so having log files with changing names might possibly create some more challenges on that end as well. It’s simpler now since the log-file is always the same name.

I’ll probably keep the current solution with restarting the node every day to re-open the logs, but would really appreciate if the devs could implement a “re-opening logs” signal

[SGC]

what can be more standard than using a date variable in the log output name?
it basically doing nothing aside from using a name that switches inside the system.
not like the system can guess what date it is… it has a variable that people use for these kinds of things…
like in your log every time there is a time code, its literally using date variables to print that into the log.

this is just using those same variables on the name and thus the logs will change names depending on days… my script might not do it exactly that way.

but thats because i have to deal with docker and then cron won’t run the command because its python codes i guess…so i have to move the command lines out of it, and thus i use some convoluted ways of essentially writing a one line log command to docker, and then i repeat it because i don’t redirect the because if i do that i loose some functionality with docker, so i export and append the 1 minute from the last one minute ago onto my log.

like you said you wanted to redirect the logs anyways, so you don’t have to deal with the scripting.
your solution should look something like this.

this is a location and at the end there are OS date and time variables which might even follow the unix standards, so you might be able to lift it directly, but not sure… this works for me in debian linux.

/storagenodes/storj/logs/storagenode_$(date +%Y-%m-%d).log

you should simply use something like this

$(date +%Y-%m-%d)

in whatever name you want to rotate

which is basically telling the system, i want a date stamp variable using %Y which is year
\ %m which is month in numbers then initials of the month would be something like %M
\ %d is ofc day and again if one makes it capital it becomes the short name for the day of the week … like say Sun for today. instead of a number.
and the $() is what contains it so stuff doesn’t become a loose mess or mistake commands for other stuff like say if out wanted to actually name your file this… or something close…

you can test this out by using something like echo i would assume.

like this in my case
echo $(date +%Y)

tell my terminal to print the year on screen in its full 4 digits

echo $(date +%Y%D)
gives me… oh right a full date in one stamp… you could essentially use that if you wanted

$(date +%D)

but the reason i wanted it to sort the Dates in the particular order is when the files are listen by alphabet then they are orderly sorted, by year,month and day, so i wrote the timestamp myself. (kinda forgot about that)

so yeah sorry if all the code confused you…

if you want to keep it as simple as possible you can ofc use something like

$(date +%D) in the name of the log file…

anyways hopes it helps…

not sure you can use / in filenames tho
the %D gives out 05/10/20
but then you just look up the $(date) pre made variables and find one that fits, or use the one i decided to make.

[littleskunk]

I am using logrotate with the copy truncate option.

[Fogel]

copytruncate is unfortunately not available in newsyslog on FreeBSD.

[Alexey]

Docker doesn’t work normally on freeBSD. The only way at the moment is using binary.

[SGC]

didn’t say he should use docker… just that he should use a date variable like i did in my docker related commands.

[Alexey]

I generally like the idea with a date-related logs. For example with --log.output option in storagenode run command. However, the date will be expanded once the service is started and will not be changed until restarted.
So, you can’t use it without restarts anyway.

[SGC]

maybe… worked in my case but that is ofc a heuristic errr recursive command, so it would then get expanded as you say each time the command runs… i’m not really much of a programmer… so

[Fogel]

Don’t get me wrong, I appreciate your solution even though it’s not for me, however someone else might have use for it!
I like to keep things as stock standard as possible, using tools built-in with the OS as much as possible. In this case either redirecting the logs from storagenode to a syslog handler (Does Storagenode support a syslog handler?) or using the --log.output and having storagenode being able to re-open the log-file on rotate.

In this sense the “standard” way to rotate logs is newsyslog rotates the log (renames /var/log/storagenode.log to /var/log/storagenode.1) and compresses it, and tells the application to re-open the log-file to continue writing to it. If the .1 file already exists that file is renamed to .2 and so on, this is probably default in all systems running initd, systemd OSes has other approaches.

[SGC]

from what i understood @Alexey said then the problem is that the storagenode logfile is an open stream of sorts and thus its difficult to move it when the storagenode is active, same issue that causes the variable $date name not working, because it’s only activated one at launch of the program… i suppose in theory if one moved the file and immediately created a new empty one, the storagenode might not have time to notice… that sort of depends upon how it see the file… it might not use the filename to identify it

only method i would figure to work would be copying the file, and then maybe clearing the storagenode active log file afterwards which essentially shouldn’t break the live connection that i understand that exist between the storagenode and the log file…

or you could do a super brief shutdown of the storagenode when you switch the log… it shuts down fast usually and boots almost just as fast

but i’m basically guessing… maybe Alexey has a better solution he is pretty well versed in all things storagenode.

on another note, i’ve been thinking of moving to FreeBSD or similar (kinda settled on Free) maybe with some GUI top layer that makes it a bit more easy to approach.

not sure it’s the right choice tho, because i kinda like the idea to hardware passthrough stuff which linux allows for these days… but linux userland is also driving me kinda crazy with the whole naming convention… and then there is the whole which platform sees most development “debian linux” ofc with FreeBSD i could develop my own gear that uses it in the future and market it with profit… so it would be useful to have it as my primary driver to make me familiar with it.

You got anything good or bad to note about FreeBSD and are you familiar with the other “options”
i get the sense that people don’t really come back from moving to FreeBSD lol

[Fogel]

I’ve worked around this “problem” by letting storagenode output everything to stdout, which is captured by ‘daemon’ which is basically just a service-wrapper in FreeBSD. Newsyslog can now rotate the logs freely and I don’t have to restart storagenode on each rotate.

Thanks for the given solutions in the thread, it made me think one step further and finally land on this solution for the moment.

For those wondering I use this rc-script (/usr/local/etc/rc.d/storagenode)

#!/bin/sh
#
# PROVIDE: storagenode
# REQUIRE: networking
# KEYWORD:

. /etc/rc.subr

name="storagenode"
rcvar="storagenode_enable"
storagenode_user="storj"
storagenode_command="/home/storj/go/bin/storagenode run --config-dir /mnt/storj/storage --identity-dir /mnt/storj/identity"
pidfile="/mnt/storj/${name}.pid"
childpidfile="/mnt/storj/${name}_child.pid"
command="/usr/sbin/daemon"
command_args="-p ${childpidfile} -P ${pidfile} -R 5 -S -s info -T storagenode ${storagenode_command}"

load_rc_config $name
: ${storagenode_enable:=no}

run_rc_command "$1"

And then I capture it in rsyslog with the following:

# /usr/local/etc/rsyslog.d/10_storagenode.conf
# This file is managed by Puppet, changes may be overwritten
if $programname == 'storagenode' then /var/log/storagenode.log
& stop

And rotate in newsyslog.conf with:

/var/log/storagenode.log storj:storj 644 30 * @T00 JC

To answer @SGC FreeBSD questions I run Arch Linux for desktops/laptops and mostly FreeBSD for servers. Also have a few Arch Linux servers and Debian servers still running because I haven’t gotten the chance to convert the apps they serve to FreeBSD yet.

I like FreeBSD because of the simplicity it provides, packages can easily be built with a build-server with your custom build-options. You can follow either the quarterly branch of packages to just get security updates and features every new quarter, or go bleeding edge on the head branch.
I like that all OS stuff resides in /etc and all packages has their stuff separate in /usr/local/etc.
It comes with some challenges that has made me lose a lot of hair with weird quirks, but at the end of the day I love the way FreeBSD just works and you get what you set up, no extra fluff that (in my opinion) you get when installing packages with Debian for example.