Alternative Docker Logging Export Script

Node Operators FAQ
, ,
1

This is my custom brew docker log script, this method keeps the ability to use the

docker log --tail 20 storagenode --follow

command which i don’t want to loose, and which would be lost when redirecting the docker logs to a file in the more commonly used methods.

i have been performance testing and refining this over a long period of almost a year and it now works like a charm for me and i’m seeing zero performance issues.
even with extensive logging going on.

any suggestions on how to make it look prettier or simpler is always welcome.
figure i would share it so it might help someone else get their logging solution to work as they want.

#filename append-log.sh
#
#!/bin/bash
#Cronjob external script for appending to storagenode log
#
#Cron commandline
#*/2 * * * * /sn3/logs/append_log.sh >> /storjlogs/$(date +\%Y-\%m-\%d)-sn3.log 2>&1
#
#Note best used with small docker logs, because the full stored log is read upon docker log commands
#
#the --log-opt max-size=1m sets the docker logs to max 1 MB for performance reasons.
#my default docker run command is below
#
#
#docker run -d -e RUN_PARAMS="--filestore.write-buffer-size 4096kiB --pieces.write-prealloc-size 4096kiB" -d --restart unless-stopped --stop-timeout 300 -p 1$
#--log-opt max-size=1m \
#-p 192.168.1.100:14002:14002 -e WALLET="0x111111111111111111111" \
#-e EMAIL="your@email.com" -e ADDRESS="global.ip.inet:28967" \
#-e STORAGE="4TB" --mount type=bind,source="/sn3/id-sn3",destination=/app/identity \
#--mount type=bind,source="/sn3/storj",destination=/app/config --name sn3 storjlabs/storagenode:latest

docker logs --since "$(date -d "$date -3 minutes" +"%Y-%m-%dT%H:%M")" --until "$(date -d "$date -1 minutes" +"%Y-%m-%dT%H:%M")" sn0001
4 Likes
2

Try

sudo docker logs -f storagenode >> /volume2/xxxx/xxxx/node.log 2>&1 &

1 Like
3

Sorry, but why do not store the log directly to the file?

docker run ... storjlabs/storagenode --log.output /app/config/storagenode.log

The log would be in the data location. Or you can mount another location for logs in your docker run command.

2 Likes
4

I use a similar script, because the way I understand it, if I redirect logs like you suggest, I would lose the ability to run docker logs storagenode --since .... and would have figure out how to get the correct portion of the log from the file.

My script runs docker logs storagenode -f and redirects it to a file. The files are numbered and the number is incremented every time I (re)start the node.

5

Yes, if you setup log to the file, you will not be able to use docker logs command (it will return empty).
To search portion of the file it is easy to do so, but you should use different tools.
For example, to get logs for 2021-11-01, you need to filter them like this:

grep "2021-11-01" /mnt/storj/storagenode/storagenode.log

For entire month:

grep "2021-10" /mnt/storj/storagenode/storagenode.log

and so on.

To rotate your logs you can setup a logrotate:

6

I run my monitoring script every minute or so and only need the portion of the log since the last time the script ran. This is very easy to do with docker logs --since .., I just save a timestamp of when the script ran and use it next time. Another script takes “the last two minutes” (--since 2m) of the log. I’m sure I could figure out how to do both with just a file (and would have to if the fie was the only option), but docker logs (and running screen to save the log to a file as a backup) is just easier.

1 Like
7

With a file you can just use

tail -f /path/to/node.log

You can always also use something like this.

awk -v sdat=`date -u -d 'now-1 hours' +%Y-%m-%dT%H:%M:%S` \
    -v edat=`date -u -d 'now' +%Y-%m-%dT%H:%M:%S` \
    '{ if ($1 >= sdat && $1 <= edat) print $0}' /path/to/node.log

That date command is very flexible, you can use relative dates (“now-1 hours” or “1 hour ago”) or just absolute dates(“2021-11-02 08:00”). I’m guessing similar to the --since option with docker. If you want you can stick that in a script and use parameters for sdat, edat and filename to simplify.

All of this would save you from having to go through the entire docker log every few minutes to add some new lines.

1 Like
8

well it works fine and i don’t see any advantages in changing it… also been working on a failsafe shutdown part of it, but haven’t gotten that working… but maybe if i actually spent some more time on it then that would help lol

@BrightSilence yeah i can see how that would do pretty much the same thing, it wouldn’t change the log name tho since docker will not allow the change of the name while the log is running.
sure then i could learn to use logrotate, but on top of that it would also take away my ability to use successrate.sh on the different logs across different nodes defined by wild cards.

i’m sure if i set it up differently from the beginning there would be other reasons i wouldn’t want to change to my solution either.

it works for me and i like how it works.
especially the whole successrate.sh with wildcards.
so i can successrate all nodes for the last day or 10 days or a single node for 10 days.
and when i start nodes i can just do a docker logs --tail 200 sn0001
then change the 1 to 2 and so on and so forth.
making it quick to see everything is running correctly, even if i don’t use it that much lol

9

No it wouldn’t, you could just point it to the files exported by the nodes. It can work with either containers or files.

I’m not saying you should change it, but I do literally all the things you say you can’t do, while also redirecting logs to a file. So I’m just providing alternative options for anyone else who comes across this thread. It’s definitely all possible.

2 Likes