# Alternative Docker Logging Export Script

**URL:** <https://forum.storj.io/t/alternative-docker-logging-export-script/15911>\
**Category:** FAQ\
**Tags:** docker, storagenode, log-rotate\
**Created:** [October 31, 2021, 1:03pm UTC](https://forum.storj.io/t/alternative-docker-logging-export-script/15911 "2021-10-31T13:03:12Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![SGC](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/sgc/32/1132_2.png) [@SGC](https://forum.storj.io/u/SGC)\
**Post date:** [October 31, 2021, 1:03pm UTC](https://forum.storj.io/t/alternative-docker-logging-export-script/15911/1 "2021-10-31T13:03:12Z")

</div>

This is my custom brew docker log script, this method keeps the ability to use the

docker log --tail 20 storagenode --follow

command which i don’t want to loose, and which would be lost when redirecting the docker logs to a file in the more commonly used methods.

i have been performance testing and refining this over a long period of almost a year and it now works like a charm for me and i’m seeing zero performance issues.  
even with extensive logging going on.

any suggestions on how to make it look prettier or simpler is always welcome.  
figure i would share it so it might help someone else get their logging solution to work as they want.

```auto
#filename append-log.sh
#
#!/bin/bash
#Cronjob external script for appending to storagenode log
#
#Cron commandline
#*/2 * * * * /sn3/logs/append_log.sh >> /storjlogs/$(date +\%Y-\%m-\%d)-sn3.log 2>&1
#
#Note best used with small docker logs, because the full stored log is read upon docker log commands
#
#the --log-opt max-size=1m sets the docker logs to max 1 MB for performance reasons.
#my default docker run command is below
#
#
#docker run -d -e RUN_PARAMS="--filestore.write-buffer-size 4096kiB --pieces.write-prealloc-size 4096kiB" -d --restart unless-stopped --stop-timeout 300 -p 1$
#--log-opt max-size=1m \
#-p 192.168.1.100:14002:14002 -e WALLET="0x111111111111111111111" \
#-e EMAIL="your@email.com" -e ADDRESS="global.ip.inet:28967" \
#-e STORAGE="4TB" --mount type=bind,source="/sn3/id-sn3",destination=/app/identity \
#--mount type=bind,source="/sn3/storj",destination=/app/config --name sn3 storjlabs/storagenode:latest

docker logs --since "$(date -d "$date -3 minutes" +"%Y-%m-%dT%H:%M")" --until "$(date -d "$date -1 minutes" +"%Y-%m-%dT%H:%M")" sn0001

```

---

<div class="post-metadata">

**Author:** ![Floux](https://storj.bcdn.literatehosting.com/letter_avatar/floux/32/5_5575768a8748004e209b776fc1b2916d.png) [@Floux](https://forum.storj.io/u/Floux)\
**Post date:** [November 1, 2021, 5:01pm UTC](https://forum.storj.io/t/alternative-docker-logging-export-script/15911/2 "2021-11-01T17:01:05Z")

</div>

Try

sudo docker logs -f storagenode \>\> /volume2/xxxx/xxxx/node.log 2\>&1 &

---

<div class="post-metadata">

**Author:** ![Alexey](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/alexey/32/41_2.png) [@Alexey](https://forum.storj.io/u/Alexey)\
**Post date:** [November 1, 2021, 10:29pm UTC](https://forum.storj.io/t/alternative-docker-logging-export-script/15911/3 "2021-11-01T22:29:45Z")

</div>

Sorry, but why do not store the log directly to the file?

```auto
docker run ... storjlabs/storagenode --log.output /app/config/storagenode.log

```

The log would be in the data location. Or you can mount another location for logs in your `docker run` command.

---

<div class="post-metadata">

**Author:** ![Pentium100](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/pentium100/32/6082_2.png) [@Pentium100](https://forum.storj.io/u/Pentium100)\
**Post date:** [November 2, 2021, 5:52am UTC](https://forum.storj.io/t/alternative-docker-logging-export-script/15911/4 "2021-11-02T05:52:10Z")

</div>

I use a similar script, because the way I understand it, if I redirect logs like you suggest, I would lose the ability to run `docker logs storagenode --since ....` and would have figure out how to get the correct portion of the log from the file.

My script runs `docker logs storagenode -f` and redirects it to a file. The files are numbered and the number is incremented every time I (re)start the node.

---

<div class="post-metadata">

**Author:** ![Alexey](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/alexey/32/41_2.png) [@Alexey](https://forum.storj.io/u/Alexey)\
**Post date:** [November 2, 2021, 5:57am UTC](https://forum.storj.io/t/alternative-docker-logging-export-script/15911/5 "2021-11-02T05:57:12Z")

</div>

Yes, if you setup log to the file, you will not be able to use `docker logs` command (it will return empty).  
To search portion of the file it is easy to do so, but you should use different tools.  
For example, to get logs for 2021-11-01, you need to filter them like this:

```auto
grep "2021-11-01" /mnt/storj/storagenode/storagenode.log

```

For entire month:

```auto
grep "2021-10" /mnt/storj/storagenode/storagenode.log

```

and so on.

To rotate your logs you can setup a logrotate:

> [@How to create daily storagenode logs](https://forum.storj.io/t/topic/6199/5):
>
> are you running in a NAS or Windows or Linux? In Ubuntu I use the “logrotate” command in a cron job to create/compress log files. First you have to make sure you have added in the “Log Location” and “Log level” fields in the “config.yaml” Then look in the “/etc/logrotate.d/” folder, you’ll see a list of files, those are the configs used to rotate logfiles ( usually found in “/var/log/”). I just put a “storj” config file in the logrotate.d/ and it’s done automatically To rotate my logs i use …

---

<div class="post-metadata">

**Author:** ![Pentium100](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/pentium100/32/6082_2.png) [@Pentium100](https://forum.storj.io/u/Pentium100)\
**Post date:** [November 2, 2021, 6:04am UTC](https://forum.storj.io/t/alternative-docker-logging-export-script/15911/6 "2021-11-02T06:04:18Z")

</div>

> [@Alexey](#):
>
> To search portion of the file it is easy to do so, but you should use different tools.  
> For example, to get logs for 2021-11-01, you need to filter them like this:

I run my monitoring script every minute or so and only need the portion of the log since the last time the script ran. This is very easy to do with `docker logs --since ..`, I just save a timestamp of when the script ran and use it next time. Another script takes “the last two minutes” (`--since 2m`) of the log. I’m sure I could figure out how to do both with just a file (and would have to if the fie was the only option), but docker logs (and running `screen` to save the log to a file as a backup) is just easier.

---

<div class="post-metadata">

**Author:** ![BrightSilence](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/brightsilence/32/6048_2.png) [@BrightSilence](https://forum.storj.io/u/BrightSilence)\
**Post date:** [November 2, 2021, 8:10am UTC](https://forum.storj.io/t/alternative-docker-logging-export-script/15911/7 "2021-11-02T08:10:29Z")

</div>

> [@SGC](#):
>
> this method keeps the ability to use the
> 
> docker log --tail 20 storagenode --follow
> 
> command which i don’t want to loose

With a file you can just use

```auto
tail -f /path/to/node.log

```

> [@Pentium100](#):
>
> I would lose the ability to run `docker logs storagenode --since ....` and would have figure out how to get the correct portion of the log from the file.

You can always also use something like this.

```auto
awk -v sdat=`date -u -d 'now-1 hours' +%Y-%m-%dT%H:%M:%S` \
    -v edat=`date -u -d 'now' +%Y-%m-%dT%H:%M:%S` \
    '{ if ($1 >= sdat && $1 <= edat) print $0}' /path/to/node.log

```

That date command is very flexible, you can use relative dates (“now-1 hours” or “1 hour ago”) or just absolute dates(“2021-11-02 08:00”). I’m guessing similar to the `--since` option with docker. If you want you can stick that in a script and use parameters for sdat, edat and filename to simplify.

All of this would save you from having to go through the entire docker log every few minutes to add some new lines.

---

<div class="post-metadata">

**Author:** ![SGC](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/sgc/32/1132_2.png) [@SGC](https://forum.storj.io/u/SGC)\
**Post date:** [November 2, 2021, 3:19pm UTC](https://forum.storj.io/t/alternative-docker-logging-export-script/15911/8 "2021-11-02T15:19:57Z")

</div>

well it works fine and i don’t see any advantages in changing it… also been working on a failsafe shutdown part of it, but haven’t gotten that working… but maybe if i actually spent some more time on it then that would help lol

@BrightSilence yeah i can see how that would do pretty much the same thing, it wouldn’t change the log name tho since docker will not allow the change of the name while the log is running.  
sure then i could learn to use logrotate, but on top of that it would also take away my ability to use successrate.sh on the different logs across different nodes defined by wild cards.

i’m sure if i set it up differently from the beginning there would be other reasons i wouldn’t want to change to my solution either.

it works for me and i like how it works.  
especially the whole successrate.sh with wildcards.  
so i can successrate all nodes for the last day or 10 days or a single node for 10 days.  
and when i start nodes i can just do a docker logs --tail 200 sn0001  
then change the 1 to 2 and so on and so forth.  
making it quick to see everything is running correctly, even if i don’t use it that much lol

---

<div class="post-metadata">

**Author:** ![BrightSilence](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/brightsilence/32/6048_2.png) [@BrightSilence](https://forum.storj.io/u/BrightSilence)\
**Post date:** [November 2, 2021, 3:26pm UTC](https://forum.storj.io/t/alternative-docker-logging-export-script/15911/9 "2021-11-02T15:26:31Z")

</div>

> [@SGC](#):
>
> it would also take away my ability to use successrate.sh on the different logs across different nodes defined by wild cards

No it wouldn’t, you could just point it to the files exported by the nodes. It can work with either containers or files.

I’m not saying you should change it, but I do literally all the things you say you can’t do, while also redirecting logs to a file. So I’m just providing alternative options for anyone else who comes across this thread. It’s definitely all possible.