Set log redirection from docker run command

[gael]

In order to get logs redirected to a file, I follow the faq entry which tell to change the config.yaml file.

But I would know if it is possible to set the log file directly from the docker command.

[nerdatwork]

@gael

directly from the docker command

May I ask what’s the need for setting it through docker when it achieves the same goal?

[SGC]

i use this to preserve the docker logs functionality, it’s based on a docker command that exports to a file, and is basically that and a cron job.

one thing to keep in mind is that if the log file in docker gets large… like hundreds of mb, then it will start taking up a bit of resources… not bad tho… had the node running for a month and by the end it took about 2% avg of cpu resources.

@SGC

Cron command for live appending logs, with date stamp and without redirecting logs from the storagenode container.

Wanted to keep this simple, which i didn’t, but now it works kinda great…
This will keep a 1min delayed live log saved, without having to redirect your docker log files and thus you can run all scripts by default or run stuff like docker logs storagenode --follow

how to make it work is explained in detail, below and everything is basically copy paste ready.
Enjoy… and i hope some else finds this useful.

[gael]

May I ask what’s the need for setting it through docker when it achieves the same goal?

• For simplicity and coherency.
  I have asked this question because I am creating a script to set up a node.

• I also think that it is a source of error to have two ways to set parameters. The config.yaml should at least be filled with the parameters set from the docker command.

• I would prefer have the log file outside the container because I have experienced some db issues while trying to read them while the node was running. I know it is due to sqlite, but I would be more tranquil if there is outside the container.

• My goal is to put put data from log to a db, may be it would be simpler to get that directly from sqlite dbs ?

@SGC in response oto your question about execute command directly in cron: try to explicitly execute bash linux - Running bash inside cronjob - Unix & Linux Stack Exchange

[nerdatwork]

@gael

The config.yaml should at least be filled with the parameters set from the docker command

Whatever you don’t specify in docker run is picked up from config.yaml. Try this link if it helps.

Configure logging drivers

When you are successful do update this thread.

[SGC]

thanks ill give that a look and test if i can get that to work… been wanting to make it into essentially a one command, even if a rather extensive one… it does make it a lot more easy for others to implement.

i forget the exact reasons i couldn’t get it to work, i think it was something like some stuff would work in docker command line other stuff would not work in cron, so a bash script became the patch.

and in regard to you wanting access to the storj db then
the databases are located with the storagenode data files… storj/storage i believe is the default
same place as the blobs folder… which is in /storj/storage/blobs … its the big one
in case you can’t find it… and ofc anything inside the /storj/storage folder is strictly off limits for you to change something to keep in mind… tho i’m sure you are aware…

sorry if the gross oversimplification is an offence… not always easy to know peoples skill level or which considerations they take.

[gael]

sorry if the gross oversimplification is an offence… not always easy to know peoples skill level or which considerations they take.

I appreciate your intention !
Whatever my skill level, I prefer read detailled explanations.

I think it would be better to read directly from the dbs, but for the moment I will stay reading the logs.

[snorkel]

To redirect logs without config.yaml edit, for Docker nodes, you can set these in docker run command:

--mount type=bind,source="/home/user/Storj/storjlog1/",destination=/app/logs \
*--log-driver json-file \*
*--log-opt max-size=10m \*
*--log-opt max-file=5 \*
--name storagenode storjlabs/storagenode:latest \
--log.output="/app/logs/storagenode.log" \
--log.level=info \

To check logs:

cat /home/user/Storj/storjlog1/storagenode.log

You can adjust the log location path /home/user/Storj/storjlog1/ and log file name to your needs.
Change the name of the container storagenode to yours.
Those 3 log options are optional, but on some systems (Synology NAS) you need them for basic log rotation, because the default log driver is not jason-file and skips the other 2 options. It saves you the last 5 log files of 10MiB each.
log.level cat be set to: fatal, error, warn, info, debug.
Also, don’t forget to change permisions and owner for the log path, to match the user.

Edit: those 3 log options can’t be used if you set a custom path for logs. See explanations below.

[nerdatwork]

@snorkel

log location path /home/user/Storj/storjlog1/

Shouldn’t there be a folder called “logs” as per your above docker command ? Effectively making the path
/home/user/Storj/storjlog1/logs which will contain the log file.

[snorkel]

I don’t understand what you mean…
I use this exact command and I get this log file in this location:

/home/user/Storj/storjlog1/storagenode.log

[snorkel]

The directories are organised, don’t worry .
I use:
~/Storj/storjdbs1/
~/Storj/storjdbs2/
~/Storj/storjlog1/
~/Storj/storjlog2/
These are on boot NVMe.
The storagenode data is on the big drives.
For logs, I could’ve used a sigle dir, and different filenames, like ~/Storj/logs/storagenode1.log, storagenode2.log, but… I guess I like each node with it’s own folders.

[BrightSilence]

@snorkel

--log-driver json-file \
--log-opt max-size=10m \
--log-opt max-file=5 \

These options won’t do much if you let the storagenode redirect the logs itself. Basically only includes logging for downloading and starting the binaries. Which is still in the docker logs.

@nerdatwork

Shouldn’t there be a folder called “logs” as per your above docker command ? Effectively making the path
/home/user/Storj/storjlog1/logs which will contain the log file.

No, the logs folder in the container is a mount of storjlog1. See the mount command.

[snorkel]

Well, I don’t think I get what you are saying, but, after using those exact parameters with redirect, I got the usual info log, with node starting up, pieces uploads, downloads, etc. So what I recommended, just works.
Machine tested: Ubuntu server + Docker.
Now I switched back to fatal, waiting for the minimum version to be the one with log level micro tuning, and then I will set a nice clean info logging, with just the useful stuff.

[Alexey]

All options like

@snorkel

--log-driver json-file \
--log-opt max-size=10m \
--log-opt max-file=5 \

only affects how your logs are handled in the DOCKER, unless you redirected logs to the file. In the later case you will be forced to manage logs differently:

Best method to redirect logs to independant storage in Linux

Hi phillipe, This is easy to do with logrotate since you can use wildcards to define your paths and files. I placed a logrotate config file in /etc/logrotate.d/ called storagenode . It is a plain text file. My logrotate config file is as follows. Don’t include comments in your config file (there is an uncommented version below). /mnt/logs-usb/storj*/storagenode*.log { #the wildcard (*) characters tell logrotate to look in any subfolder that #begins with storj, and rotate any log that starts wi…

[snorkel]

Ou, I didn’t know that. I’ll look into it. I assumed that the path dosen’t matter, if it’s default (managed by docker) or set by user.

[Alexey]

The difference here that you changed the log path for the application, not for docker (you provided option --log.output after the image name, so it will be passed to the application).
See also Configure logging drivers | Docker Docs.