Follow multiple nodes logs on 1 terminal

Hi, is there a simple way to follow logs on multiple nodes (same PC) ?

I would like to see tailed and followed the docker logs for my 3 nodes on the same terminal if possible.

basically what this command does, but for 3 nodes :

docker logs -t storagenode --tail 100 -f

Thanks

terminal -> open new tab -> docker logs -t storagenode --tail 100 -f
-> open new tab -> docker logs -t storagenode1 --tail 100 -f
-> open new tab -> docker logs -t storagenode2 --tail 100 -f
etc

sorry, i mean mixed logs in one terminal

It’s possible to do something like that if you redirect the logs to a file. Unfortunately the tail output when following multiple files is very noisy. It prints an empty line and a line with the file name every time it gets lines from a different file and the log will be switching back and forth constantly. So I wrote a little script using awk to make it more readable. It also does color highlighting which may not be something you want. Anyway, here it is.

tail -F /volume1/storj/v3/data/node.log /volume1/storj/usb/db/node.log /volume1/storj/drobo/db/node.log | stdbuf -o0 awk '/^==> / {
    if (substr($0, 20, 2) == "v3"   ) a="Syno";
    if (substr($0, 20, 3) == "usb"  ) a="USB";
    if (substr($0, 20, 5) == "drobo") a="Drobo";next}
/pieces\tdeleted piece/ {next}
    {
      c="";
      if ($0 ~ /download/                ) c="\033[92m";
      if ($0 ~ /upload/                  ) c="\033[36m";
      if ($0 ~ /GET_REPAIR/              ) c="\033[32m";
      if ($0 ~ /PUT_REPAIR/              ) c="\033[94m";
      if ($0 ~ /GET_AUDIT/               ) c="\033[96m";
      if ($0 ~ /delete/                  ) c="\033[93m";
      if ($0 ~ /retain/                  ) c="\033[33m";
      if ($0 ~ /cancel/                  ) c="\033[95m";
      if ($0 ~ /failed|error|ERROR|FATAL/) c="\033[91m";
      line=$0;
      gsub("upload started",              "upload start",   line);
      gsub("download started",            "download start", line);
      gsub("upload canceled",             "upload cancel",  line);
      gsub("download canceled",           "download cancel",line);
      gsub("deleted\t",                   "deleted\t\t",    line);
      gsub("About to move piece to trash","\tmove to trash",line);
      gsub("Z\t",                         "Z  ",            line);
    }
    {if (line != "") print c a " >\t" line "  < " a "\033[0m"}'

thank you, my strange ask it’s not worth the headache :slight_smile:

Well that may have made it look more daunting than it actually is. I still have an old version without all the color stuff and replacements in it. Should look a lot more palattable.

tail  -F /volume1/storj/v3/data/node.log /volume1/storj/usb/db/node.log /volume1/storj/drobo/db/node.log | awk '/^==> / {
    if (substr($0, 20, 2) == "v3"   ) a="Syno";
    if (substr($0, 20, 3) == "usb"  ) a="USB";
    if (substr($0, 20, 5) == "drobo") a="Drobo";next}
    {if ($0 != "") print a"\t"$0}'
1 Like

You can take Bright’s code and put it in a bash script in your home directory so you only have to remember a quick command. For example, create a text file called storjlogs.sh in your home directory. Place #!/bin/bash on the first line of the file followed by the rest of the script (modifying your paths/names as needed):

#!/bin/bash

tail  -F /volume1/storj/v3/data/node.log /volume1/storj/usb/db/node.log /volume1/storj/drobo/db/node.log | awk '/^==> / {
    if (substr($0, 20, 2) == "v3"   ) a="Syno";
    if (substr($0, 20, 3) == "usb"  ) a="USB";
    if (substr($0, 20, 5) == "drobo") a="Drobo";next}
    {if ($0 != "") print a"\t"$0}'

Then make the file executable:

sudo chmod +x storjlogs.sh

From the command line you just have to type storjlogs.sh and the script will execute. ctrl+c as normal to stop. You should really consider redirecting your logs to files anyway since it can make troubleshooting a lot easier in the future.

1 Like

Sorry for stupid question - why not tmux or screen with tile mode?

1 Like

Or both :slight_smile:

This is my setup

2 Likes

I use graylog - i haven’t fully configured the interface to work great yet. But it stores all my nodes logs on one centralized logging server that is easy to search since day one. (Of course you can also output those huge log files for other tools as well).

Thanks for all your advises.

Graylogs looks very close to Splunk