SNO's ColoredLog for Windows/Linux/Mac using GUI/docker

[nerdatwork]

It’s a powershell script to show storagenode’s log in colors.

It works for docker and GUI

---

Link to repository

nerdatwork/Nerdatwork-s-ColoredLog

---

Sample output:

---

How to install powershell on Linux?

# Download the Microsoft repository GPG keys
wget -q https://packages.microsoft.com/config/ubuntu/16.04/packages-microsoft-prod.deb

# Register the Microsoft repository GPG keys
sudo dpkg -i packages-microsoft-prod.deb

# Update the list of products
sudo apt-get update

# Install PowerShell
sudo apt-get install -y powershell

# Start PowerShell
pwsh

Reference: Installing PowerShell on Linux - PowerShell | Microsoft Docs

---

How to install powershell on MacOS?

# Download the powershell '.tar.gz' archive
curl -L -o /tmp/powershell.tar.gz https://github.com/PowerShell/PowerShell/releases/download/v7.0.3/powershell-7.0.3-osx-x64.tar.gz

# Create the target folder where powershell will be placed
sudo mkdir -p /usr/local/microsoft/powershell/7.0.3

# Expand powershell to the target folder
sudo tar zxf /tmp/powershell.tar.gz -C /usr/local/microsoft/powershell/7.0.3

# Set execute permissions
sudo chmod +x /usr/local/microsoft/powershell/7.0.3/pwsh

# Create the symbolic link that points to pwsh
sudo ln -s /usr/local/microsoft/powershell/7.0.3/pwsh /usr/local/bin/pwsh

Reference: Installing PowerShell on macOS - PowerShell | Microsoft Docs

[SGC]

i don’t suppose there is some simple way of doing something like this on the logs that comes out of the docker logs --follow?

[nerdatwork]

@SGC

some simple way

Define simple

[BrightSilence]

Pipe either the docker logs tail follow command or a tail -f command into something like this.

awk '
  /INFO/ {print "\033[32m" $0 "\033[39m"}
  /ERROR/ {print "\033[31m" $0 "\033[39m"}
'

[SGC]

shortly after i got V3 running i wanted to make my storj logs into a green matrix like screen saver…
ofc it would be pretty cool if the color was the general health or current state of the node… so it will start to have a few reds here and there and then if it’s really bad it will turn it all read… to sort of clearly indicate something is wrong…

seemed like a fun little project… but then after i started using putty and docker logs --tail 20 storagenode --follow

it sort of started to seem like a pointless thing… and to be fair i don’t really use it much these days…
but still i really like the idea of it’s as this passive indicator in my workspace, so that i can loosely during the day keep an eye on it…

if nothing else scrolling text is kinda mesmerizing

[SGC]

@nerdatwork
do you have a trick for getting them looking so orderly… ?

might just be my old monitor that has to low resolution tho…
but your log output does seem a lot better structured than mine…
i even tried going to a font size 6 in full screen… and still…

[nerdatwork]

Can you show how your log looks?

[SGC]

something like this, isn’t cut particularly well…
but u get the idea, it overlaps in a annoying way…
wouldn’t mind if the text would just continue and then i could scroll to it if i needed it…

as you can see i got the awk thing working… using tail
only issue now is my entire live log export setup, which cron runs ever minute, so the tail awk thing will only update once a minute… but i’m pretty sure i can fix that with a redirection of the log in docker, that would make it live updated and then tail would update every new log entry and thus it should be live… i think

also might use much less processing…
apparently grabbing the bottom of a long log takes work in docker…
not a ton but enough…

if one could make them stop breaking and just continue into the infinite… so one can scroll to them if needed, but else they don’t disrupt the aesthetics of the log

but then again it might take the life out of it… going to test over a while and see what i like the most…
when i find ways of dealing with it…

[nerdatwork]

I am using font consolas and size 14 on a 21" FHD screen. In your screenshot I see everything green which is same as having everything in white but more “matrix” friendly.

@SGC

processing

After prolonged use I have observed powershell uses at max 0.2% CPU and 45MB RAM.

[SGC]

yeah, well error log lines will be in red…

the reason for the processing with my docker live export is related to the docker log size, the docker logs --since 2 minutes … command seems to read the entire log to grab 1minute and append it to the external kept storagenode daily log…

so at first it’s minimal, but as the log grows it will slowly start to show up in netdata… and after being like a month or so of logs and exporting 1hr in each grab it takes a minute or two… using 5-7% cpu time for that minute… so it’s not horrible… i now set a max log size in my storagenode docker run command.

the issue i got now is that the log printed on screen is in 1 minute chunks… so doesn’t have that scrolling matrix look
no clue how much memory it uses, but nothing significant on my setup i’m sure…

also your solution is far more practical for actually decoding the logs, i’m just trying to make a matrix lookalike screensaver type deal…

i seem to be able to replicate the line up if i push my resolution up beyond what my screen native res are…
also working on a 21" but beyond antique…

when i look at the output the black lines mixed in did sort of break up the whole column deal…
maybe ill end up using your solution… would be practical to be able to turn on the practical coloring layout for better monitoring…

lol terminal font size 5 xD

[BrightSilence]

I liked this idea a lot, so I decided to combine it into the script I already used to combine logs from 3 nodes in one tail.

normal traffic in blue (up dark/down light)
audits in cyan
repair in green (up dark/down light)
deletes in magenta
failed/error/fatal etc in red
canceled in yellow
The rest in white

Outdated script removed, please look at SNO's ColoredLog for Windows/Linux/Mac using GUI/docker - #24 by BrightSilence for the latest version

[SGC]

very nice…
i think ill move / realign my available space…

i kinda like how the lines break up into blocks rather than long columns rolling over the screen…

so to get the black lines, ill put the available space, by itself with just that on that line and the rest black… like it almost is in this screenshot of the log output.

i really like the color scheme you got going tho…
may have to steal your color scheme lol… now mine seems so plain

[BrightSilence]

You can do a lot with awk. You could actually remove parts or reorganize the lines as well if you want. For just monitoring the piece id isn’t always that interesting, so if you have limited screen space you could adjust things to remove that part too.

I might also make some adjustments to download started etc terminology so the next tab always aligns.

@SGC

i really like the color scheme you got going tho…
may have to steal your color scheme lol…

That’s why I share it . Took a while to fine tune, most combinations quickly look messy. I think the way I have it now works pretty well though.

[SGC]

well initially when i looked at the started, upload, uploaded, downloaded, download started thing in the logs i thought it was by mistake that it was like that… because why would one make it look like such a mess…

well i think it’s to break the columns and to enable one to at a glance sort the blocks.
it looks better than having those massive uniform columns scrolling across the screen…

not saying it cannot be improved… but i don’t think aligning it in columns is the ascetically right answer

and yeah AWK is amazing… i will need to really dig into that some more… didn’t have much luck with it the first time around… this time tho i managed to make it work… also didn’t really change much with it… the syntax of AWK isn’t exactly very intuitive xD
and didn’t really realize just how much could be done with AWK

[BrightSilence]

@BrightSilence

I might also make some adjustments to download started etc terminology so the next tab always aligns.

Much better!

Outdated script removed, please look at SNO's ColoredLog for Windows/Linux/Mac using GUI/docker - #24 by BrightSilence for the latest version

@SGC

well i think it’s to break the columns and to enable one to at a glance sort the blocks.
it looks better than having those massive uniform columns scrolling across the screen…

I don’t think so, more likely it’s just the terminology they wanted to use. Logs aren’t usually meant to constantly look at, but to search through for problems. Actually me adding a tab to the deleted line is something you wouldn’t want to do in the original log as it would shift the tab delimited “columns”. But it’s fine just for visualization.
Either way, in my setup the colors now break those lines up so it isn’t even useful for that anymore either.

[SGC]

i would still take the available space and move it off the end of log lines where it exists at and to a line below…

that way one can easily read it while the log is running, and it breaks up the long columns on screen into a more randomized block pattern… which i think gives a better look of the whole thing when running.

whats a good source for AWK syntax and command lookup?

[deathlessdd]

[SGC]

@BrightSilence

changed it a bit, still having trouble with the syntax… but slowly getting there… AWK is AWKSOME xD
so have to deep dive on that… great tool

Adjusted the colors, so that both downloads, and repair downloads are green, because they make money… so kinda made sense to group those, went with bright green for regular downloads and the slightly darker green for repair downloads because they are less profitable… might switch then around… not sure…

uploads i left blues, maked good sense that storjing is blue…

deletes changed into gray instead of the pink, made most sense and didn’t draw attention to a regular function, data fading into the aether…

and then i set a bright red for the errors…

still want to add an empty line after the available space, which i moved onto the next line and moved in a bit… not sure if thats going to work like i wanted it to… but when the extra empty line is added below it i’m sure it makes it much more readable while the log is running.

additionally i’m going to add a changing color scheme for the available space, so that it will go from some color… maybe white for easy readability when log is live, and warning maybe red orange or such…

oh yeah also left the yellow cancelled color … made good sense…

tail -F /tank/logs/storagenode-$(date +\%Y-\%m-\%d).log | awk '/^==> / 
	{
      c="";
      if ($0 ~ /download/                ) c="\033[92m";
      if ($0 ~ /upload/                  ) c="\033[94m";
      if ($0 ~ /GET_REPAIR/              ) c="\033[32m";
      if ($0 ~ /PUT_REPAIR/              ) c="\033[36m";
      if ($0 ~ /GET_AUDIT/               ) c="\033[96m";
      if ($0 ~ /delete/                  ) c="\033[30m";
      if ($0 ~ /cancel/                  ) c="\033[33m";
      if ($0 ~ /failed|error|ERROR|FATAL/) c="\033[31m";
      line=$0;
      gsub("upload started",   "upload start",   line);
      gsub("download started", "download start", line);
      gsub("upload canceled",  "upload cancel",  line);
      gsub("download canceled","download cancel",line);
      gsub("deleted\t",        "deleted\t\t",    line);
      gsub("Available", "\n\t\t Available", line);
    }
    {if (line != "") print c line "\033[0m"}'

[BrightSilence]

Interestingly enough I made nearly the same color changes you did since my last post. Using bright green for downloads, green for repair downloads. Same with blue for uploads, light for normal, darker for repair. I changed deletes to light yellow and canceled to magenta. I also added yellow for garbage collection and shortened the “About to delete piece id” to “move to trash”.

I’m not at my PC right now, but I can post the new script tomorrow.

[BrightSilence]

@SGC

/^==> /

You don’t need this part btw. That was part of the code I used to prefix the node name to the log line by using title lines of multi file tail output.

@SGC

if (line != "")

You can remove that condition too, the empty lines were also a result of multi file tail.