How to deal with log file size under Windows?

[Balage76]

I run my nodes under Windows and have the default log level set.
Due to the recently started test, the number of generated log lines increase drastically and this way the file size as well. It is growing by about 1GB / day / node.
I understand that there are different log levels and I can set it to WARN, but this way I believe I would miss the usefull infos about the startup of the node and the filewalkers.
Is there a way to limit the log generation to WARN, but store the mentioned infos?

[Alexey]

You need to use a logrotate or custom scripts

How to create daily storagenode logs

are you running in a NAS or Windows or Linux? In Ubuntu I use the “logrotate” command in a cron job to create/compress log files. First you have to make sure you have added in the “Log Location” and “Log level” fields in the “config.yaml” Then look in the “/etc/logrotate.d/” folder, you’ll see a list of files, those are the configs used to rotate logfiles ( usually found in “/var/log/”). I just put a “storj” config file in the logrotate.d/ and it’s done automatically To rotate my logs i use …

Native logs rotation in Windows with a simple PowerShell script

Hello, Unfortunately, in the current windows version of the Storage Node there is no way to rotate or limit the size of the generated logs files. You cannot delete them from the GUI because they are locked by the services processes. Using Clear-Content in PowerShell to reset the files works but there is no easy solution if you want to keep a few days / weeks of logs. I have written a small PowerShell script that can rotate the Storjs Logs without the need of Stop>Start the services to release …

Best method to redirect logs to independant storage in Linux

Hi phillipe, This is easy to do with logrotate since you can use wildcards to define your paths and files. I placed a logrotate config file in /etc/logrotate.d/ called storagenode . It is a plain text file. My logrotate config file is as follows. Don’t include comments in your config file (there is an uncommented version below). /mnt/logs-usb/storj*/storagenode*.log { #the wildcard (*) characters tell logrotate to look in any subfolder that #begins with storj, and rotate any log that starts wi…

All my nodes keep restarting since last night: "/var/log/supervisor/supervisord.log does not exist"

Yes, the simplest way is to configure logrotate to manage your storagenode logs. It should already be running on your system. I recommend placing a logrotate config specifically for your node(s) in the /etc/logrotate.d folder. For example, I have a file /etc/logrotate.d/storagenode (it can have any name). The file contents are: /path/to/logs/storj*/storagenode*.log { su username username weekly maxsize 200M rotate 52 dateext dateformat -%Y%m%d …

How to create daily storagenode logs

are you running in a NAS or Windows or Linux? In Ubuntu I use the “logrotate” command in a cron job to create/compress log files. First you have to make sure you have added in the “Log Location” and “Log level” fields in the “config.yaml” Then look in the “/etc/logrotate.d/” folder, you’ll see a list of files, those are the configs used to rotate logfiles ( usually found in “/var/log/”). I just put a “storj” config file in the logrotate.d/ and it’s done automatically To rotate my logs i use …

[Balage76]

Thank you Alexey, but except your second quote, these are about Linux solutions.
Is it possible to modify/delete the log file in Windows without stopping the node?

[naxbc]

Yap, this is a .bat file I created and then created a Monthly Scheduled Task that runs it:

net stop storagenode
TIMEOUT /T 10 /NOBREAK
echo .> “C:\StorjLogs\storagenode.log”
TIMEOUT /T 10 /NOBREAK
net start storagenode

[Balage76]

I was thinking about something similar, but this does not solve the issue, I will have 30+GB log files.
Yes, I can set shorter time range, but then the filewalker will start/run all the time.

Maybe it would be usefull to add one more log level, which is WARN + the info level system messages (like startup + filewalker).

[naxbc]

Well, set log level to “info” and schedule the task to run monthly.
By my experience, nodes tend to be less saturated on Sundays.
My task runs on each last sunday of the month.

[Ruskiem]

@Balage76

Is it possible to modify/delete the log file in Windows without stopping the node?

I found Yes.

With this code:

#-----------------------------------------------
#-------------------- INFOS --------------------
#Script:   StorjLogsRotation
#Version:  1.0.1.2
#Date:     27/04/2020
#Author:   JDA
#------------------- Variables ------------------
$PathLogLive      = 'C:\Program Files\Storj\Storage Node\'
$PathLogArchive   = 'F:\logs_auto_clean\'
$DateTimeFormat   = "yyyy-MM-dd HH'h'mm"
$RententionInDays = 88888
#-----------------------------------------------
 
 
Function StorjLogsRotation{
 ForEach($L in (Get-ChildItem -LiteralPath $PathLogLive -Filter '*.log' -Recurse:$False)){
  RotateLog -Source $L.FullName -DestinationPath $PathLogArchive -RententionInDays $RententionInDays
 }
}
 
 
Function RotateLog([String]$Source, [String]$DestinationPath,[Int]$RententionInDays){
 $Format = '{0}_[{1}]{2}'
 
 If(Test-Path -LiteralPath $Source){
  $S = Get-Item -LiteralPath $Source
  $Destination = Join-Path -Path $DestinationPath -ChildPath ($Format -F $S.BaseName, ((Get-Date).ToString($DateTimeFormat)), $S.Extension)
 
  If(!(Test-Path -LiteralPath $DestinationPath)){$Null = New-Item -Path $DestinationPath -Type Directory -Force}
  Copy-Item -Path $Source -Destination $Destination -Force
  Clear-Content -LiteralPath $Source -Force
 
  Get-ChildItem -LiteralPath $DestinationPath -File -Filter ($Format -F $S.BaseName, '*',$S.Extension) | ? LastWriteTime -le ((Get-Date).AddDays(-$RententionInDays)) | Remove-Item -ErrorAction SilentlyContinue
 }
}
 
StorjLogsRotation

but i had to help my self with autoit .au3 file, and compile to .exe with this 2nd code:

#include <AutoItConstants.au3>
#include <MsgBoxConstants.au3>
#include <Array.au3> 
#RequireAdmin
 
;$CmdPid = Run("C:\Users\urmom\Desktop\logrotate.ps1
sleep(3000)
;Powershell.exe -File C:\Users\urmom\Desktop\logrotate.ps1
;powershell.exe -executionpolicy bypass -File C:\Users\urmom\Desktop\logrotate.ps1
;powershell.exe -noexit "& 'C:\Users\urmom\Desktop\logrotate.ps1'"
 
Local $iPid = Run("powershell.exe -executionpolicy bypass -windowstyle hidden -noninteractive -nologo -file C:\Users\" & @UserName & "\Desktop\logrotate.ps1", "c:\", @SW_HIDE, $STDOUT_CHILD)
processWaitClose($iPid)
Local $sOutput = StdoutRead($iPid)

because somehow it was the fastest way to solve it for me.
Probably can be made simpler, but it was working so yeah.
Obviously You have to update that with Your OWN user name and C:\ file paths!

So basically You Task Schedule the program of 2nd_code_theAutoit.exe
which kick starts the 1st code, made by JDA, and saved in .ps1 file.

I discovered it won’t interrupt the storagenode windows GUI,
and the logs are cleaned as often as You want,
i set it daily in Task Scheduler, like every 2 days lately, makes me a 350-500MB files,
on other busy node 1,2-2,5GB files every 2 days.

[BigDaddio]

This is what i do, I just use a scheduled task on the 1st of the month to run it, creates a new folder for the old exe versions and log files and moves the files into them. pretty basic but works for me…

I should add I mostly use the batch file, the powershell script, while it worked well manually, was a bit problematic setting the scheduled task on a couple of PCs

Batch file

@echo off
set "verPath=C:\Program Files\Storj\Storage Node\old_versions"
set "logPath=C:\Program Files\Storj\Storage Node\logs"

if not exist "%verPath%" (
    mkdir "%verPath%"
)
if exist "%ProgramFiles%\Storj\Storage Node\storagenode.old.*.exe" move "%ProgramFiles%\Storj\Storage Node\storagenode.old.*.exe" "%verPath%"


if not exist "%logPath%" (
    mkdir "%logPath%"
)
net stop storagenode
move "%ProgramFiles%\Storj\Storage Node\storagenode.log" "%logPath%\storagenode-%date:~10,4%-%date:~7,2%-%date:~4,2%_%time:~0,2%-%time:~3,2%.log"
net start storagenode
exit 0

Powershell version

$exePath = "C:\Program Files\Storj\Storage Node\old_versions"
$logPath = "C:\Program Files\Storj\Storage Node\logs"

if (!(Test-Path $exePath -PathType Container)) {
    New-Item -ItemType Directory -Force -Path $exePath
}
mv "$env:ProgramFiles\Storj\Storage Node\storagenode.old.*.exe" "$env:ProgramFiles\Storj\Storage Node\old_versions"


if (!(Test-Path $logPath -PathType Container)) {
    New-Item -ItemType Directory -Force -Path $logPath
}
Stop-Service storagenode
mv "$env:ProgramFiles\Storj\Storage Node\storagenode.log" "$env:ProgramFiles\Storj\Storage Node\logs\storagenode-$(date -Format 'yyyy-MM-dd_hh-mm').log"
Start-Service storagenode

[Alexey]

You may use my version too:

GitHub - AlexeyALeonov/storagenode_win_log_rotate: Windows GUI storagenode...

or this script:

Native logs rotation in Windows with a simple PowerShell script

Hello, Unfortunately, in the current windows version of the Storage Node there is no way to rotate or limit the size of the generated logs files. You cannot delete them from the GUI because they are locked by the services processes. Using Clear-Content in PowerShell to reset the files works but there is no easy solution if you want to keep a few days / weeks of logs. I have written a small PowerShell script that can rotate the Storjs Logs without the need of Stop>Start the services to release …

However, perhaps better is to use a native logrotate:
https://sourceforge.net/p/logrotatewin/wiki/LogRotate/