Native logs rotation in Windows with a simple PowerShell script

Hello,

Unfortunately, in the current windows version of the Storage Node there is no way to rotate or limit the size of the generated logs files. You cannot delete them from the GUI because they are locked by the services processes. Using Clear-Content in PowerShell to reset the files works but there is no easy solution if you want to keep a few days / weeks of logs.

I have written a small PowerShell script that can rotate the Storjs Logs without the need of Stop>Start the services to release the files.

Simply save the code below in a .ps1 file and change the variables to you needs. Then every time you start the script (can be scheduled easily in a task) it will:

• Look for any *.log files in $PathLogLive

• Copy all those logs to $PathLogArchive in a new file with a datetime suffix (you can change the format with $DateTimeFormat)

• Clear the existing logs

• Delete any *.log file in the to $PathLogArchive that is older that $RententionInDays

It’s been running on my node on a scheduled task for a few days without issues. Hope this helps!

PowerShell source code:

#-----------------------------------------------
#-------------------- INFOS --------------------
#Script:   StorjLogsRotation
#Version:  1.0.1.2
#Date:     27/04/2020
#Author:   JDA
#------------------- Variables ------------------
$PathLogLive      = 'S:\Storj\Storage Node\'
$PathLogArchive   = 'S:\Storj\Storage Node\Logs\'
$DateTimeFormat   = "yyyy-MM-dd HH'h'mm"
$RententionInDays = 8
#-----------------------------------------------


Function StorjLogsRotation{
 ForEach($L in (Get-ChildItem -LiteralPath $PathLogLive -Filter '*.log' -Recurse:$False)){
  RotateLog -Source $L.FullName -DestinationPath $PathLogArchive -RententionInDays $RententionInDays
 }
}


Function RotateLog([String]$Source, [String]$DestinationPath,[Int]$RententionInDays){
 $Format = '{0}_[{1}]{2}'

 If(Test-Path -LiteralPath $Source){
  $S = Get-Item -LiteralPath $Source
  $Destination = Join-Path -Path $DestinationPath -ChildPath ($Format -F $S.BaseName, ((Get-Date).ToString($DateTimeFormat)), $S.Extension)
 
  If(!(Test-Path -LiteralPath $DestinationPath)){$Null = New-Item -Path $DestinationPath -Type Directory -Force}
  Copy-Item -Path $Source -Destination $Destination -Force
  Clear-Content -LiteralPath $Source -Force

  Get-ChildItem -LiteralPath $DestinationPath -File -Filter ($Format -F $S.BaseName, '*',$S.Extension) | ? LastWriteTime -le ((Get-Date).AddDays(-$RententionInDays)) | Remove-Item -ErrorAction SilentlyContinue
 }
}

StorjLogsRotation

PS: Depending on your disk speed, every time the logs are rotated there is a small change you loose a few lines of logs.

If your copy takes too long, wouldn’t the copy contain older values than the current log file. Stopping makes sure no entries are lost.

Yes, it is not impossible, but stopping / restarting the node daily have disadvantages also. It hurt downtime and the node restart is heavy on disk if you store lots of data (probably some kind of health check going on). Logrotate does something similar, so the risk is the same, but unless your disks are really slow the risk is minimal.

For me node uptime and stability are far superior to a few lines of logs that could be missing, I’m just providing an alternative to something that should be implemented in the program. Log rotation is a basic function that any service should implement by himself (and that way it guaranties you don’t lose anything.)

PS: I have updated the post with a warning about it

doing that has zero effect on the log file

You cannot delete the logs without stopping the Services. You can zero them with a command line, but no from the GUI. I have updated the post to make it clearer about the why you could need this script.

Furthermore if a significant number of nodes restarts scheduled at midnight, that could hurt file availability. Please never schedule a restart at set moments.

Amen.

I am doing this by hand, time to time, with my Toolbox.

done by 2 click, and it not delete it only archive

i take it that this is related to the windows storagenode software that doesn’t use docker?

Yes only tested on the native service version of Storage Node.
I was usind docker on windows at first but not real reason to continue now that you have a service.

most likely faster that way anyways…

Faster i dont know but it sure take less ressources and it’s way easier to install and maintain.

Yeah it is only usable with GUI version. I not see any advantage to use docker in windows at all, it like emulate windows on windows.

The script works great.

I did find that to run it as a scheduled task that I had to pass these parameters to powershell on windows. I had to bypass the execution policy for the script.

-executionpolicy bypass -File C:\directoryName\logCleaner.ps1

100% agree, thank you for the script
my logfile got up to 12GB size as i was not aware that storj is missing logrotation.
sometimes i wonder how they do “all that fancy stuff” without having such basic “no brainers” implemented

Usually log rotation is integrated to the OS. Except Windows. Windows has log rotation for system events though, but not usual text logs.
So, you need to use some additional tools like LogRotateWin / Wiki / LogRotate or such PowerShell scripts.

Most Windows software I know handles this by creating a new log file for each day. That way another script can fake rotate the logs by deleting every log file except the last X.
X is the number of days you wanna keep.

Thank you! I just got done setting up multiple nodes, and this works great! I already had my nodes configured to keep each node’s log file in the same folder, originally so I could keep an eye on the log file sizes in one place - but I’m glad I set it up that way because after updating the 2 paths in your script, and then adding it to the task scheduler, it works perfectly for all of the log files!