Checking log after error, but file is too big to open - 17GB

Node Operators troubleshooting
6

I use this powershell script to split my large log files:

$from = "C:\Program Files\Storj\Storage Node\Logs\storagenode.log"
$rootName = "C:\Program Files\Storj\Storage Node\Logs\storagenode.2022.09"
$ext = "log"
$upperBound = 1000MB

$fromFile = [io.file]::OpenRead($from)
$buff = new-object byte[] $upperBound
$count = $idx = 0
try {
    do {
        "Reading $upperBound"
        $count = $fromFile.Read($buff, 0, $buff.Length)
        if ($count -gt 0) {
            $to = "{0}.{1}.{2}" -f ($rootName, $idx, $ext)
            $toFile = [io.file]::OpenWrite($to)
            try {
                "Writing $count to $to"
                $tofile.Write($buff, 0, $count)
            } finally {
                $tofile.Close()
            }
        }
        $idx ++
    } while ($count -gt 0)
}
finally {
    $fromFile.Close()
}

Just change the $from and $rootName to your liking and you’ll get 1GB files.

Note - it doesn’t care about lines so will split the files in the middle of a line.

4 Likes