I am running a node on windows, and currently the storagenode.log file is a combination of tab delimited text and JSON.
I would like to propose changing it to store the log information in SqlLite. A table with the appropriate fields would make storing and querying the data much easier. This would also have the benefit of running a truncate command to clear out old entries so the size of the log is more manageable.
Welcome to the forum @mfinnerty !
As someone who has had to deal with corrupted DBs. I strongly oppose adding another sqllite DB in to the mix.
Storing INFO-level log entries would increase per-request random I/O even more, and random I/O is already a bottleneck for SNOs. Storing a subset of logs, let say, WARN and above, would be nice though, it would make it easy to display these messages in the web console. But would it then be as useful for you? In a healthy there’s not many WARN/ERROR-level entries anyway, so one doesn’t really need a complex tool for querying them.
This would be a database that you can throw away and recreate from scratch any time, so it shouldn’t be a problem.
And lose your logs. Logs would be useful in case of db corruption or some other problem, but if the log db is corrupt then you might as well have no logs.
Logging to a database may be interesting, but I do not like sqlite. Something like mysql would be better IMO. Still, the table would need to be partitioned or something because large tables are usually slow and the mysql db is more susceptible to corruption than a text file.
I would not like to have 50GB or larger sqlite database (I do not delete logs). Mysql would be better, but splitting the log to multiple text files is probably better than mysql.
Can’t you set up log rotation on Windows? I use docker on Linux, so my setup would not work for you.
I think what you actually suggesting, is to not reinvent the wheel with SQLite and instead use the OS native logging mechanism. Windows has one (event log), macOS has one (log utility, see man log). They have benefit of saving tons of performance by not evaluating format strings at the wite time, only at display time, (I.e. they store format string and arguments, not completed string), are space efficient, and have a benefit of already existing and shipping with the OS.
The problem is — most storage nodes are not run on windows or macOS. Linux and FreeBSD, as far as I know, lack the standardized logging facility. It’s all text files there.
This is not a bad thing. While it’s probably possible to install an off the shelf logging facility to these oses with similar features, this facility can fail. It’s an extra level of complexity. Logs are needed to triage for when something went wrong. If that something is also affecting logging system — well, now you can’t debug. Text files as dumb, and it’s their benefit. You can delete them, you can compress them with logrotate or newsyslog or whathaveyou, and you can grep and analyze them with standard tools.
I don’t think disk IO associated with logs is a problem here that needs solving. You can alsways add a $10 SSD for them if that is a problem, but likely there are other issues with the system that need to be addressed first before writing text files becomes a bottleneck.