Am I the only one extremely annoyed by log sizing. I would prefer to store long term logs of relevant issues. However dealing with a massive amount of “context canceled” that should technically be living in the “info” verbosity of the log data is an issue. Moving to fatal removes valuable log entries.
The log size, is so bad, that it’s an obvious issue for new operators. People don’t even know what to look for etc. I feel like improving the log verbosity dataset could help quite a bit in the management of nodes.
Yea, and i can share what i did, i used this script here, and i keep “info” level, and copy them every 3 days to BIG HDD, and then i plan to winrar them with “best” it compress that very well.
Neither of those are really solutions when it comes to properly managing a service. It’s like a needle in a hatystack theory, and your solution is to bail all the hay. Which will likely make finding the needle harder…
By compressing the logs you can keep years of history, and all your log analysis still works. You can still grep for whatever you want. Want to see errors? Filter by errors. Want to see context around — add context flags (e.g. -B 10). In case you lost a piece and need to track it down for example.
I’m the first one to advocate to delete databases on every start as they are completely useless and at the same time I’m the one advocating for keeping full logs forever.
More logs is always better than fewer, especially highly compressible logs like storagenode is producing that are essentially free to store.
It happened to me that chunk was uploaded, deleted, and failed audit, in that order. Was not my nodes fault, was some experiment storj was running, but keeping all logs was helpful to figure this out. I never delete logs.
I think, OP wants the log to be more trimmed by not showing context canceled and other “irrelevant” entries thereby making logs shorter. OP is not looking for options on how to rotate the logs.
You can also always switch to the error log level only (or warning).
I personally uses a logrotate and the info log level on case if we introduce a bug. My logs keeping for the last 3 years.
This helped me to prove a bug at least once (my node was disqualified and then restated because I have had an evidence).
Ummmmm, I have no clue why this convo is degrading so badly. The verbosity of the logs is not organized correctly. If someone wants to save all the logs and rotate them that’s fine, use info. If someone wants to use error, they should be able to use error without a bunch of garbage.
If it cheers anyone up, I put in a PR last night to move the “context canceled” stuff to Info level, though Storj’s CI seems to not like it very much - the tests worked on my machine (though I only ran them for that specific part since that’s all I changed) and that part appears to pass tests on their end, but it looks like other parts aren’t happy with the change I made so I’ll need to sit down and do some debugging over the next couple of days. Don’t worry, the log spam has been irritating me too
Ha, no worries! I’m pretty new to the language Storj is written in too, I just had an idea of where I should be looking. Right now I probably need someone more experienced than me to take a look and see if the errors are my fault or not, since I don’t really know how to read Go’s test output - the test failures seem to be in modules that I didn’t touch (and that GoLand seems pretty confident don’t interact with the parts of the code I changed when I asked it to check references for me). Either way, dipping my toes in and learning on a project I actually use is always fun
(my node was disqualified and then restated because I have had an evidence).