is it normal to have so many storagenode processes running ?
Youāve cut off all the interesting bits.
Run ps -axw -o command | grep storagenode, you will get more clarity.
1 Like
You probably have the Hide userland process threads htop switch off. These are threads, not really full separate processes.
1 Like
so i managed to get the wait i/o under control. besides removing some other services accessing the the drive (such as plocate, that i never used) i did the following changes to storagenode config:
- set
log.custom-level: piecestore=FATAL,collector=ERRORas without this I got a 270 GB log file in a week - storage2.piece-scan-on-startup: false
- pieces.enable-lazy-filewalker: false
Everything seems fine, too fine even. The issue is, that after these changes I have no filewalker events in my logs. Did i disable them somehow? Did I disable the logging for them? What would be the impact of filewalker not running? Trash space on the dashboard seems to be changing from time to time, so Id assume garbage collection is running (although my trash is 10% of the used space)
what is your overall log level? it needs to be INFO to see filewalker messages.
you can grep the log for the word āemptyā or āusedā or āretainā to see walker status.
2 Likes
That makes me smile:
āHelp Iāve turned off some of my log messagesā¦ and now I canāt see some of my log messages!ā
2 Likes
my logs in general are still on info
lol,i turned on the piecestore messages because i was getting insane amount of logs. i was not expecting to loose filewalker logging.
i still have my old log file and rechecked the events, actualy there were two filewalker events:
- pieces used-space-filewalker - i have turned this off on startup with the
storage2.piece-scan-on-startup: falsesetting. but it should still run periodically, no? - lazyfilewalker.* something - i have also turned this off with
pieces.enable-lazy-filewalker: falsebut i was expecting this to be replaced by some zealousfirewalker or whatever unlazy stuff
so disabling lazy-filewalker disables filewalker totally, instead of running the same stuff, just not in a lazy mode?
also please note my great achievement, that after multiple frustrating experiences i have realized its filewalker and not firewalker, as i initially thought it is
As far as I know, used-space-filewalker only ever ran on-startup - so the node knows if anything changed while it was off. So the reason it was still run periodically by SNOs was because of the automated upgrades/restarts every few weeks. While a node is running it tracks all other used-space changes dynamically (and saves-state I think hourly?).
1 Like
This disables only the lazy mode. So, you should see some filewalkers, however with this:
You unlikely will see it.
But you can always use a debug port to check, what processes are running.