What is the high CPU and disk activity at storagenode start-up?

Node Operators FAQ
1

I found several posts on here regarding high CPU usage and lots of IO wait issues in general… but that doesn’t seem to be what I’m seeing. I have a question regarding what happens after a storage node is started.

When the storage node(s) start on all 3 of my hosts, it seems that for at least ~30-60 minutes after they start up, there is a high CPU load. ‘top’ shows only a few percent usage for the storagenode process, but the load on my system is basically pegged at 1.0 for awhile after the docker container starts up. The HDDs are also very active during this time. (I’m not seeing high iowait either).

To be clear, I don’t think there is a problem here, I’m just curious what it is doing.

I’ve looked in the logs and haven’t seen any messages about what the activity could be. And the node appears to be serving requests without any problem. So I’m just wondering what is it doing? Is there a start-up process that scans over all the files on the node? Is it doing random spot checks to make sure files are there and valid?

Thanks!

2

It’s the filewalker process that scans the storage directory for all pieces to update its database and the space used by the storagenode. It doesn’t use much CPU but causes a lot of IOWait.

2 Likes
3

Ahh, ok. I figured it had to something like that. Maybe I’m missing them, but, I wonder if there are logs that speak to that activity. It might be nice to see periodic update messages of “Filewalker started”, “1000 files checked”, “2000 files checked” etc. And/or at least a “filewalker process complete, results: XXXXX files space usage updated success.”

1 Like
4

Maybe some of those messages are DEBUG log level. You can check that if you change your debug level in your config.yml but I’m not sure.

1 Like