Are there any specific log entries I could observe that would signify when the file walker process starts and ends? I don’t see any at the INFO level, and as the file walker process consumes a lot of resources, I suspect it would be good to see it there.
This has been suggested before in a thread, but I don’t think it made it to a poll or github issue.
Voted. This is important information for SNOs in my opinion.
Was bored, peeked into some source code. Though, I don’t speak golang, so I might be wrong in some parts.
As far as I understand, there are two uses for what the community here calls the file walker process:
- estimate the amount of disk space used by blobs (of a specific satellite or of all satellites, functions
- run the bloom filter (function
Both essentially call
stat() on each blob for good reasons. The latter already has a DEBUG-level log item. So all it would take to implement this request would be to annotate the first two functions with log items before and after the call to
WalkSatellitePieces, and maybe bumping the level of the existing log entries in the third one to INFO.
That sounds right to me.
One more idea here. It would be nice to see how many files has been marked for removal by the bloom filter during the file walking process. This statistic would be helpful to estimate the effect of downtime on storing files that were supposed to be deleted.