Are there any specific log entries I could observe that would signify when the file walker process starts and ends? I don’t see any at the INFO level, and as the file walker process consumes a lot of resources, I suspect it would be good to see it there.
2 Likes
This has been suggested before in a thread, but I don’t think it made it to a poll or github issue.
2 Likes
Voted. This is important information for SNOs in my opinion.
1 Like
Was bored, peeked into some source code. Though, I don’t speak golang, so I might be wrong in some parts.
As far as I understand, there are two uses for what the community here calls the file walker process:
- estimate the amount of disk space used by blobs (of a specific satellite or of all satellites, functions
SpaceUsedBySatelliteandSpaceUsedTotalAndBySatelliteinpieces/store.go). - run the bloom filter (function
retainPiecesinretain/retain.go).
Both essentially call stat() on each blob for good reasons. The latter already has a DEBUG-level log item. So all it would take to implement this request would be to annotate the first two functions with log items before and after the call to WalkSatellitePieces, and maybe bumping the level of the existing log entries in the third one to INFO.
4 Likes
That sounds right to me.
2 Likes
One more idea here. It would be nice to see how many files has been marked for removal by the bloom filter during the file walking process. This statistic would be helpful to estimate the effect of downtime on storing files that were supposed to be deleted.
1 Like