Reading 100% and "ERROR piecestore failed to add bandwidth usage"

Node Operators troubleshooting
1

Hi, Today my nodes are using all HDD in reading and I start to see ERROR piecestore failed to add bandwidth usage in my logs.
How can I check if filewalker is running?

2

The so-called filewalker runs in two cases:

  1. [Covers pieces for all satellites] When the node process starts, to calculate how much space is being used
  2. [Covers pieces for one satellite] During garbage collection

I don’t think there are log lines you can look at for the first case, but if you recently started your node then that is probably the culprit.

For the second case, the log line "Prepared to run a Retain request" is output at INFO level before the scan begins, and "Moved pieces to trash during retain" is output at INFO level after the scan is complete.

2 Likes
3

I have a lot of " INFO piecedeleter delete piece sent to trash". How does it work? How long it takes? How often thi happen? My hdds are all 100% busy on all nodes for two days

PS: I reached maximum node space in my config “1tb” 3/4 days ago…

4

if the customer deletes objects, their pieces are moved to the trash. After 7 days they will be deleted permanently. These requests can come in any time.
There is also a garbage collector, which moves previously deleted pieces to the trash if your node were offline or busy to perform the delete request immediately. It runs periodically, roughly once per week.