High Trash usage - 27 TB - no more uploads

Node Operators troubleshooting
29

Only wait until all used-space-filewalkers will finish the scan since the last restart.
Because right now you logs shows only several starts of the used-space-filewalkers but only a few completed.

I have a suspicion that we might introduced a bug in used-space-filewalker, when we tried to implement a continuation of the used-space-filewalker.
You can test that:

  1. Stop and remove the container (or stop the service, if you use a service based setup)
  2. Rename the used_space_per_prefix.db database to used_space_per_prefix.db.bak
  3. Move all databases (*.db) to the temp folder
  4. Start the node, wait until it would re-create all databases
  5. Stop the node
  6. Move databases from the temp folder back to the storage location with replace
  7. Start the node
  8. Wait until all used-space-filewalkers will finish the scan without issues (no database or filewalker errors)
  9. Check the dashboard