Hmm, thanks. It looks like there are some debug logs that would not have been printed with the default log level (example) - thanks @littleskunk for finding this
There are three main ways data could be deleted (thanks @Egon for outlining these)
- deletion due to user request (this comes from the satellite)
- deletion due to expiration
- deletion due to garbage collection
Based only on the information here, it is difficult for me to determine exactly what the trigger for these deletes was.
If a piece was deleted because it expired or was not in the garbage collection bloom filter, and a subsequent request came in to delete the piece, the behavior here would make sense. But it does not seem to me like that would be possible except in a very rare race condition.
If you are willing to try it out, I would suggest:
- set log level to debug
- if you see this piecedeleter error again, check the logs and see if any additional information about this piece ID was logged
Unfortunately because (I’m assuming) your logs don’t go infinitely back, we will be limited in our abilities to try to figure out what happened with the pieces you’ve posted about already, but with debug logs, we might find out more information if it happens again. Since you shared a few different times of it happening, it seems likely that it would happen again.