This is something new for me. What is it?
That is garbage collection live in action.
This is normal behaviour, nothing to worry about unless it’s excessive.
Your node stores more pieces than the satellite expects it to. This might be because your node was offline while a piece was requested to be deleted, and your node simply missed this request. This might also be because your node actually lost a race during the upload, but managed to receive the whole file before the uploader dropped its connection to your node. These messages signify cleaning up some if these superfluous pieces. These pieces are not immediately deleted, but moved to a special directory called trash, where they wait for about a week just in case the garbage collection procedure made a mistake. After a week they will be deleted.
Having some amounts of trash (and hence some amounts of these superfluous pieces) is probably unavoidable with the current design of Storj. You should worry about trash only if its size, as reported by the dashboard, becomes a significant chunk of the whole storage. I’d start worrying about trash if it becomes more than 10% of stored data, though this is just my opinion and there is no official threshold saying it’s not ok anymore. Seeing large amounts of trash means there are some connectivity problems that stop your node from eagerly remove no longer useful data.
IIRC garbage collection happens roughly every five days. It involves verifying all of your stored pieces using a data structure called a Bloom filter, you may find some more information about how it works in this thread. The process involves some increased workload on your storage, you may observe your drives being more busy for some period. Again, nothing to worry about, this is normal.
2 Likes
Thanks, but I’m specific about those “About to move piece to trash”, I’ve witnessed a lot of lines like
2023-01-02T08:05:38.455Z INFO piecedeleter delete piece sent to trash {"Process": "storagenode", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "627A6P6JYBSIDI64KM5L6YRWEXAOC3VSFXEUPHAHQRB22H3Q7TZQ"}
But never before have I seen those “About to move”
This is exactly a message stating that a piece with ID 627A6P… was recognized as trash during the garbage collection process.
1 Like
Just out of curiosity. It doesn’t impact log size so much, just some debug lines here and there
1 Like