Hey everyone, just a quick reality check! While many of us are feeling optimistic that the problem in the topic When will "Uncollected Garbage" be deleted? is solved, it’s important to note that we’ve only patched things up for now. The underlying issue is still hanging around.
Not all TTL data is getting removed by the collector because some TTL entries don’t make it into the database. The real fix for that hasn’t been released and fully tested yet. Instead, we’ve got a workaround where uncollected TTL data gets cleaned up by BF and GC. But it’s not perfect—there’s still that 7-day trash time and the waiting game for the BF to do its thing.
So, while we’re on the right track, the journey isn’t over yet. I’m eagerly anticipating the fix to be released and put to the test!
The problem of this topic is resolved. The problem which led to the uncollected garbage is not yet.
The 10TB garbage on my 14TB took just short of 10 days to be collected.
Now it looks like it will take the same amount of time to empty the trash.
During this time the used and trash values are not updated and I’m not receiving new data even though I have free space physicly on this disk now. I would prefer that these values would be updated after completion of each xx folder.
Well, if we’re being precise, this topic isn’t fully resolved just yet since your ‘Uncollected Garbage’ hasn’t been completely deleted yet as you point out yourself. But it’s in progress, so just a little more patience!
I totally get the desire to update those values after each xx folder – it’s a neat idea! However, dealing with TBs of data isn’t something we encounter every day. Under normal circumstances, there’s no urgent need, and adding extra database operations will not contribute to performance. So, I’m a bit skeptical it will get implemented.
Instead of hijacking the release v1.112 thread, I figured it’s better to continue the conversation here.
Maybe I’m missing something, but I’m scratching my head trying to figure out why this feature didn’t make it into v1.112. It’s controlled by a switch, right? So @Alexey
and @littleskunk, why not release it and let the SNOs test it out?
Just to clarify—does the fix not fully resolve the issue, or is the bigger concern around the migration and the potential flood of TTL data on the horizon? Would love to understand it a bit better! Thanks.
@littleskunk A possible solution might be that the collector process checks both implementations when the flat file storage is enabled. This way also entries in the db will be cleaned up on expiration. Could that work?
Yes that would work but it isn’t implemented. I don’t know what the current status is. The ticket looks to me as if they wanted to close the bug with just 50% of the fix merged.
I see this feature is rolling out in version v1.113—great news! Were there any updates made to address the migration issue, or is that still in the works? Just checking in to see if we’re moving forward on that front. Thanks!
I now see my suggestion to check the piece expiration database next to the flat file during the collect process was implemented but did not make it into v1.113 yet.
trying to figure out why