# When will "Uncollected Garbage" be deleted? **URL:** \ **Category:** troubleshooting\ **Tags:** ttl, garbage-collector\ **Created:** [July 28, 2024, 8:32pm UTC](https://forum.storj.io/t/when-will-uncollected-garbage-be-deleted/27327 "2024-07-28T20:32:08Z")\ **Posts on this page:** 20\ **Page:** 15
**Author:** ![elek](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/elek/32/9189_2.png) [@elek](https://forum.storj.io/u/elek)\ **Post date:** [August 23, 2024, 1:48pm UTC](https://forum.storj.io/t/when-will-uncollected-garbage-be-deleted/27327/300 "2024-08-23T13:48:25Z")
> [@BrightSilence](#): > > And just to satisfy my own curiosity, what happens if there are 1000 files that need to be included, but you calculate the BF size assuming 500 files and thus try to generate a BF with too low a size? Would it fail to generate? Would it suddenly have a false negative rate? (Which would be really bad) Probability will be changed. Try out this calculator: > **[Bloom filter calculator](https://hur.st/bloomfilter/?n=10000000&p=&m=4MB&k=1)** > > Calculate the optimal size for your bloom filter, see how many items a given filter can hold, or just admire the curvy graphs. Also borrow my MIT licensed Javascript for your own programs. Use 10M / 4Mb / 1 hash. And change it to 2Mb. For me it shows that the false-postitivate rate will be increased from 26% to 46%. Still it will match all the 10M (good fishes), but you will catch more bad fishes. Let me try an other inaccurate metaphor: šŸ˜‰ You would like to send a vase to me. You need a box, the vase, and some packing peanuts to fill the gapsā€¦ You need a box which is big enough, but not too big (because it would be too expensive), it should have enough space for the packing materials around the vase. Bloom filter should be big enough to include the (hashed bits of) items, + some additional space for zeros to decrease false positive matches. With smaller box, but with the same vase, you will have less space for packing peanutsā€¦ which increases the false positive rate (or the chance of a broken vase ;-)) ) ---
**Author:** ![alpharabbit](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/alpharabbit/32/1129_2.png) [@alpharabbit](https://forum.storj.io/u/alpharabbit)\ **Post date:** [August 23, 2024, 3:30pm UTC](https://forum.storj.io/t/when-will-uncollected-garbage-be-deleted/27327/301 "2024-08-23T15:30:18Z")
In my slc blobs folder I still have over 50% of files older than 30 days. Shouldnā€™t TTL or BF have deleted those files? ---
**Author:** ![elek](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/elek/32/9189_2.png) [@elek](https://forum.storj.io/u/elek)\ **Post date:** [August 23, 2024, 7:25pm UTC](https://forum.storj.io/t/when-will-uncollected-garbage-be-deleted/27327/302 "2024-08-23T19:25:51Z")
Depends on the files. There can be old, immortal data, but new testdata is uploaded with TTL. TLDR; wait for the next BF run(s), which will be way better. Check it on Monday. ---
**Author:** ![BrightSilence](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/brightsilence/32/6048_2.png) [@BrightSilence](https://forum.storj.io/u/BrightSilence)\ **Post date:** [August 23, 2024, 7:36pm UTC](https://forum.storj.io/t/when-will-uncollected-garbage-be-deleted/27327/303 "2024-08-23T19:36:32Z")
> [@elek](#): > > Probability will be changed. Yeah, I had a brain fart with my previous post. Somehow I thought there would be a minimum size required to match all pieces. But it would eventually just degrade into 100% false positives. The calculator is useful, thanks for that link. But the vase analogy didnā€™t really work for me. Regardless, I get it now. Though you didnā€™t mention whether my assumption here was correct. > [@BrightSilence](#): > > Maybe I could make an educated guess based on the rest of your post, but please correct me if Iā€™m wrong. > The BF creation process requires you to specify the BF size beforehand. So you calculate which size would lead to 10% false positive rate based on the number of files. So when those expired files are part of the count you generate a larger BF than necessary. Nor this question > [@BrightSilence](#): > > Assuming the above is correct, does that mean those BFā€™s actually get a lower than 10% false positive rate? Iā€™m thinking yes to both, based on what you said and the calculator. But just wanted to check. ---
**Author:** ![Alexey](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/alexey/32/41_2.png) [@Alexey](https://forum.storj.io/u/Alexey)\ **Post date:** [August 24, 2024, 1:07pm UTC](https://forum.storj.io/t/when-will-uncollected-garbage-be-deleted/27327/304 "2024-08-24T13:07:50Z")
I would add - please, expect a large amount of deletions. I would say about 50% or more. ---
**Author:** ![donald.m.motsinger](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/donald.m.motsinger/32/71_2.png) [@donald.m.motsinger](https://forum.storj.io/u/donald.m.motsinger)\ **Post date:** [August 24, 2024, 5:21pm UTC](https://forum.storj.io/t/when-will-uncollected-garbage-be-deleted/27327/305 "2024-08-24T17:21:40Z")
> [@elek](#): > > wait for the next BF run(s), which will be way better. Check it on Monday I got a BF today, but the result is disappointing. Or will there be another BF on Monday? ```auto 2024-08-24T13:03:54Z INFO retain Prepared to run a Retain request. {"Process": "storagenode", "cachePath": "config/retain", "Created Before": "2024-08-17T17:59:59Z", "Filter Size": 35000003, "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE"} 2024-08-24T16:01:30Z INFO retain Moved pieces to trash during retain {"Process": "storagenode", "cachePath": "config/retain", "Deleted pieces": 81848, "Failed to delete": 0, "Pieces failed to read": 0, "Pieces count": 37422647, "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Duration": "2h57m36.019750481s", "Retain Status": "enabled"} ``` If the reports from SL are correct, then I have 0.73TB paid data from this satellite. The used-space-filewalker from yesterday reported 11TB data in that folder. ```auto 2024-08-23T12:18:36Z INFO pieces used-space-filewalker completed {"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Lazy File Walker": false, "Total Pieces Size": 11064824212224, "To tal Pieces Content Size": 11027026300672} ``` > [@Alexey](#): > > I would add - please, expect a large amount of deletions. I would say about 50% or more. I wish you were right. ---
**Author:** ![Mark](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/mark/32/1128_2.png) [@Mark](https://forum.storj.io/u/Mark)\ **Post date:** [August 24, 2024, 9:46pm UTC](https://forum.storj.io/t/when-will-uncollected-garbage-be-deleted/27327/306 "2024-08-24T21:46:15Z")
> [@donald.m.motsinger](#): > > `"Filter Size": 35000003` Does that mean the bloom filter was 35MB? Is that the Max size currently allowed? > [@donald.m.motsinger](#): > > If the reports from SL are correct, then I have 0.73TB paid data from this satellite. A 35MB BF for 0.73TB of paid data? Is that reasonably possible? Seems strange. > [@donald.m.motsinger](#): > > ` "Pieces count": 37422647` Thatā€™s a lot of pieces. Hopefully the 0.73TB satellite report is just temporarily broken. Are you looking at the report for a single day or the ā€œAverageā€ displayed at the top of the graph? > [@donald.m.motsinger](#): > > `"Deleted pieces": 81848` So it sent 0.2% of your pieces to the trash? That doesnā€™t seem like very much, but I guess if the TTL collector is deleting files correctly there shouldnā€™t be much remaining for the BF to delete right? I donā€™t currently run any SLC nodes. Is all SLC test data set for a TTL of 30 days? Just out of curiosity, Have you looked inside one of your blobs/xx folders for SLC to get an idea of how old your pieces are? Perhaps you could check to see if most of the files in one of those folders are less than 40 days old? ---
**Author:** ![Alexey](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/alexey/32/41_2.png) [@Alexey](https://forum.storj.io/u/Alexey)\ **Post date:** [August 25, 2024, 2:36am UTC](https://forum.storj.io/t/when-will-uncollected-garbage-be-deleted/27327/307 "2024-08-25T02:36:41Z")
> [@Mark](#): > > A 35MB BF for 0.73TB of paid data? Is that reasonably possible? Seems strange. Because it includes the TTL pieces which were not cleared yet on the satelliteā€™s side. It should be improved with the latest change, but I would expect the effect only on the next week. > [@Mark](#): > > Hopefully the 0.73TB satellite report is just temporarily broken. Likely yes, if itā€™s significantly differ from a previous report (you should pay attention to the daily reports, not the average used space). > [@Mark](#): > > but I guess if the TTL collector is deleting files correctly there shouldnā€™t be much remaining for the BF to delete right? Yes, itā€™s expected behavior. However, since not all TTL pieces were registered in the TTL database, these orphaned pieces still should be considered as a garbage and should be excluded from the BF independently of were they deleted from the satelliteā€™s databases or not, because they are expired already and should be removed anyway. After the latest change > [@elek](#): > > delete expired objects with GC even if satellite was slow to delete TTL pieces: [https://review.dev.storj.io/c/storj/storj/+/14272](https://review.dev.storj.io/c/storj/storj/+/14272) the GC should remove these orphaned TTL pieces too. ---
**Author:** ![Mark](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/mark/32/1128_2.png) [@Mark](https://forum.storj.io/u/Mark)\ **Post date:** [August 25, 2024, 3:15am UTC](https://forum.storj.io/t/when-will-uncollected-garbage-be-deleted/27327/308 "2024-08-25T03:15:20Z")
> [@Alexey](#): > > Because it includes the TTL pieces which were not cleared yet on the satelliteā€™s side. Interesting. I wonder how long the delay has been lately for the satellite to clear TTL pieces on its side. ---
**Author:** ![Alexey](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/alexey/32/41_2.png) [@Alexey](https://forum.storj.io/u/Alexey)\ **Post date:** [August 25, 2024, 3:34am UTC](https://forum.storj.io/t/when-will-uncollected-garbage-be-deleted/27327/309 "2024-08-25T03:34:58Z")
I do not have an access there, but I believe it struggles with these deletions of millions of segments, otherwise the GC would collect much more expired data, which was not collected by the TTL collector. ---
**Author:** ![andrew2.hart](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/andrew2.hart/32/6415_2.png) [@andrew2.hart](https://forum.storj.io/u/andrew2.hart)\ **Post date:** [August 25, 2024, 5:43am UTC](https://forum.storj.io/t/when-will-uncollected-garbage-be-deleted/27327/310 "2024-08-25T05:43:49Z")
I got a TB of uncollected garbage ā€œcollectedā€ last night ---
**Author:** ![jammerdan](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/jammerdan/32/1121_2.png) [@jammerdan](https://forum.storj.io/u/jammerdan)\ **Post date:** [August 25, 2024, 5:51am UTC](https://forum.storj.io/t/when-will-uncollected-garbage-be-deleted/27327/311 "2024-08-25T05:51:48Z")
Same here. Some massive amount vanished. It looks like it has been moved to trash. ---
**Author:** ![edo](https://storj.bcdn.literatehosting.com/letter_avatar/edo/32/5_5575768a8748004e209b776fc1b2916d.png) [@edo](https://forum.storj.io/u/edo)\ **Post date:** [August 25, 2024, 6:49am UTC](https://forum.storj.io/t/when-will-uncollected-garbage-be-deleted/27327/312 "2024-08-25T06:49:25Z")
![storj gc slc increase](https://storj-s3.bcdn.literatehosting.net/original/3X/0/0/0096201f14e80b6728c3fce080bc564bef6ec029.png) I can confirm! I received a BF from SLC on one of my nodes, which added more than 400 GB to the trash. But hereā€™s the thing: this node originally had 2 TB of uncollected garbage, so a bit less than 25% has been trashed so far. šŸ˜… @Alexey and/or @elek, do we expect the next SLC BF to take out more of the garbage, or are the BF/GC issues still playing hard to get? Fingers crossed we get this cleaned up soon! ---
**Author:** ![Alexey](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/alexey/32/41_2.png) [@Alexey](https://forum.storj.io/u/Alexey)\ **Post date:** [August 25, 2024, 6:51am UTC](https://forum.storj.io/t/when-will-uncollected-garbage-be-deleted/27327/313 "2024-08-25T06:51:35Z")
I think it should capture much more garbage than before. ---
**Author:** ![edo](https://storj.bcdn.literatehosting.com/letter_avatar/edo/32/5_5575768a8748004e209b776fc1b2916d.png) [@edo](https://forum.storj.io/u/edo)\ **Post date:** [August 25, 2024, 6:57am UTC](https://forum.storj.io/t/when-will-uncollected-garbage-be-deleted/27327/314 "2024-08-25T06:57:21Z")
It did make a big jump, going from 60 GB to 400 GBā€”definitely progress! But when you stack that against the 2 TB of uncollected garbage, it still feels like thereā€™s a bit of catching up to do. Do we expect the upcoming BFs to be even more efficient at taking out the trash, or was this the expected big cleanup BF already? ---
**Author:** ![Alexey](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/alexey/32/41_2.png) [@Alexey](https://forum.storj.io/u/Alexey)\ **Post date:** [August 25, 2024, 7:14am UTC](https://forum.storj.io/t/when-will-uncollected-garbage-be-deleted/27327/315 "2024-08-25T07:14:23Z")
Itā€™s both. We implemented a change, which allows us to enable a feature flag to do not add an expired TTL data to the BF, so it would be moved to the trash on the nodes, and the BF should become more efficient (we always working on improvements in this area). The latter means that these sporadically performed audits from the trash doesnā€™t happen so often to assume the bug somewhere. However, I would prefer to be more careful there, as some SNOs empty the trash from time to time. ---
**Author:** ![edo](https://storj.bcdn.literatehosting.com/letter_avatar/edo/32/5_5575768a8748004e209b776fc1b2916d.png) [@edo](https://forum.storj.io/u/edo)\ **Post date:** [August 25, 2024, 7:33am UTC](https://forum.storj.io/t/when-will-uncollected-garbage-be-deleted/27327/316 "2024-08-25T07:33:35Z")
Just did a deep dive into the logs for this node, and a few things jumped out at me: 1. There are lots of retain errors popping up over the last few days. Plenty of warnings about files not being foundā€”sounds like a classic case of ā€˜ghost piecesā€™ hanging around. ```auto 2024-08-22T08:54:29Z WARN retain failed to trash piece {"Process": "storagenode", "cachePath": "config/retain", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Piece ID": "6YPK6D4E4ZBQIJXNP6CZBFISX7IRHBVJTGP43NBCBZXXRNHHMQPA", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist\n\tstorj.io/storj/storagenode/blobstore/filestore.(*blobStore).Stat:124\n\tstorj.io/storj/storagenode/pieces.(*BlobsUsageCache).pieceSizes:340\n\tstorj.io/storj/storagenode/pieces.(*BlobsUsageCache).TrashWithStorageFormat:406\n\tstorj.io/storj/storagenode/pieces.(*Store).Trash:422\n\tstorj.io/storj/storagenode/retain.(*Service).trash:428\n\tstorj.io/storj/storagenode/retain.(*Service).retainPieces.func1:387\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*TrashHandler).processTrashPiece:112\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*TrashHandler).writeLine:99\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*TrashHandler).Write:78\n\tio.copyBuffer:431\n\tio.Copy:388\n\tos.genericWriteTo:269\n\tos.(*File).WriteTo:247\n\tio.copyBuffer:411\n\tio.Copy:388\n\tos/exec.(*Cmd).writerDescriptor.func1:578\n\tos/exec.(*Cmd).Start.func2:728"} ``` 1. I also spotted something quirky: the ā€˜Failed to deleteā€™ messages are showing a negative number. Iā€™m guessing that means it tried to delete a piece that didnā€™t even exist? How could this be the case? ```auto 2024-08-25T00:52:44Z INFO retain Moved pieces to trash during retain {"Process": "storagenode", "cachePath": "config/retain", "Deleted pieces": 23678, "Failed to delete": -23678, "Pieces failed to read": 0, "Pieces count": 5633273, "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Duration": "17m24.819728138s", "Retain Status": "enabled"} ``` 1. I could **not** find a _ā€œMoved pieces to trash during retainā€_ log event for SLC yet, while the process started a while back ago. Does it means it is still running? Fingers crossed more data will be trashed! ```auto 2024-08-25T03:38:27Z INFO retain Prepared to run a Retain request. {"Process": "storagenode", "cachePath": "config/retain", "Created Before": "2024-08-19T17:23:28Z", "Filter Size": 9708635, "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE"} ``` ---
**Author:** ![Alexey](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/alexey/32/41_2.png) [@Alexey](https://forum.storj.io/u/Alexey)\ **Post date:** [August 25, 2024, 7:37am UTC](https://forum.storj.io/t/when-will-uncollected-garbage-be-deleted/27327/317 "2024-08-25T07:37:39Z")
> [@edo](#): > > `filestore error: file does not exist` This is mean, that the TTL collector was faster šŸ™‚ > [@edo](#): > > I also spotted something quirky: the ā€˜Failed to deleteā€™ messages are showing a negative number. Perhaps a bug. Itā€™s also usually comes with this WARN message: > [@Pieces failed to migrate v0 piece. Piece may not be recoverable](https://forum.storj.io/t/pieces-failed-to-migrate-v0-piece-piece-may-not-be-recoverable/27497): > > Iā€™m getting lots of these warnings on my only node that is on 1.110. Havenā€™t seen those before. Anything of concern? 2024-08-11T20:50:19Z WARN pieces failed to migrate v0 piece. Piece may not be recoverable {"Process": "storagenode"} 2024-08-11T20:50:19Z WARN retain failed to trash piece {"Process": "storagenode", "cachePath": "config/retain", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Piece ID": "74MBENAXPOZNXY2MNI37UFR2FBK3SCJ27SKVWCCFKKP7WMA2PHQQ", "error": "piecā€¦ > [@edo](#): > > Does it means it is still running? Fingers crossed more data will be trashed! It should, unless itā€™s failed somewhere. ---
**Author:** ![edo](https://storj.bcdn.literatehosting.com/letter_avatar/edo/32/5_5575768a8748004e209b776fc1b2916d.png) [@edo](https://forum.storj.io/u/edo)\ **Post date:** [August 25, 2024, 7:41am UTC](https://forum.storj.io/t/when-will-uncollected-garbage-be-deleted/27327/318 "2024-08-25T07:41:12Z")
Thanks Alexey for the information! > [@Alexey](#): > > It should, unless itā€™s failed somewhere. I did a quick search for ERROR in my logs and didnā€™t find anything GC-related, so Iā€™m cautiously optimistic that itā€™s still chugging along. šŸ¤ž ---
**Author:** ![Alexey](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/alexey/32/41_2.png) [@Alexey](https://forum.storj.io/u/Alexey)\ **Post date:** [August 25, 2024, 7:48am UTC](https://forum.storj.io/t/when-will-uncollected-garbage-be-deleted/27327/319 "2024-08-25T07:48:55Z")
You can be sure, if you would request `/mon/ps` on the debug port > [@Guide to debug my storage node, uplink, s3 gateway, satellite](https://forum.storj.io/t/guide-to-debug-my-storage-node-uplink-s3-gateway-satellite/1372): > > We have integrated a package called monkit in all storage nodes. It is very usefull for debuging. By default it will open a random port. In order to connect to the debug port you will need to: Add debug.addr: ":5999" to the config file. Now the port is fixed and will not change on every restart. Add -p 127.0.0.1:5999:5999 to the docker run command. Ready for action! You can request the following informations: curl localhost:5999/mon/funcs Find out how long a function like GetExpired needā€¦ [Previous page](https://forum.storj.io/t/when-will-uncollected-garbage-be-deleted/27327.md?page=14) [Next page](https://forum.storj.io/t/when-will-uncollected-garbage-be-deleted/27327.md?page=16)