# Current situation with garbage collection **URL:** \ **Category:** Node Operators\ **Tags:** garbage-collector, storagenode\ **Created:** [April 2, 2024, 4:47pm UTC](https://forum.storj.io/t/current-situation-with-garbage-collection/25711 "2024-04-02T16:47:47Z")\ **Posts on this page:** 20\ **Page:** 10
**Author:** ![Alexey](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/alexey/32/41_2.png) [@Alexey](https://forum.storj.io/u/Alexey)\ **Post date:** [April 14, 2024, 12:15pm UTC](https://forum.storj.io/t/current-situation-with-garbage-collection/25711/186 "2024-04-14T12:15:04Z")
I do not think so… But, yes, you can run it manually (not sure could it help?): > [@Manually start filewalker](https://forum.storj.io/t/manually-start-filewalker/25698/6): > > This is where the power of scripting comes in. For example (need to first uncomment the line in the config): #!/bin/bash CONTAINER\_NAME=storagenode CONFIG\_PATH=/mnt/storj/config.yaml docker container stop $CONTAINER\_NAME sed -i "s/^storage2.piece-scan-on-startup: .\*/storage2.piece-scan-on-startup: true/g" $CONFIG\_PATH docker container start $CONTAINER\_NAME sed -i "s/^storage2.piece-scan-on-startup: .\*/storage2.piece-scan-on-startup: false/g" $CONFIG\_PATH However I do agree with having more f… However, I’m not sure that it could help with a BF, because it’s sent from the satellites… and if your node version doesn’t support to store it locally… well, it will be forced to wait for the next one. ---
**Author:** ![snorkel](https://storj.bcdn.literatehosting.com/letter_avatar/snorkel/32/5_5575768a8748004e209b776fc1b2916d.png) [@snorkel](https://forum.storj.io/u/snorkel)\ **Post date:** [April 14, 2024, 12:37pm UTC](https://forum.storj.io/t/current-situation-with-garbage-collection/25711/187 "2024-04-14T12:37:42Z")
So, ipothetically, if a node is verry big and to slow to finish a GC, and the next BF comes in, the GC starts again and so no, and it never finishes. I guess this can be solved maybe if the GC does partial deletes and dosen’t wait to finish the run to move the garbage all at once to trash. This way, even if it dosen’t finishes, in time, there will still be pieces moved to trash, and only a small percentage will be kept from the ones that should be trashed. Maybe some dev can shed some light on this case. ---
**Author:** ![jammerdan](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/jammerdan/32/1121_2.png) [@jammerdan](https://forum.storj.io/u/jammerdan)\ **Post date:** [April 14, 2024, 12:50pm UTC](https://forum.storj.io/t/current-situation-with-garbage-collection/25711/188 "2024-04-14T12:50:40Z")
> [@snorkel](#): > > I guess this can be solved maybe if the GC does partial deletes and dosen’t wait to finish the run to move the garbage all at once to trash. I think this is what will happen: > [@Release preparation v1.100](https://forum.storj.io/t/release-preparation-v1-100/25578/1): > > [52881d2](https://github.com/storj/storj/commit/52881d2) storagenode/pieces: trash pieces immediately in GC Filewalker ---
**Author:** ![Alexey](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/alexey/32/41_2.png) [@Alexey](https://forum.storj.io/u/Alexey)\ **Post date:** [April 14, 2024, 12:51pm UTC](https://forum.storj.io/t/current-situation-with-garbage-collection/25711/189 "2024-04-14T12:51:17Z")
I would guess, but you have two options: 1. Wait until the proper GC will be implemented for a big nodes 2. Reduce the allowed amount to be used. I do not see any other solution so far (I do not have any node which can be exclusion so far). ---
**Author:** ![Alexey](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/alexey/32/41_2.png) [@Alexey](https://forum.storj.io/u/Alexey)\ **Post date:** [April 14, 2024, 12:55pm UTC](https://forum.storj.io/t/current-situation-with-garbage-collection/25711/190 "2024-04-14T12:55:43Z")
This change should improve things! You will have a one GC filewalker, which must move pieces to the trash instead of TWO: - gc-filewalker which consider what is could be moved to the trash; - AND the retention filewalker, which is ACTUALLY moved pieces to the trash. So, it must improve things ---
**Author:** ![jammerdan](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/jammerdan/32/1121_2.png) [@jammerdan](https://forum.storj.io/u/jammerdan)\ **Post date:** [April 15, 2024, 4:12am UTC](https://forum.storj.io/t/current-situation-with-garbage-collection/25711/191 "2024-04-15T04:12:17Z")
> [@jammerdan](#): > > Does this include a log information that these processes start resumed. That would be helpful to track that it is working. I can’t see something like that here: [storagenode/retain: add more logging to GC filewalker · storj/storj@8a7b305 · GitHub](https://github.com/storj/storj/commit/8a7b305) or here [storagenode/{pieces,blobstore}: save-state-resume feature for GC file… · storj/storj@0f90f06 · GitHub](https://github.com/storj/storj/commit/0f90f06) or here [storagenode/pieces/lazyfilewalker: more logging to GC filewalker · storj/storj@780df77 · GitHub](https://github.com/storj/storj/commit/780df77) To distinguish a resume from a fresh start and to monitor resume is working as expected a log entry would be good. ---
**Author:** ![flwstern](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/flwstern/32/7821_2.png) [@flwstern](https://forum.storj.io/u/flwstern)\ **Post date:** [April 15, 2024, 6:18am UTC](https://forum.storj.io/t/current-situation-with-garbage-collection/25711/192 "2024-04-15T06:18:54Z")
Im seeing the exact same numbers on my big nodes i probably have over 100tb unpaid data accross all nodes, not impressed at all by storj calculations. Its a joke at the current state for the operators. ---
**Author:** ![jammerdan](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/jammerdan/32/1121_2.png) [@jammerdan](https://forum.storj.io/u/jammerdan)\ **Post date:** [April 15, 2024, 6:40am UTC](https://forum.storj.io/t/current-situation-with-garbage-collection/25711/193 "2024-04-15T06:40:31Z")
> [@flwstern](#): > > Its a joke at the current state for the operators. +1 (Post must be at least 20 characters) ---
**Author:** ![nerdatwork](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/nerdatwork/32/1135_2.png) [@nerdatwork](https://forum.storj.io/u/nerdatwork)\ **Post date:** [April 15, 2024, 7:10am UTC](https://forum.storj.io/t/current-situation-with-garbage-collection/25711/194 "2024-04-15T07:10:58Z")
Shouldn’t gc’s failure be logged in INFO level than DEBUG level ? > > > Change-Id: I55e855258eb2eaf9f4ee68d478b58a8b550a9294 ```auto log.Debug("gc-filewalker failed", zap.Error(err)) ``` ---
**Author:** ![striker43](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/striker43/32/1135_2.png) [@striker43](https://forum.storj.io/u/striker43)\ **Post date:** [April 15, 2024, 7:55am UTC](https://forum.storj.io/t/current-situation-with-garbage-collection/25711/195 "2024-04-15T07:55:02Z")
> [@nerdatwork](#): > > Shouldn’t gc’s failure be logged in INFO level than DEBUG level ? I would log it as warning or error because if it always fails, it will affect the functionality of the node… ---
**Author:** ![elek](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/elek/32/9189_2.png) [@elek](https://forum.storj.io/u/elek)\ **Post date:** [April 15, 2024, 11:25am UTC](https://forum.storj.io/t/current-situation-with-garbage-collection/25711/196 "2024-04-15T11:25:45Z")
> [@BrightSilence](#): > > It looks like the GC runs are finished on my nodes, but I’m not impressed with the results unfortunately. […] but it seems it only cleaned up less than 25% of garbage. That’s a lot worse than I was expecting (or maybe hoping). > > […]is this to be expected when hitting the max BF size? Can you please share the number of pieces files (or used `blobs` space) before and after? Looks like you have 11.13 TB blobs usage after GC run, which deleted `2890475` of files. Adn all together you have 39M With 39M pieces, the false positive rate is 44% (66 % o the trash couldn’t be found). But we use different seeds for different BF, so multiple BF runs will clean it up. > [@BrightSilence](#): > > …If so, I think more urgency is required to implement the new way of sending larger BF’s. Agree. We are following the progress of the rollout. Big nodes are usually up-to-date, so probably we can start using new filters soon… ---
**Author:** ![BrightSilence](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/brightsilence/32/6048_2.png) [@BrightSilence](https://forum.storj.io/u/BrightSilence)\ **Post date:** [April 15, 2024, 12:05pm UTC](https://forum.storj.io/t/current-situation-with-garbage-collection/25711/197 "2024-04-15T12:05:25Z")
> [@elek](#): > > With 39M pieces, the false positive rate is 44% (66 % o the trash couldn’t be found). Those two numbers don’t add up to 100%. I’m guessing it should be 34% removed vs 66% that couldn’t be cleaned up. My quick and dirty calculation looked a little worse, but to be fair, I did assume all garbage was from US1 in that calculation. Still even if 44% gets cleaned up, that’s not exactly a matter of “multiple runs will get it”. With that percentage, there would always be a significant amount of garbage. I’ll keep an eye out for the updates. Hopefully you decide to send out the larger BF’s after the rollout is complete. ---
**Author:** ![Alexey](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/alexey/32/41_2.png) [@Alexey](https://forum.storj.io/u/Alexey)\ **Post date:** [April 16, 2024, 8:33am UTC](https://forum.storj.io/t/current-situation-with-garbage-collection/25711/198 "2024-04-16T08:33:12Z")
Yes, and we hope to solve it soon. ---
**Author:** ![BrightSilence](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/brightsilence/32/6048_2.png) [@BrightSilence](https://forum.storj.io/u/BrightSilence)\ **Post date:** [April 16, 2024, 9:54am UTC](https://forum.storj.io/t/current-situation-with-garbage-collection/25711/199 "2024-04-16T09:54:20Z")
I understand and I’m not complaining. I was just responding to this line. > [@elek](#): > > But we use different seeds for different BF, so multiple BF runs will clean it up. I should probably have quoted it in my previous message. But I don’t agree with that assesment. Multiple runs won’t be enough to clean it up, especially when new deletes keep coming in. But I think you are all aware of that and working on a better solution, so the point might be moot. ---
**Author:** ![elek](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/elek/32/9189_2.png) [@elek](https://forum.storj.io/u/elek)\ **Post date:** [April 16, 2024, 10:38am UTC](https://forum.storj.io/t/current-situation-with-garbage-collection/25711/200 "2024-04-16T10:38:34Z")
> [@BrightSilence](#): > > I should probably have quoted it in my previous message. But I don’t agree with that assesment. Multiple runs won’t be enough to clean it up, especially when new deletes keep coming in. Well, it depends on the number of new deletions. Because I love the technical details, let me add some more information 😉 Here is a simple simulation with spreadsheet: ![image](https://storj-s3.bcdn.literatehosting.net/original/2X/f/f622f165ee0d443f30c70481cece4bb3e6425796.png) Let’s say you have 25M real blobs, and 14M blobs which should be deleted (which is very rare, and bad situation). With 39M (25+14) blobs, the false positive rate of a 4100003 bytes BF is 66%. The 25M will always kept (guaranteed by the BF), but only the 34% of the 14M will be properly deleted. So we will have 24M + 14M \* 0.34 + newly deleted blobs for the next BF. Here the code has a nice trick. Next time a different seed is used for BF, so the pieces which are affected by false-postitives hits, will be different. There is good chance that some of the missed pieces will be deleted this time. As you see in the garbage column, the original 14M is slowly decreasing (it requires 10 BF execution to get only 269k as long as only 100k pieces are deleted during one BF period). And yes, it depends on the number of new deletions, but even with 1M new deletions, it will work eventually… That’s the math. We have hope. But there is no question: we should use the new protocol ASAP, as 66% false positive rate is unacceptable (even if it’s 0.66 \* 0.66 at the second time). Note: false-postitive rate is calculated with the formula `=POW(2,4100003/sum(E2:F2)/-1.44*8)` ---
**Author:** ![BrightSilence](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/brightsilence/32/6048_2.png) [@BrightSilence](https://forum.storj.io/u/BrightSilence)\ **Post date:** [April 16, 2024, 11:18am UTC](https://forum.storj.io/t/current-situation-with-garbage-collection/25711/202 "2024-04-16T11:18:44Z")
I didn’t mean to put you up to so much work with my comment. 😆 Of course I agree with all that and you’re right. I am aware of how the system works including the random seed. But before I could respond there is already a comment from someone else showing pretty clearly that that isn’t good enough. Time matters and deletions of free account data are still planned as well, so more large deletions are expected. I for one don’t need to be convinced that you are working hard on better solutions either and I don’t discount the value of this stop gap solution. Anything that frees up some space will help as it gives full nodes new room to grow. That helps a lot as a temporary solution. But it doesn’t stop me from looking forward to the full solution soon. Anyway, keep up the good work and sorry if I took up your valuable time by sparking that calculation. 😉 I personally don’t appreciate when someone brings an argument without calculations and expects detailed numbers in response. So that wasn’t my intention, but your efforts are appreciated anyway. > [@elek](#): > > Note: false-postitive rate is calculated with the formula `=POW(2,4100003/sum(E2:F2)/-1.44*8)` I do love having this little nugget of knowledge though. 😁 ---
**Author:** ![snorkel](https://storj.bcdn.literatehosting.com/letter_avatar/snorkel/32/5_5575768a8748004e209b776fc1b2916d.png) [@snorkel](https://forum.storj.io/u/snorkel)\ **Post date:** [April 16, 2024, 11:57am UTC](https://forum.storj.io/t/current-situation-with-garbage-collection/25711/203 "2024-04-16T11:57:24Z")
I don’t know if GC run time is an issue for now, I know only about FW. But, I will throw this ideea to you anyway… What if we store pieces in daily directories? It will make GC scan only those directories that corespond to date of creation for the pieces that must be kept/ deleted… Because we know that old data remains untouched for long time, and the majority of deletes are from the recent times. Why scan old untouched directories? The pieces that must be deleted will be concentrated in a few recent directories. Directory structure variants: 1. /data/US1/2024/12/31/… pieces 2. /data/US1/2024-12-31/… pieces The UTC can be used to choose the right time and GC can check the actual folder and the next, for pieces that slipped in the next day. ---
**Author:** ![mrkeyboardcommando](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/mrkeyboardcommando/32/71_2.png) [@mrkeyboardcommando](https://forum.storj.io/u/mrkeyboardcommando)\ **Post date:** [April 16, 2024, 2:58pm UTC](https://forum.storj.io/t/current-situation-with-garbage-collection/25711/204 "2024-04-16T14:58:49Z")
wouldn’t you run into filesystem issues with loads of folders? ---
**Author:** ![Roxor](https://storj.bcdn.literatehosting.com/letter_avatar/roxor/32/5_5575768a8748004e209b776fc1b2916d.png) [@Roxor](https://forum.storj.io/u/Roxor)\ **Post date:** [April 16, 2024, 3:05pm UTC](https://forum.storj.io/t/current-situation-with-garbage-collection/25711/205 "2024-04-16T15:05:34Z")
> [@snorkel](#): > > What if we store pieces in daily directories? Creation dates are going to be all over the place, and have almost zero relation to what customers choose to delete at any given time. And… any information about dates is space that would get taken up instead of the normal BF details (as we’re already bumping up against max BF sizes). So you’d reduce the precision of BF data (by having less room to include it)… and instead you’d have more date data: which is going to be near-random anyways? ---
**Author:** ![snorkel](https://storj.bcdn.literatehosting.com/letter_avatar/snorkel/32/5_5575768a8748004e209b776fc1b2916d.png) [@snorkel](https://forum.storj.io/u/snorkel)\ **Post date:** [April 16, 2024, 3:54pm UTC](https://forum.storj.io/t/current-situation-with-garbage-collection/25711/206 "2024-04-16T15:54:42Z")
So I get that it’s a bad ideea. 😅 [Previous page](https://forum.storj.io/t/current-situation-with-garbage-collection/25711.md?page=9) [Next page](https://forum.storj.io/t/current-situation-with-garbage-collection/25711.md?page=11)