# When will "Uncollected Garbage" be deleted?

**URL:** <https://forum.storj.io/t/when-will-uncollected-garbage-be-deleted/27327>\
**Category:** troubleshooting\
**Tags:** ttl, garbage-collector\
**Created:** [July 28, 2024, 8:32pm UTC](https://forum.storj.io/t/when-will-uncollected-garbage-be-deleted/27327 "2024-07-28T20:32:08Z")\
**Posts on this page:** 20\
**Page:** 23

<div class="post-metadata">

**Author:** ![donald.m.motsinger](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/donald.m.motsinger/32/71_2.png) [@donald.m.motsinger](https://forum.storj.io/u/donald.m.motsinger)\
**Post date:** [September 12, 2024, 9:52am UTC](https://forum.storj.io/t/when-will-uncollected-garbage-be-deleted/27327/464 "2024-09-12T09:52:39Z")

</div>

So it almost finished 14/32 of the xx folders if it was at folde nz this morning. Maybe better check with `lsof`.

---

<div class="post-metadata">

**Author:** ![jammerdan](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/jammerdan/32/1121_2.png) [@jammerdan](https://forum.storj.io/u/jammerdan)\
**Post date:** [September 12, 2024, 9:58am UTC](https://forum.storj.io/t/when-will-uncollected-garbage-be-deleted/27327/465 "2024-09-12T09:58:52Z")

</div>

> [@agente](#):
>
> ps: only ext4 nodes.

As far as I understand a potential reason why the way of deletion is so terribly slow on ext4 is that for every piece its metadata must be retrieved additionally to get its size so the node data can be updated with how much space has been freed.

---

<div class="post-metadata">

**Author:** ![Rassah](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/rassah/32/7828_2.png) [@Rassah](https://forum.storj.io/u/Rassah)\
**Post date:** [September 12, 2024, 11:21am UTC](https://forum.storj.io/t/when-will-uncollected-garbage-be-deleted/27327/466 "2024-09-12T11:21:36Z")

</div>

My /mnt/x/storagenode/storage/blobs/pmw6tvzmf2jv6giyybmmvl4o2ahqlaldsaeha4yx74n5aaaaaaaa folder is full of stuff. The du command would take forever, and I suspect will come back with about 3TB.

> This suggests that your node maybe disqualified or suspended on that satellite.

It says it’s not. My Suspension, Audit, and Online are all at 100%. There was no reason to suspend, since it was never online for any extended period of time

> Please also remove the data of untrusted satellites: [How To Forget Untrusted Satellites](https://forum.storj.io/t/how-to-forget-untrusted-satellites/23821)

Wouldn’t this block me from using saltlake at all?

---

<div class="post-metadata">

**Author:** ![donald.m.motsinger](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/donald.m.motsinger/32/71_2.png) [@donald.m.motsinger](https://forum.storj.io/u/donald.m.motsinger)\
**Post date:** [September 12, 2024, 11:25am UTC](https://forum.storj.io/t/when-will-uncollected-garbage-be-deleted/27327/467 "2024-09-12T11:25:58Z")

</div>

There is a faster way to get a rough estimate of the disk usage for 1 satellite. Get the size of on xx folder (or the average of a few of these folders) and multiply it by 1024.

---

<div class="post-metadata">

**Author:** ![donald.m.motsinger](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/donald.m.motsinger/32/71_2.png) [@donald.m.motsinger](https://forum.storj.io/u/donald.m.motsinger)\
**Post date:** [September 12, 2024, 11:32am UTC](https://forum.storj.io/t/when-will-uncollected-garbage-be-deleted/27327/469 "2024-09-12T11:32:58Z")

</div>

@Alexey please ban this scammer

---

<div class="post-metadata">

**Author:** ![Rassah](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/rassah/32/7828_2.png) [@Rassah](https://forum.storj.io/u/Rassah)\
**Post date:** [September 12, 2024, 12:09pm UTC](https://forum.storj.io/t/when-will-uncollected-garbage-be-deleted/27327/470 "2024-09-12T12:09:30Z")

</div>

3.0G on one of them, so times 1024 is 3.072G 🙂

---

<div class="post-metadata">

**Author:** ![MarviBiene](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/marvibiene/32/1124_2.png) [@MarviBiene](https://forum.storj.io/u/MarviBiene)\
**Post date:** [September 12, 2024, 12:28pm UTC](https://forum.storj.io/t/when-will-uncollected-garbage-be-deleted/27327/471 "2024-09-12T12:28:13Z")

</div>

Best is to report him as spam. Then a team member will deal with him

---

<div class="post-metadata">

**Author:** ![Aitor](https://storj.bcdn.literatehosting.com/letter_avatar/aitor/32/6_22734026db12803ebb3e059622ac3534.png) [@Aitor](https://forum.storj.io/u/Aitor)\
**Post date:** [September 13, 2024, 10:25am UTC](https://forum.storj.io/t/when-will-uncollected-garbage-be-deleted/27327/472 "2024-09-13T10:25:13Z")

</div>

Finally, I have back some free space. I didn’t do anything, just wait. Thank you for the fix.

![Captura de pantalla 2024-09-13 122408](https://storj-s3.bcdn.literatehosting.net/original/3X/5/9/599e8ff2710e7723e32db230619677830d2b9a01.png)

---

<div class="post-metadata">

**Author:** ![Vadim](https://storj.bcdn.literatehosting.com/letter_avatar/vadim/32/6_22734026db12803ebb3e059622ac3534.png) [@Vadim](https://forum.storj.io/u/Vadim)\
**Post date:** [September 13, 2024, 10:47am UTC](https://forum.storj.io/t/when-will-uncollected-garbage-be-deleted/27327/473 "2024-09-13T10:47:46Z")

</div>

Interesting how it can have free space and overused at same time?  
@Alexey any idea?

---

<div class="post-metadata">

**Author:** ![edo](https://storj.bcdn.literatehosting.com/letter_avatar/edo/32/6_22734026db12803ebb3e059622ac3534.png) [@edo](https://forum.storj.io/u/edo)\
**Post date:** [September 13, 2024, 11:24am UTC](https://forum.storj.io/t/when-will-uncollected-garbage-be-deleted/27327/474 "2024-09-13T11:24:59Z")

</div>

See:

> [@andrew2.hart](#):
>
> That is multinode dashboard, yes? If not, I don’t understand how you can have free and overused at the same time?

> [@Aitor](#):
>
> Yes, multinode. The overused appeared last night. I had free space all the time.

Aitor shows his multi node dashboard where one or more nodes are overused, while others have free space available.

---

<div class="post-metadata">

**Author:** ![Aitor](https://storj.bcdn.literatehosting.com/letter_avatar/aitor/32/6_22734026db12803ebb3e059622ac3534.png) [@Aitor](https://forum.storj.io/u/Aitor)\
**Post date:** [September 13, 2024, 12:29pm UTC](https://forum.storj.io/t/when-will-uncollected-garbage-be-deleted/27327/475 "2024-09-13T12:29:41Z")

</div>

Yes, multinode. I still don’t understand the overused because I read that after detect 5 GB more stops accepting data.

---

<div class="post-metadata">

**Author:** ![Alexey](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/alexey/32/41_2.png) [@Alexey](https://forum.storj.io/u/Alexey)\
**Post date:** [September 14, 2024, 4:49am UTC](https://forum.storj.io/t/when-will-uncollected-garbage-be-deleted/27327/476 "2024-09-14T04:49:08Z")

</div>

Easy. Some of the nodes has overused space, some - not. The summary for multinode dashboard just sum-ups all indicators.

> [@Aitor](#):
>
> I still don’t understand the overused because I read that after detect 5 GB more stops accepting data.

Yes. But overused meaning that the node used more space than allocated. When it has had the issue with the incorrectly calculated used space, it was not aware that it’s actually used all allocation. When the issue has been fixed (the used-space-wilewalker did its job), the node become aware about an overusage.

---

<div class="post-metadata">

**Author:** ![Rassah](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/rassah/32/7828_2.png) [@Rassah](https://forum.storj.io/u/Rassah)\
**Post date:** [September 14, 2024, 10:58am UTC](https://forum.storj.io/t/when-will-uncollected-garbage-be-deleted/27327/477 "2024-09-14T10:58:26Z")

</div>

> Please also remove the data of untrusted satellites: [How To Forget Untrusted Satellites](https://forum.storj.io/t/how-to-forget-untrusted-satellites/23821)

This node is too new and was started after those two satellites were shut down. I could use it to forget saltlake, since it’s not doing anything but taking up space, but I have about $4 held back by it. What do you suggest? I don’t know why saltlake is not seeing any of the 3TB of data that it’s storing.

---

<div class="post-metadata">

**Author:** ![pdeline06](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/pdeline06/32/1129_2.png) [@pdeline06](https://forum.storj.io/u/pdeline06)\
**Post date:** [September 14, 2024, 8:23pm UTC](https://forum.storj.io/t/when-will-uncollected-garbage-be-deleted/27327/478 "2024-09-14T20:23:34Z")

</div>

How you think it is necessary to keep the “storage2.piece-scan-on-startup” option “true” all the time? Or is it enough to run it occasionally to collect information on files and display information correctly on the dashboard, and mostly keep the option disabled “false”?

I don’t know if it makes sense to do a full scan of files after each node restart, even with the “badger cache” enabled. Although it takes less time, it still takes hours.

---

<div class="post-metadata">

**Author:** ![pasatmalo](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/pasatmalo/32/1127_2.png) [@pasatmalo](https://forum.storj.io/u/pasatmalo)\
**Post date:** [September 14, 2024, 10:48pm UTC](https://forum.storj.io/t/when-will-uncollected-garbage-be-deleted/27327/479 "2024-09-14T22:48:11Z")

</div>

If you face frequent restarts and the additional load caused by the scan is being problematic, it could be a good idea to disable the piece scan on startup feature.

The drawbacks from disabling start up piece scans is that the statistics on the dashboard may drift from the real values (Im not sure if there are still bugs remaining which cause this drift).

The main issue that can arise from the inaccurate stats, is that the node might believe its full when its not actually full (stopping ingress), or that its using more space than it should because it believes its using less space than it actually is.

If you dont mind that the dashboard shows possibly inaccurante values, and you keep an eye out for these statistics, then turning it off should be fine.

---

<div class="post-metadata">

**Author:** ![Alexey](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/alexey/32/41_2.png) [@Alexey](https://forum.storj.io/u/Alexey)\
**Post date:** [September 15, 2024, 2:20am UTC](https://forum.storj.io/t/when-will-uncollected-garbage-be-deleted/27327/480 "2024-09-15T02:20:17Z")

</div>

If you do not have data of old satellites, then you do not need to run the forget satellite command. You also do not need to forget the Saltlake satellite. If you want to leave this satellite, you may call a Graceful Exit on it: [Graceful Exit Guide (new procedure as of 2023-10-?)](https://forum.storj.io/t/graceful-exit-guide-new-procedure-as-of-2023-10/23882)

The garbage should be moved by the `retain` process to the trash and deleted 7 days later.  
When your node did receive a BF from the Saltlake satellite last time?  
I didn’t find records from it in your logs excerpt.

```auto
docker logs storagenode 2>&1 | grep "retain" | grep "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE" | tail

```

I have a suspicion: since the Saltlake satellite is sure that your node doesn’t have any data (it’s literally zero accordingly your graphs and the fact that other nodes in the network received at least one correct report from this satellite), and also amount of audits doesn’t change for days accordingly your logs, I can guess that the satellite do not issue the BF too, because it should contain pieces, which your node should keep. But since there shouldn’t be any pieces accordingly its databases, it doesn’t generate a BF…

In that case you need to receive something from that satellite, to allow it to generate a BF. But it cannot, because it’s full even with overusage… You need to have a free space to allow the node to receive something from the Saltlake satellite.

I can only offer to remove all files from the `temp` folder. If there is nothing, the next suggestion is to perform an extremely dangerous action - try to remove pieces from one of the subfolders in the blobs folder for that satellite (see [Satellite info (Address, ID, Blobs folder, Hex)](https://forum.storj.io/t/satellite-info-address-id-blobs-folder-hex/17183)) and restart the node to allow it to have a free space. Then maybe something will arrive from the Saltlake satellite and then wait when it would send the next BF.

---

<div class="post-metadata">

**Author:** ![Rassah](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/rassah/32/7828_2.png) [@Rassah](https://forum.storj.io/u/Rassah)\
**Post date:** [September 15, 2024, 8:59pm UTC](https://forum.storj.io/t/when-will-uncollected-garbage-be-deleted/27327/481 "2024-09-15T20:59:00Z")

</div>

> I can only offer to remove all files from the `temp` folder. If there is nothing

There was nothing…

> the next suggestion is to perform an extremely dangerous action - try to remove pieces from one of the subfolders in the blobs folder for that satellite (see [Satellite info (Address, ID, Blobs folder, Hex)](https://forum.storj.io/t/satellite-info-address-id-blobs-folder-hex/17183)) and restart the node to allow it to have a free space.

I was going to, but I’m going to try slightly different approach. It’s a 4TB drive (as in 3.6TB), so I increased my storage to 3.2TB to give it some space. Hopefully that will give it some space to write and check as you said it might. I had to reduce it to 3TB because when it was filling up space before, every time I restarted it, it kept claiming that it didn’t use up any space, and kept filling it up until the drive was full. This time I restarted and it’s reporting the amount I actually have. Hopefully this fixes it. Thanks

---

<div class="post-metadata">

**Author:** ![Alexey](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/alexey/32/41_2.png) [@Alexey](https://forum.storj.io/u/Alexey)\
**Post date:** [September 16, 2024, 4:01am UTC](https://forum.storj.io/t/when-will-uncollected-garbage-be-deleted/27327/482 "2024-09-16T04:01:31Z")

</div>

I submitted a bug for your case:

> <https://github.com/storj/storj/issues/7147>
>
> \<!--
> Please make sure that we do not have any duplicates already open. 
> You ca…n ensure this by searching the issue list for this repository. 
> If there is a duplicate, please close your issue and add a comment to the 
> existing issue instead.
> 
> For more information about reporting issues, see
> https://github.com/storj/storj/blob/main/docs/storagenode/CONTRIBUTING.md
> 
> \---------------------------------------------------
> GENERAL SUPPORT INFORMATION
> \---------------------------------------------------
> 
> The GitHub issue tracker is for bug reports and feature requests.
> General support can be found at the following locations:
> 
> \- Storj Community Forum - https://forum.storj.io
> \- File a ticket at https://support.storj.io/
> 
> \---------------------------------------------------
> BUG REPORT INFORMATION
> \---------------------------------------------------
> \--\>
> 
> \*\*Description\*\*
> We discovered an edge case when the node is full with overusage, has an uncollected garbage because of TTL issue #7025 and the satellite doesn't have any accounted data for the node, so the satellite will not generate a Bloom Filter for that node (it should include only pieces, which should remain on the node, but this node should not have pieces accordingly the satellite), so the uncollected garbage will remain forever. To trigger the clean process the node should have at least a one piece for that satellite.
> 
> See
> \* https://forum.storj.io/t/when-will-uncollected-garbage-be-deleted/27327/447?u=alexey
> \* https://forum.storj.io/t/when-will-uncollected-garbage-be-deleted/27327/451?u=alexey
> 
> \<!--
> Provide a more detailed introduction to the issue itself, and why you consider it to be a bug
> \--\>
> 
> \*\*Screenshots\*\*
> If applicable, add screenshots to help explain your problem.
> 
> \*\*Steps to reproduce the issue:\*\*
> 
> 
> 
> 1. Upload TTL data to the node, there shouldn't be any other data from that satellite.
> 2. Drop the TTL database on the node (to emulate a bug with missing TTL records)
> 3. Make the node full by reducing the allocation below a usage (to emulate the problem with the used space calculation resulted to an overusage)
> 4. Wait for expiration on the satellite, it should remove the expired records from its databases
> 5. Try to generate a Bloom Filter for this node.
> 6. The BF will not be generated, because this node should not have any pieces accordingly the satellite.
> 
> \*\*Describe the results you expected:\*\*
> The uncollected garbage should be collected for the case when the node should not have any pieces.
> 
> \*\*Describe the results you received:\*\*
> The uncollected garbage has not been collected, because there shouldn't be any pieces on that node and BF cannot be generated.
> 
> \*\*Your environment\*\*
> 
> \- Operating system and version:
> \- Additional environment details (Raspberry PI, Docker, VMWare, etc.): https://forum.storj.io/t/when-will-uncollected-garbage-be-deleted/27327/447?u=alexey

---

<div class="post-metadata">

**Author:** ![Rassah](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/rassah/32/7828_2.png) [@Rassah](https://forum.storj.io/u/Rassah)\
**Post date:** [September 16, 2024, 10:58am UTC](https://forum.storj.io/t/when-will-uncollected-garbage-be-deleted/27327/483 "2024-09-16T10:58:14Z")

</div>

Saltlake added (ingress) 21mb to my node (the remaining almost 200 gigs were added by other satellites). Let’s see if this will be enough to trigger it to work. Despite adding 21mb, my node is reporting that 42.16KB is now being used by saltlake.

---

<div class="post-metadata">

**Author:** ![Alexey](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/alexey/32/41_2.png) [@Alexey](https://forum.storj.io/u/Alexey)\
**Post date:** [September 17, 2024, 8:05am UTC](https://forum.storj.io/t/when-will-uncollected-garbage-be-deleted/27327/484 "2024-09-17T08:05:57Z")

</div>

Then it shall receive a BF soon. This should trigger a GC and retain.

[Previous page](https://forum.storj.io/t/when-will-uncollected-garbage-be-deleted/27327.md?page=22)

[Next page](https://forum.storj.io/t/when-will-uncollected-garbage-be-deleted/27327.md?page=24)
