When will "Uncollected Garbage" be deleted?

[jammerdan]

@Roxor

I don’t mind losing. But it would suck to use the disk space anyways…

It is extremely annoying this way because it gives a false impression too.

The node dashboard would show the space a occupied. SNO believes used data is accumulating while in reality he is storing garbage.
Plus this garbage gets removed only by BF, which means only after a BF it will get moved to the trash where it lies around for another while before finally removed.
So it could be weeks after a cancelled upload that the corresponding piece gets removed from the drive.

So maybe what could be done is to rename a cancelled uploaded file. Maybe to a different extension. Trash collector could recognize this extension and remove the piece whenever it runs.
And used space file walker could ignore the files with different extension or account them separately. Then SNO could see what is “used” but also how much trash from cancelled uploads is there.

[Alexey]

@jammerdan

So maybe what could be done is to rename a cancelled uploaded file. Maybe to a different extension.

it should be removed if the transfer is canceled. However, if the cancelation happen right after the piece upload is finished (just the node was not fast to report back), the piece will remain until the next gc.
If the canceled upload is a not removed piece, then perhaps the node were restarted in that moment.
So renaming wouldn’t solve anything.
Or do you mean when the retain is moving pieces to the trash? And how it would detect that this piece is partially uploaded?

[jammerdan]

@Alexey

it should be removed if the transfer is canceled.

According to this issue, they are not removed immediately:

Cancelled upload files aren't immediately deleted

Cancelled upload files aren't immediately deleted · Issue #7058 · storj/storj · GitHub

I looked at this issue and as you noticed there is a case where piece will stay on disk when upload was canceled (log entry upload canceled (race lost or node shutdown) ).

Also this: Cancelled upload files aren't immediately deleted · Issue #7058 · storj/storj · GitHub

So when the node can produce a log entry that a file has been cancelled it should be able to rename the file. Instead it will get stored as regular .sj1 file. So to every process it looks like a regular stored piece.

So I don’t see how currently any other process than retain after a BF will remove that file and it will be removed by going through the normal trash deletion workflow as far as I understand.

Instead, if we know the file has been cancelled and rename it to let’s say .sjc extension any process could recognize that this is a file that should not be there. And the collector filewalker (the one that runs frequently to remove expired pieces) could be made aware to remove files with extension .sjc as well.

Of course it is true what is written in the Github issue that it would be better if these cancelled uploads would not touch the disk in the first place. This should remain the long term goal. But for more short term solution, maybe my idea could work.

[Alexey]

@jammerdan

it should be able to rename the file.

Then it can remove it instead as well and you can read the reason in the linked issue, why it was not implemented like this - the canceled upload may happen very often when the node is overloaded by IO and you suggesting to add one more IO operation. If we going to add an operation, then it should be a deletion.

[jammerdan]

Yes, I don’t know what file operations cost the least IO.
Maybe renaming is less heavier than deletion and then the collector comes later, configurable for finally removing it. I mean it is clear that “something” will come later to remove it no matter what.

But I also don’t know at which point the node knows that the race is lost. So maybe it could not store the file as .sj1 but as .sjc immediately. Then there would be no renaming required at all. But I don’t know if the node knows that the upload has been cancelled before it has written the file to disk.

Edit:

@Alexey

you suggesting to add one more IO operation.

I don’t know. At least currently to me it seems we

1. Write the cancelled piece.
2. Garbage collect the cancelled piece
3. Retain the cancelled piece
4. Remove the cancelled piece from trash.

While all the time the SNO does not get paid for it, could be weeks.

My idea:

1. Write the cancelled piece as .sjc (if possible)
2. (Rename the cancelled piece to .sjc only if writing it immediately like that is not possible)
3. Collect the cancelled piece through expiry collector

Big plus: The expiry collector has already a configurable interval. So a SNO could run it whenever he wants like only every 12 hours or something.

2nd edit:
I see I made a mistake here: The expiry collector does not walk the files but get them from the database, right? So in my idea the deletion would need to be part of some other process that walks the files to recognize the different extensions.

[Mitsos]

@Alexey: please stop moving my replies into other random topics. The reply was for capacity planning based on the events that are happening in the past couple of months. Just because it mentions uncollected garbage, it does not mean that it belongs in a topic about it.

[Alexey]

@jammerdan

But I also don’t know at which point the node knows that the race is lost.

It may never know this, it may end as a timeout (context canceled). But as showed in the issue, we already have a different behavior for known cancelation (canceled) and unknown (failed): the former doesn’t delete the uploaded piece, the latter removes it (or does not save?).

@jammerdan

The expiry collector does not walk the files but get them from the database, right?

Yes, you are correct, so it cannot be used to remove these pieces.
It could be a retain process, but it should not move the partially uploaded piece to the trash and remove it instead (this of course would slow down it), or the trash filewalker, but the trash filewalker is going only over expired subfolders, so it will remove this piece anyway.

[Alexey]

No, I cannot. It’s still offtopick. The whole side discussion about full/not full has been moved here. You explained, that you have a free space still and do not want to expand, this is enough.
This one is reply to Roxor’s post, which is offtopick there too.

[MarviBiene]

Hello, I found today following error message in my logs:

marvi@debian:/storagelogs/node1$ grep "retain" node.log
2024-08-03T02:54:54Z    INFO    retain  Prepared to run a Retain request.                                              {"Process": "storagenode", "cachePath": "config/retain", "Created Before": "2024-07-28T17:59:59Z", "Filter Size": 22243799, "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE"}
2024-08-03T03:38:45Z    ERROR   pieces  lazyfilewalker failed   {"Process": "storagenode", "error": "lazyfilewalker: signal: killed", "errorVerbose": "lazyfilewalker: signal: killed\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*process).run:85\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*Supervisor).WalkSatellitePiecesToTrash:160\n\tstorj.io/storj/storagenode/pieces.(*Store).WalkSatellitePiecesToTrash:562\n\tstorj.io/storj/storagenode/retain.(*Service).retainPieces:379\n\tstorj.io/storj/storagenode/retain.(*Service).Run.func2:265\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78"}
2024-08-03T03:38:45Z    ERROR   retain  retain pieces failed    {"Process": "storagenode", "cachePath": "config/retain", "error": "retain: filewalker: context canceled", "errorVerbose": "retain: filewalker: context canceled\n\tstorj.io/storj/storagenode/pieces.(*FileWalker).WalkSatellitePieces:74\n\tstorj.io/storj/storagenode/pieces.(*FileWalker).WalkSatellitePiecesToTrash:181\n\tstorj.io/storj/storagenode/pieces.(*Store).WalkSatellitePiecesToTrash:569\n\tstorj.io/storj/storagenode/retain.(*Service).retainPieces:379\n\tstorj.io/storj/storagenode/retain.(*Service).Run.func2:265\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78"}
2024-08-03T03:38:52Z    INFO    retain  Prepared to run a Retain request.                                              {"Process": "storagenode", "cachePath": "config/retain", "Created Before": "2024-07-28T17:59:59Z", "Filter Size": 22243799, "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE"}
2024-08-03T03:38:57Z    ERROR   pieces  lazyfilewalker failed   {"Process": "storagenode", "error": "lazyfilewalker: signal: killed", "errorVerbose": "lazyfilewalker: signal: killed\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*process).run:85\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*Supervisor).WalkSatellitePiecesToTrash:160\n\tstorj.io/storj/storagenode/pieces.(*Store).WalkSatellitePiecesToTrash:565\n\tstorj.io/storj/storagenode/retain.(*Service).retainPieces:379\n\tstorj.io/storj/storagenode/retain.(*Service).Run.func2:265\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78"}
2024-08-03T03:38:57Z    ERROR   retain  retain pieces failed    {"Process": "storagenode", "cachePath": "config/retain", "error": "retain: filewalker: context canceled", "errorVerbose": "retain: filewalker: context canceled\n\tstorj.io/storj/storagenode/pieces.(*FileWalker).WalkSatellitePieces:74\n\tstorj.io/storj/storagenode/pieces.(*FileWalker).WalkSatellitePiecesToTrash:181\n\tstorj.io/storj/storagenode/pieces.(*Store).WalkSatellitePiecesToTrash:572\n\tstorj.io/storj/storagenode/retain.(*Service).retainPieces:379\n\tstorj.io/storj/storagenode/retain.(*Service).Run.func2:265\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78"}
2024-08-03T03:39:19Z    INFO    retain  Prepared to run a Retain request.                                              {"Process": "storagenode", "cachePath": "config/retain", "Created Before": "2024-07-28T17:59:59Z", "Filter Size": 22243799, "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE"}
2024-08-03T22:09:49Z    INFO    retain  Moved pieces to trash during retain                                            {"Process": "storagenode", "cachePath": "config/retain", "Deleted pieces": 3516255, "Failed to delete": 0, "Pieces failed to read": 0, "Pieces count": 32102961, "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Duration": "18h30m30.583054324s", "Retain Status": "enabled"}
2024-08-04T04:25:15Z    INFO    retain  Prepared to run a Retain request.                                              {"Process": "storagenode", "cachePath": "config/retain", "Created Before": "2024-07-30T17:59:59Z", "Filter Size": 963532, "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs"}
2024-08-04T05:16:32Z    INFO    retain  Moved pieces to trash during retain                                            {"Process": "storagenode", "cachePath": "config/retain", "Deleted pieces": 96939, "Failed to delete": 0, "Pieces failed to read": 0, "Pieces count": 1785214, "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Duration": "51m16.453344018s", "Retain Status": "enabled"}

I think that is the reason saltlake never collects trash in my node. But it’s the first time I saw this error

How can I figure out why the node got killed by os? I have sufficient CPU and Ram. I have like 8 cores and 8Gb of ram for 3 Nodes at the moment.

[Alexey]

Usually this a result of the pressure. And usually this is RAM. Please check you system journals to figure out the reason, why is it killed.
As a workaround you may disable the lazy mode.

[ACarneiro]

@Alexey

Only payout is always correct

Well, without any reliable information that I can use to cross-check what I was paid with what I was expecting to be paid, I have to take this on faith. I have no way of knowing.
It is not ideal.

[Luis]

I have 10 nodes at the moment. Last month, my average used space was 56 TB and I received 100 dollars. The space used has been decreasing every month. Even with all the problems, my average online in the last month was 98.9% Effectively, if the problems are not resolved, I will continue to turn off my machines. There were 5, now there are 4.

[donald.m.motsinger]

@Alexey

I never said that the reports from the satellites are always correct. Only payout is always correct because of reason, which you quoted.

I’m speechless. How is it even possible that the satellite’s reports are not always correct? I can understand that it might not finish in time and doesn’t report for one day. But wrong reports? And then you say the payout is always correct? How can we trust this? If the payout is always correct, why is this data not also used for the daily reports?

So there is no way of locally verifiying how much paid data I have? I guess the used-space filwalker just counts all data (paid and garbage) in the blobs folder.

@Alexey

By the way, I have a suspicion, that the report from SL on Aug 1 is incorrect. Because the blob for it on my nodes is much greater than it’s reported. And the almost daily BF didn’t clean too much from it.

…unless the BF is buggy too. How can we tell? At the moment I cannot trust anything from STORJ in terms of reporting and GC.

[donald.m.motsinger]

…and July’s payment is in. I have had >30TB (31.77TB on average) every single day in July according to the dashboard, so I would have expected >$47 from storage alone. Yet the payment was less than $35 including egress.

So please everyone, stop denying that there is a problem with garbage collection.

[Balage76]

On july 1st 1 storj was equal with 0,44 USD, today it is 0,32 USD.
I believe we all will suffer the same loss in payments.

[donald.m.motsinger]

Conversion is done at the moment of payment.

[Ruskiem]

iiiii just got ~the same STORJ amount as month ago,
and it happend to be even the same price in USD as then (at $0,33)
for 11 nodes, same TB allocated, same setup.
As long its not less im good.
Waiting for improvements for it to be more,
I expect to see more not sooner than in Octobers payment.
because things takes time on such scale to arrive, and effects of today’s written code to be visible to us SNOs.

[edo]

@donald.m.motsinger

July’s payment is in. I have had >30TB (31.77TB on average) every single day in July according to the dashboard, so I would have expected >$47 from storage alone. Yet the payment was less than $35 including egress.

So please everyone, stop denying that there is a problem with garbage collection.

As we can’t rely on daily satellite reports but can on the payment report, and since the payments are now in, I agree with your observations. I can confirm that I also didn’t receive as much as I expected for having full nodes practically the whole month of July. This confirms that the satellites believe we have much less data stored than what is actually present on our nodes. Yet, garbage collection doesn’t seem to be successfully executed.

A few days have passed since I last reported, and my nodes are still full, preventing me from accumulating more (paid) data. I do receive BFs, but they are only marking very little data as trash. It’s becoming increasingly clear that there is a significant issue that needs addressing. I believe it’s crucial that we look deeper into what’s going wrong and find a solution.

@elek

If you think that BF doesn’t delete enough data, please fill this form:

Reporting ineffective BF runs...

@elek, are there any (preliminary) results from the form submissions regarding this issue?

[HGPlays]

I Can also confirm that The used space my nodes think that they have, and the expected payment (from my calculations) are not aligned with the actual payment I have just gotten. Either I have a shit ton (tens of TB) of uncollected garbage or the payments are wrong (I hope not and doubt). I had between 180tb to 220tb of used data and was payed over 100 dollars less then expected based on the used data.

And that is just counting used space on nodes. Not even egress payments.

I believe enough people have already risen their voice and claimed to have issues. And it might be time for storj to at least acknowledge some issue is a hand. Other than the usual answer: it’s your problem, your hardware, not supposed to count on satellites data, run fw, do this and do that.
We have many experts here that knows how to run the fw and get real data of their disk, and can now compare to payouts and conclude that there IS a problem.

Also - i also have a continuous stream of 30-40TB of “tracked” garbage at any moment the past 2-3 months. I cannot agree that all this used space should be unpaid. How can that be fair, that for your (storjs) safety you need to keep deleted data for 2 weeks (often more) yet on the expense of MY running disks, my power, and my loss of income because it is eating away space that could be payed. We should at least be heard as SNO’s that some don’t wanna store unpaid “tracked” garbage for multiple weeks. It’s not our problem, yet it’s our responsibility. How so?
I have been patient for months and gave the benefit of the doubt that garbage whould clear and be reduced. Yet over 15-20% of my allocated used space goes unpaid because storj needs safety. If storj needs safety they must pay. Not me.

Sorry. Rant over for now (first in months so I feel it’s justified.)

[Aitor]

Only payout is always correct… wow, I have no words for this. ALWAYS? That’s a lot of time.