When I check the log for “trash” on more than one node, the last entry I get is from 24th resp. 25th.
Now I am wondering if something is wrong.
One node has accumulated more than 600GB trash. Yet the last time I see “emptying trash started” in the logs is from the 24th.
Isn’t this expected to run at least once a day?
As I remember it run once or twice a week
But if files in trash are supposed to stay there for 7 days only then it should not run only once a week I think. It should run once a day.
1 Like
The piece:trash “filewalker” is performing every 24h since start of the node.
yes, it is.
Do you have logs?
That’s why I wondering if everything is running like I should.
I would expect to see a line like emptying trash started every day in in the logs for each satellite.
However when I grep for “trash”, the last log line that contains the word “trash” is dated 25th of February:
2024-02-25T17:18:40Z INFO retain Moved pieces to trash during retain {"process": "storagenode", "Deleted pieces": 1403480, "Failed to delete": 100, "Pieces failed to read": 0, "Pieces count": 13374633, "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Duration": "57h33m9.51829974s", "Retain Status": "enabled"}
And this "Duration": "57h33m9.51829974s does it mean that it was running 57 hours moving pieces around?
It seems emptying trash has resumed today.
The last lines before that were from 24th:
2024-02-24T04:05:52Z INFO pieces:trash emptying trash started {"process": "storagenode", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs"}
Now I have new lines for all satellites starting on 28th:
2024-02-28T10:51:16Z INFO pieces:trash emptying trash started {"process": "storagenode", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs"}
I can also confirm that after I have seen these lines I checked the trash amount and it was lower than before. So some emptying has been done.
But does that mean trash has not been emptied for 3 days? If so why is this?
My understanding would be that this is a chore that shall be running every day.
I have also checked if I may have files that are too old in the trash. It is my understanding that So I ran a check for files older than 7 days and was surprised that it gave me 246GB of files older than 7 days. Again my believe was, files older than 7 days should not exist in trash.
1 Like
seems so.
yes, the trash is checked every 24h and this is hardcoded value.
You may check that you do not have pieces there older than 7 days:
I do not have any idea, why you might have had them, maybe your node is restarted? It’s just run every 24h, it’s not tied to any date.
But it did not run 3 days in a row and I did not do any manual restarts during that.
I have been running the command with mtime +7. This should count everything older than 7 days right? That returned 246GB of data in trash, older than 7 days.
The find/du command always takes ages…
Now it has finished. I have been running it with mtime +8 setting and it returned 143GB.
So something does not work right with the trash or I fail to understand how it should work.
My understanding is that there should be no files older than 7 day in the trash which is obviously not the case here.
no, it will account everything older than 6 days
these are pieces older than 7 days. So, it should delete them in the next start of piece:trash.
Could you please search in your logs "Node " (please note the space after “Node”), it should show lines when node is started. I suspect that it has restarts between piece:trash runs in these 3 days.
Ah I see, my bad then.
Grepping for “Node " (with space”) returns nothing. Not a single line.
Then it’s interesting. I do not know, what can prevent from running a hardcoded chore.
Maybe clocks skew? It’s checked only on start, but… this is just an internal scheduler.
Yes very weird.
I have no idea.
Last actual line with “emptying trash started” is:
2024-02-28T23:03:02Z INFO pieces:trash emptying trash started {"process": "storagenode", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S"}
No line on the 29t and none on the 1st so far.
Ran it again with mtime +8 and this time it returned 52 GB.
Less than before but still over the 7 days limit.
I saw few times that “pieces are deleted from trash after 7 days”
Well, one of my nodes has over 900GB of trash at least 3 weeks now. Nothing gets deleted.
And as all SNOs saw in last few days trash size increased even more.
Image shows 0.91TB after restart, somehow that 1.3TB from before restart didn’t even saved into db. Following this road, I’ll have node with 0 free space and full of (unpaid) trash.
Any advices how to resolve this issue by not deleting entire trash folder?
1 Like
You’re fortunate, because my situation is quite different: I’m dealing with approximately 30TB of data that hasn’t been paid for, and there’s zero data in the trash.
Most likely deleted files get replaced with new files in the trash. You can check for files in the trash folder older than 7 days. It should return 0.
Are you sure?
Please provide results of the scripts