Something wrong with trash?

[Dunc4n1d4h0]

Hi,
it toook very long time to run and results are very surprising to me… no old files in trash folders, but trash size in node GUI shows still same constant 0.9 TB all time, like input and output rate were constant thru weeks which is very unlikely.
(I chose this specific date because “problems” started about 2-3 weeks earlier)

find pmw6tvzmf2jv6giyybmmvl4o2ahqlaldsaeha4yx74n5aaaaaaaa/ -type f -not -newermt 2024-03-18 | wc -l
0
find pmw6tvzmf2jv6giyybmmvl4o2ahqlaldsaeha4yx74n5aaaaaaaa/ -type f -not -newermt 2024-03-18 | wc -l
0
find qstuylguhrn2ozjv4h2c6xpxykd622gtgurhql2k7k75wqaaaaaa/ -type f -not -newermt 2024-03-18 | wc -l
0
find ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/ -type f -not -newermt 2024-03-18 | wc -l
0
find v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa/ -type f -not -newermt 2024-03-18 | wc -l
0

edit:
I’m starting to think that the possible reason is that file-walkers processes did not have time to finish because an update for a new version of node was released

[Alexey]

@Dunc4n1d4h0

no old files in trash folders, but trash size in node GUI shows still same constant 0.9 TB

This is only mean that you DBs are not updated. You need to figure out - why.
Do you have database locks or a malformed/destroyed databases? (database is locked, file is not a database, database is malformed, i/o error, etc.)?
Do you have FATAL errors? Or maybe restarts in the middle of filewalkers?
Do you have all filewalkers successfully finished for all satellites?

Filewalker status

Logs should be at least info level Used space filewalker docker logs storagenode -f 2>&1 | grep "used-space-filewalker" | grep -E "started|finished" GC filewalker docker logs storagenode -f 2>&1 | grep "gc-filewalker" | grep -E "started|finished" retain docker logs storagenode -f 2>&1 | grep "retain" collector docker logs storagenode -f 2>&1 | grep "collector" trash docker logs storagenode -f 2>&1 | grep "piece:trash"

[Dunc4n1d4h0]

@Alexey

This is only mean that you DBs are not updated. You need to figure out - why.
Do you have database locks or a malformed/destroyed databases? (database is locked, file is not a database, database is malformed, i/o error, etc.)?
Do you have FATAL errors? Or maybe restarts in the middle of filewalkers?
Do you have all filewalkers successfully finished for all satellites?

Thanks, I checked for db problems, all is okay there.
News are, that after file-walker process itself finally finished after working for a week - I monitored that every day, trash size started to shrink slowly, now it’s ~ 300GB.
So… like I said before, reason was file-walkers processes did not have time to finish because of restart/update for a new version of node.

“Patience is a virtue”

[Alexey]

The trash will shrink without involving of filewalkers, you need only piece:trash process finished successfully - it removes everything older than 7 days from there.

@Dunc4n1d4h0

So… like I said before, reason was file-walkers processes did not have time to finish because of restart/update for a new version of node.

or just before the restart (for any reason).

@Dunc4n1d4h0

“Patience is a virtue”

100%

[pdeline06]

On my dashboard I see “Trash: 2.89 TB”. But my “Trash” foldier contains summary 0.3 TB. Where is the different? And when the “Trash” from dashboard will be cleared?

[jammerdan]

@Alexey

No. It doesn’t look right.

The nested trash folders with date from 13th April are still present. Today is 23rd.

/storage/trash/pmw6tvzmf2jv6giyybmmvl4o2ahqlaldsaeha4yx74n5aaaaaaaa/2024-04-13/2024-04-13/2024-04-13/2024-04-13/2a

[Alexey]

The difference is in not finished filewalkers. You need to search for errors related to walk and trash.

[Alexey]

@Toyoo

You’ve nerdsniped me. Please admit that this was your goal.

Is it empty or have some data older that that date?

[jammerdan]

I checked only one folder and there is data from 12-04 in it.

[Alexey]

Please check with the scripts

How long does trash stay before getting deleted?

For bash: For Windows PowerShell:

[jammerdan]

It gives me 4721 MB.

[Alexey]

Wonderful. [Sarcasm]
I do not have it… What’s the difference?
My nodes on v1.101.3 and v1.102.3
But they are a Windows Service (1) and Docker Desktop for Windows (2)

[jammerdan]

This node tells me it is at 1.99.3.

[Alexey]

oh. I see, I do not have an old one at the moment. Could you please check after 1.10x.x?

[jammerdan]

Ok, let’s wait then.

[jammerdan]

@Alexey

Could you please check after 1.10x.x?

Node is at 1.102.3 now.
Note: It is one wit nested date dir:

TOTAL 791.123 MB
count 2867

So it is less than before. Something got deleted.

[Alexey]

Is it still like this now?

[jammerdan]

Today is TOTAL 7722.91 MB count 62371

[Alexey]

I believe you can remove subfolders named as a date which is older than 7 days:

Trash does not go away in 7 days

I’ll take those, I guess. While we would like to think that we are smart enough to foresee all possible bugs, it turns out we are still human . As to why there wasn’t a fallback mechanism, this is because the migration step is supposed to be extremely simple and quick (it consists only of three renames/moves) and in the unlikely event it was interrupted it could be fixed manually. But somehow the migration seems to have been kicked off multiple times all at once, and in some cases d…

[jammerdan]

I am running more than one node and I absolutely don’t want to do this manually.