I think I have a trash accounting problem

Node Operators troubleshooting
,
21

Yeap. However, these files should be finally removed. Do you have any old files right now?

22

So I had to restart the node again and increase available space because if it continues this way, it will take weeks until the trash my be correctly displayed and calculated.

After restart I grepped for trash. Does this look like a healthy completed used space filewalker run?

2024-11-30T09:55:02Z    INFO    pieces  used-space-filewalker started   {"Process": "storagenode", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S"}
2024-11-30T09:55:02Z    INFO    pieces  used-space-filewalker completed {"Process": "storagenode", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Lazy File Walker": false, "Total Pieces Size": 913913456960, "Total Pieces Content Size": 911850072384, "Total Pieces Count": 4030048, "Duration": "27.627323ms"}
2024-11-30T09:55:02Z    INFO    pieces  used-space-filewalker started   {"Process": "storagenode", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs"}
2024-11-30T09:55:02Z    INFO    pieces  used-space-filewalker completed {"Process": "storagenode", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Lazy File Walker": false, "Total Pieces Size": 256652959232, "Total Pieces Content Size": 256417563648, "Total Pieces Count": 459757, "Duration": "16.340904ms"}
2024-11-30T09:55:02Z    INFO    pieces  used-space-filewalker started   {"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE"}
2024-11-30T09:55:02Z    INFO    pieces  used-space-filewalker completed {"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Lazy File Walker": false, "Total Pieces Size": 6864012544, "Total Pieces Content Size": 6859217152, "Total Pieces Count": 9366, "Duration": "43.939305ms"}
2024-11-30T09:55:02Z    INFO    pieces  used-space-filewalker started   {"Process": "storagenode", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6"}
2024-11-30T09:55:02Z    INFO    pieces  used-space-filewalker completed {"Process": "storagenode", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Lazy File Walker": false, "Total Pieces Size": 54428535040, "Total Pieces Content Size": 54350977280, "Total Pieces Count": 151480, "Duration": "24.84229ms"}

To me this looks like all satellites have been successfully finished. So does that mean the the filewalker has also accounted the trash?

23

It should. However, there is a bug likely:

24

Good then. Because again the value for trash did not go down it went up.

Yes, and I am still waiting for the date folder to get deleted. It is still working in it. If that is finally over we will see if it will get updated after.

1 Like
25

Finally, I can see that the trash has decreased. I’m not sure what triggered the update; the date folder is still being processed, and I can’t identify any specific action in the log that appears to have triggered it.

What I do see is that the version of that node is now 1.116.7; I believe it was 1.115.x before. So perhaps that was the issue. Let’s see if this resolves the situation on the other nodes as well.

26

Maybe it’s flushed the collected info to the database. It should do it in a timely manner and in batches, so maybe it’s.

The update should restart the node, so I would expect that all memory caches should be flushed to the databases as well.

And it’s good, that the trash usage is not affected by the filewalker bug discussed above.