What is the garbage folder used for?

donald.m.motsinger · April 23, 2024, 10:48am

Can somebody explain, what the garbage folder is used for? It’s empty on some nodes, however on some nodes there are 1 or more files in it named after the satellites like the folders in the blobs folder. The youngest of these files is from 2024-02-29 and the oldest from last year.

jammerdan · April 23, 2024, 11:03am

github.com/storj/storj

storagenode/blobstore: remove "garbage" step during deletion

committed 06:08PM - 16 Feb 24 UTC

thepaul

+5 -172

Previously, the blobstore would move files into a directory called "garbage" bef…ore trying to delete them (whether they were being deleted directly because of an API call, deleted after removal of a satellite, deleted after an expiration time, or deleted while cleaning up the trash dir). Then we would actually perform the filesystem deletion. This was done so that if the os.Remove() call failed, we could tell immediately that the file was supposed to be deleted because it was in a "garbage" directory. The idea was that we would periodically go through and delete things in the garbage directory. There was also an in-memory deletion queue connected to the garbage directory. The periodic garbage deletion was supposed to process that queue as well. However, we don't ever call the garbage deletion routine except in tests, and I think we never have. Furthermore, it is confusing that we have both "trash" and "garbage" directories which have totally different meanings. We also have "garbage collection" (gc) which is another totally different thing. And there is another deletion queue at the piece store layer which works in a different way from the garbage delete queue, but which also contributes to the confusion. Now that we rely on gc for most node cleanup, it seems safe to skip this "garbage" step and the attached deletion queue and remove the related code. This should improve performance of the trash filewalker to some extent, since we won't need the extra directory lookup and the extra directory writes to accomplish the move before every deletion. It will still be possible for the os.Remove() call to fail, to be sure. This should be fine, because we should be able to do the deletion with the next Retain() call. Change-Id: I908ae54c7f20d0f59149231288277d3b41036fff

donald.m.motsinger · April 23, 2024, 11:13am

Am I seeing this right that the use of the garbage folder was removed in 1.99.1?

Alexey · April 25, 2024, 4:14am

seems so:

main

v1.103.1-rc

…

v1.99.0-rc