How long ago was "The Big Purge"?

Node Operators
1

Bit of a stupid question but I’m trying to remember when “The Big Purge” was, as my nodes still have a lot of trash that’s not been “released” and I could swear it was over 7 days ago.

Reason I ask is that one of my nodes is currently full and missing out on the all the testing because it has 1.5TB in trash that doesn’t seem to bloody go away…

1 Like
2

Welcome in the club.
This can have many reasons:

  • Botched migration
  • Nested date folders
  • Involuntarily downgrade
  • Old and new style trash folders mixes
  • Old style trash folder but date trash folders enabled

And probably more.

2 Likes
3

INFO pieces:trash emptying trash started {"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE"}
cleaning started this morning

4

That’s interesting. I’ll grep my logs for that and see if something is happening :slight_smile:

EDIT: no trash cleanups on that node since the 18th…

And should I be worried about

2024-05-23T06:50:48Z INFO retain Moved pieces to trash during retain {"Process": "storagenode", "cachePath": "config/retain", "Deleted pieces": 11444, "Failed to delete": 0, "Pieces failed to read": 0, "Pieces count": 496848, "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Duration": "20.707503259s", "Retain Status": "enabled"}

5

i dont think so. It means more data have been sent to trash.

1 Like
6

I expect a lot of it will be deleted today. The trash cleanup from yesterday had such lines in the log

lazyfilewalker.trash-cleanup-filewalker.subprocess      trash-filewalker started        {"Process": "storagenode", "satelliteID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Process": "storagenode", "dateBefore": "2024-05-16T14:52:11Z"

The oldest folders in my trash are 2024-05-16 with a lot of data in it.

7

image
image825×404 19.9 KB

It’s mine. :rofl:

8

We are going to see big deleting on nodes (trash delete not gc) from now to entire next week. Nodes will be stressed a lot

1 Like
9

Let’s hope

8
8888×789 29.1 KB

10

image
image750×786 29.8 KB

Yes! Give me back my free space already! :smile:

11

изображение
изображение790×1337 74.6 KB

12

Yep, same here. Doesn’t seem to have deleted anything…

13

it will take some hours :smiley: or days.

1 Like
14

As expected half of my trash across all nodes (~3.5TB out of ~7TB) got deleted today. Took 30 minutes max. on my nodes on xfs and only 10 minutes on my biggest node with the most trash on ext4.

15

I lost a small amount (300GB out of 1.3TB on my full node). I’m hoping the rest will go over the weekend. I was worried I’d be “stuck” with that garbage as @jammerdan has been having issues for a while, but it seems like it may be working.

1 Like