What happened with trash?

Node Operators
1

It looks like almost all trash suddenly was recovered on all of my nodes regardless version.
Used space in dashboard increased, trash decreased to symbolic values and this is not dashboard bug. Trash folder is really empty.
Is anyone else seeing the same node behavior?

3 Likes
2

Same thing for me today, I was surprised too.

2 Likes
3

I dont know about trash, but node suddenly razed in space for at least 1-2 TB, but ingress is low.

4

It seems like the trash was moved to “used” again. Maybe they had to restore for some reason?

5

A rare trash restore:

2024-11-02T06:53:46-07:00	INFO	pieces:trash	restore trash started	{"Process": "storagenode", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S"}
2024-11-02T06:56:32-07:00	INFO	pieces:trash	restore trash started	{"Process": "storagenode", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs"}
2024-11-02T07:00:55-07:00	INFO	pieces:trash	restore trash started	{"Process": "storagenode", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6"}
2024-11-02T07:13:11-07:00	INFO	pieces:trash	restore trash finished	{"Process": "storagenode", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6"}
2024-11-02T07:32:34-07:00	INFO	pieces:trash	restore trash finished	{"Process": "storagenode", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs"}
2024-11-02T10:13:08-07:00	INFO	pieces:trash	restore trash finished	{"Process": "storagenode", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S"}

This node recovered about 400GB.

4 Likes
6

Yes all my trash is gone and i now have 2 TB more space used.
And thats why you dont touch trash yourself. This could have easily killed a node if wrong files were deleted via user themself.
Good thing is, that storjstats dont show lost segment. Bad thing is, there has to be something wrong.

3 Likes
7

used but not paid… :wink:

8

If space is showing as “used”, it is being paid for at $1.5 a month, right?

Same here. One node is showing 200KB, other 0 trash. Yay.

1 Like
9

The engineers restored data from trash as they are conducting some research around test data that was recently deleted. It will likely move back to trash in some days as their tests complete. I don’t have a timeline for this however so enjoy the extra paid data while testing continues.

12 Likes
10

So to recover trash, do you have to recover everything from all satellites? Or can you recover specific satellites or file too?

11

I believe they can do individual files or all files. From what I understand they did all in this case.

2 Likes
12

Restore is per satellite but not per file. It is all or nothing. Handing over a list of pieceIDs to restore doesn’t really work. At the moment we only know that something is off so we better restore everything to not risk anything over the weekend and continue on Monday with further investigations.

8 Likes
13

But why did some nodes have trash remaining?
du -hs /node?/node?/storage/trash/
30G /node1/node1/storage/trash/
43M /node2/node2/storage/trash/
30G /node3/node3/storage/trash/
19G /node4/node4/storage/trash/
29G /node5/node5/storage/trash/
29G /node6/node6/storage/trash/

14

What folders there?

tree -L 2 /node?/node?/storage/trash/
15 
/node1/node1/storage/trash/
├── pmw6tvzmf2jv6giyybmmvl4o2ahqlaldsaeha4yx74n5aaaaaaaa
├── qstuylguhrn2ozjv4h2c6xpxykd622gtgurhql2k7k75wqaaaaaa
│   ├── 2024-10-27
│   ├── 2024-10-30
│   └── 2024-10-31
├── ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa
│   ├── 2024-10-28
│   └── 2024-10-31
└── v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa
    ├── 2024-10-27
    ├── 2024-10-28
    ├── 2024-10-31
    └── 2024-11-03
/node2/node2/storage/trash/
├── pmw6tvzmf2jv6giyybmmvl4o2ahqlaldsaeha4yx74n5aaaaaaaa
├── qstuylguhrn2ozjv4h2c6xpxykd622gtgurhql2k7k75wqaaaaaa
│   ├── 2024-10-28
│   ├── 2024-10-29
│   └── 2024-10-31
├── ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa
│   ├── 2024-10-28
│   └── 2024-11-01
└── v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa
    ├── 2024-10-26
    ├── 2024-10-29
    ├── 2024-10-30
    └── 2024-11-02
/node3/node3/storage/trash/
├── pmw6tvzmf2jv6giyybmmvl4o2ahqlaldsaeha4yx74n5aaaaaaaa
├── qstuylguhrn2ozjv4h2c6xpxykd622gtgurhql2k7k75wqaaaaaa
│   ├── 2024-10-27
│   ├── 2024-10-29
│   └── 2024-11-01
├── ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa
│   ├── 2024-10-26
│   ├── 2024-10-28
│   └── 2024-10-31
└── v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa
    ├── 2024-10-27
    ├── 2024-10-29
    ├── 2024-10-31
    └── 2024-11-03
/node4/node4/storage/trash/
├── pmw6tvzmf2jv6giyybmmvl4o2ahqlaldsaeha4yx74n5aaaaaaaa
├── qstuylguhrn2ozjv4h2c6xpxykd622gtgurhql2k7k75wqaaaaaa
│   ├── 2024-10-27
│   ├── 2024-10-30
│   └── 2024-10-31
├── ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa
│   ├── 2024-10-28
│   └── 2024-11-01
└── v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa
    ├── 2024-10-26
    ├── 2024-10-29
    ├── 2024-10-30
    └── 2024-11-03
/node5/node5/storage/trash/
├── pmw6tvzmf2jv6giyybmmvl4o2ahqlaldsaeha4yx74n5aaaaaaaa
├── qstuylguhrn2ozjv4h2c6xpxykd622gtgurhql2k7k75wqaaaaaa
│   ├── 2024-10-27
│   ├── 2024-10-29
│   └── 2024-10-31
├── ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa
│   ├── 2024-10-28
│   └── 2024-10-31
└── v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa
    ├── 2024-10-27
    ├── 2024-10-28
    ├── 2024-10-30
    └── 2024-11-03
/node6/node6/storage/trash/
├── pmw6tvzmf2jv6giyybmmvl4o2ahqlaldsaeha4yx74n5aaaaaaaa
├── qstuylguhrn2ozjv4h2c6xpxykd622gtgurhql2k7k75wqaaaaaa
│   ├── 2024-10-28
│   ├── 2024-10-30
│   └── 2024-10-31
├── ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa
│   ├── 2024-10-26
│   ├── 2024-10-28
│   └── 2024-10-31
└── v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa
    ├── 2024-10-26
    ├── 2024-10-29
    ├── 2024-10-31
    └── 2024-11-03
16

Mystery solved

du -hs /node?/node?/storage/trash/*/* | grep G
30G     /node1/node1/storage/trash/v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa/2024-11-03
30G     /node3/node3/storage/trash/v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa/2024-11-03
19G     /node4/node4/storage/trash/v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa/2024-11-03
29G     /node5/node5/storage/trash/v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa/2024-11-03
29G     /node6/node6/storage/trash/v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa/2024-11-03
1 Like
17

So some data was untrashed… but will soon be retrashed?

Got it! :+1:

18

It has already been retrashed. My nodes again showing 270GB and 47GB trash, previously they were showing 0.

1 Like
19

wow catching up on this thread has been an emotional roller coaster.

2 Likes
20

Yes, so if the untrash everything gets used, the very next bloom starts trashing again.
So the “enjoy your extra data that is paid for” straight away become “enjoy your 7 days extra trash” averaged to 3.5 days of course.
And enjoy thrashing your disk to untrash and then trash again.

It’s the trash-untrash-retrash-thrash-nocash scandal!!!

5 Likes