Big loss of stored data, gone to trash, during one night

Node Operators
1

Hi guys.
Here is the context:
I recently modified my storage node size, previously full, from 3TB to 3.5TB, by modifying the Docker run command, in the same 3x2TB Raid 5 array (4TB) in a Synology server, and keeping ~9% of free disk space as recommended.
It was done without problem and the node size began to grow smoothly, up to be again nearly full, some days ago. My node version is v1.67.1 today.
Yesterday i saw that, ~1.2 TB gone !:
image
And trash continue to grow… From 1.21TB to 1,23TB after 15mn…
Is it normal ? Is test data being removed ? Or version problem ?
Or big Storj customer erasing his data ?
Thank you.

2

Some system-level corrections. Big deletions happening?

Don’t worry, everything’s fine.

3 Likes
3

Apparently there has been a lot of catching up from the network’s garbage collector so I suspect your node is catching up.
I’ve seen an increase in Trash data on my nodes, although not quite THAT much…

I suspect it’s probably normal.

3 Likes
4

I also have several nodes with trash 30-340 GB soo looks like storj delete some test data to free up space

5

… as long as there’s engress, all is fine :sweat_smile:

1 Like
6

I even havent noticed, because used space didnt even dropped

7

It’s been moved to trash. It’ll drop in a week.

1 Like
8

WOW! Mine currently is pending deleting about 160GB

9

can we somehow, delete them earlier?

10

No, the whole reason garbage collection was paused was because they had to roll back an earlier garbage collection run. If that needs to happen again and you removed the data, your node will get disqualified for losing data. Never remove anything yourself.

5 Likes
11

ty for the response!

12

Ok, thank you all for your reassuring answers.

13

Yea, it happens. I lost about 3.8TB from all my nodes.

1 Like
14

You didn’t really lose anything. As that data was unpaid to begin with its more reasonable to see it as you gaining 3.8TB of free space.

4 Likes
15

Glass half empty, glass half full… :wink:

1 Like
16

[snorkel] previously said: “… as long as there’s engress, all is fine”
Not only, trash continued to grow, up to 1.5TB.
image

But ingress is completely stopped since 4 days.
image

All is normal, really ?
Or i must only wait trash to be cleared, for ingress to restart ?

17

Ingress will restart when trash has been cleared, as currently you do not have any free space left on your node. We apologize for the inconvenience.

2 Likes
18

dont worry, after 7 days it will go free.

1 Like
19

Upload tb, send tb to trash, restore, send,restore,send, repeat while won’t get bored. All zfs (and other COW systems) nodes will die =). If client dont pay for trash, then this look like as attack

20

it is storj lab deleting old test data, to free up space for customers.