Big loss of stored data, gone to trash, during one night

Node Operators
4

I also have several nodes with trash 30-340 GB soo looks like storj delete some test data to free up space

5

… as long as there’s engress, all is fine :sweat_smile:

1 Like
6

I even havent noticed, because used space didnt even dropped

7

It’s been moved to trash. It’ll drop in a week.

1 Like
8

WOW! Mine currently is pending deleting about 160GB

9

can we somehow, delete them earlier?

10

No, the whole reason garbage collection was paused was because they had to roll back an earlier garbage collection run. If that needs to happen again and you removed the data, your node will get disqualified for losing data. Never remove anything yourself.

5 Likes
11

ty for the response!

12

Ok, thank you all for your reassuring answers.

13

Yea, it happens. I lost about 3.8TB from all my nodes.

1 Like
14

You didn’t really lose anything. As that data was unpaid to begin with its more reasonable to see it as you gaining 3.8TB of free space.

4 Likes
15

Glass half empty, glass half full… :wink:

1 Like
16

[snorkel] previously said: “… as long as there’s engress, all is fine”
Not only, trash continued to grow, up to 1.5TB.
image

But ingress is completely stopped since 4 days.
image

All is normal, really ?
Or i must only wait trash to be cleared, for ingress to restart ?

17

Ingress will restart when trash has been cleared, as currently you do not have any free space left on your node. We apologize for the inconvenience.

2 Likes
18

dont worry, after 7 days it will go free.

1 Like
19

Upload tb, send tb to trash, restore, send,restore,send, repeat while won’t get bored. All zfs (and other COW systems) nodes will die =). If client dont pay for trash, then this look like as attack

20

it is storj lab deleting old test data, to free up space for customers.

21

Customers can’t restore anything from trash. They’d have to upload new data and delete for the “attack” you suggest. Which has no benefit to them and quite a bit of cost and effort for pretty much no effect.

It wasn’t. This was garbage collection which hasn’t run for a while. It’s merely cleaning up (unpaid) data nodes should no longer hold.

22

That makes sense. My reported “Disk Average Month” is way lower than it should be if all previously held data was paid for.

23

Yep, that’s how I first noticed it, before the garbage collection even started. See: Discrepancy between B*h reported by satellites and storage used on node (impacting node payouts)