Piece_expiration.db - almost 1GB

Hi all,

Is it expected, that this db is growing to almost 1GB in size.

on other nodes, is sits about 150MiB - 200MiB

Thanks

I checked mine and they’re all around 5-8GB

Well that is certainly interesting then…

wonder if anyone else has seen a large DB size from “whatever has changed” recently.

As testing is currently pushing a lot of TTL data, it is normal for the piece_expiration.db file to expand as that is the database in charge of recording the expiration of the pieces.

6 Likes

Yep. This is also why I put expiration timestamp right next to other piece metadata in my proposal. The way it is now, we end up on average with ~120 bytes per piece with expiration timestamp of additional storage, accumulating to around a gigabyte per terabyte of TTL-ed pieces. Maybe not that much, but still requires additional planning from SNOs.

1 Like

My piece_expiration.db file was 7.8 GB and had become corrupted.

Rebuilding it would have taken over a day. I just deleted all .DB files and restarted Storj.

I appreciate your proposal and what you’re trying to do. Until it’s possibly adopted, I think I’m just going to delete all .db files annually so they don’t grow to that absurd size.

Would that not make the node keep the expired pieces forever, though?

Garbage collection (bloom filter) will delete them. I just takes some more time.

2 Likes

We should be paid for storing databases too… :grin:
They get too large to be ignored.

2 Likes

No. They would be collected by GC, and kept in the trash for a week, as any other garbage.
Just this database allows the node to remove them right away and do not use a trash.