# Quick growth of piece\_expiration.db

**URL:** <https://forum.storj.io/t/quick-growth-of-piece-expiration-db/24965>\
**Category:** troubleshooting\
**Tags:** database\
**Created:** [January 15, 2024, 5:58pm UTC](https://forum.storj.io/t/quick-growth-of-piece-expiration-db/24965 "2024-01-15T17:58:17Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![Walter1](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/walter1/32/7825_2.png) [@Walter1](https://forum.storj.io/u/Walter1)\
**Post date:** [January 15, 2024, 5:58pm UTC](https://forum.storj.io/t/quick-growth-of-piece-expiration-db/24965/1 "2024-01-15T17:58:17Z")

</div>

Hello together,

only some months ago I started a new node and it has already ~200 MB of the piece\_expiration.db. Can somebody explain the high growth rate and how to shrink it? The filewalker is not active.

 ![grafik](https://storj-s3.bcdn.literatehosting.net/original/2X/6/61c076cc1b9608c3edc6eaa37585476d6bbb06e5.png)

Thanks and kind regards,

---

<div class="post-metadata">

**Author:** ![Knowledge](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/knowledge/32/10681_2.png) [@Knowledge](https://forum.storj.io/u/Knowledge)\
**Post date:** [January 15, 2024, 6:06pm UTC](https://forum.storj.io/t/quick-growth-of-piece-expiration-db/24965/2 "2024-01-15T18:06:19Z")

</div>

Well, without Filewalker running to purge expired segments, you are going to build them up (And not get paid for having them) which subsequently increases the database size

---

<div class="post-metadata">

**Author:** ![nerdatwork](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/nerdatwork/32/1135_2.png) [@nerdatwork](https://forum.storj.io/u/nerdatwork)\
**Post date:** [January 15, 2024, 6:31pm UTC](https://forum.storj.io/t/quick-growth-of-piece-expiration-db/24965/3 "2024-01-15T18:31:18Z")

</div>

![image](https://storj-s3.bcdn.literatehosting.net/original/2X/1/1436230514452899296280c1c98861d880ecb0df.png)

Considering above data, does it mean I have 435 pieces that got left behind from the year 2022 ?

---

<div class="post-metadata">

**Author:** ![Walter1](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/walter1/32/7825_2.png) [@Walter1](https://forum.storj.io/u/Walter1)\
**Post date:** [January 15, 2024, 6:32pm UTC](https://forum.storj.io/t/quick-growth-of-piece-expiration-db/24965/4 "2024-01-15T18:32:44Z")

</div>

I got also way older nodes who have like 60 MB. Don’t know if it is filewalker-related.

---

<div class="post-metadata">

**Author:** ![Knowledge](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/knowledge/32/10681_2.png) [@Knowledge](https://forum.storj.io/u/Knowledge)\
**Post date:** [January 15, 2024, 6:47pm UTC](https://forum.storj.io/t/quick-growth-of-piece-expiration-db/24965/5 "2024-01-15T18:47:15Z")

</div>

Fair enough. I don’t know off the top of my head the database retention rules. Will ask an engineer.

---

<div class="post-metadata">

**Author:** ![daki82](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/daki82/32/1128_2.png) [@daki82](https://forum.storj.io/u/daki82)\
**Post date:** [January 15, 2024, 7:24pm UTC](https://forum.storj.io/t/quick-growth-of-piece-expiration-db/24965/6 "2024-01-15T19:24:09Z")

</div>

well ,my filwalker is runnig for sure, and i have same 250MB .db… at 3TB node data

maybe clients add data for only a short time?

---

<div class="post-metadata">

**Author:** ![Toyoo](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/toyoo/32/1126_2.png) [@Toyoo](https://forum.storj.io/u/Toyoo)\
**Post date:** [January 15, 2024, 9:23pm UTC](https://forum.storj.io/t/quick-growth-of-piece-expiration-db/24965/7 "2024-01-15T21:23:13Z")

</div>

Try [vacuuming](https://forum.storj.io/t/vacuum-databases-in-a-ramdisk-to-reduce-downtime/7066) it.

---

<div class="post-metadata">

**Author:** ![Walter1](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/walter1/32/7825_2.png) [@Walter1](https://forum.storj.io/u/Walter1)\
**Post date:** [January 15, 2024, 9:44pm UTC](https://forum.storj.io/t/quick-growth-of-piece-expiration-db/24965/8 "2024-01-15T21:44:39Z")

</div>

Thanks alot for the vacuum tipp.

Have tried it with a 440 MB piece\_expiration. Brought only 40 MB or 10% to 400 MB. Will have to delete the piece\_expiration.db as they are clearly the reason for the grafana-dashboard failures and very likely slow read-rate.

 ![grafik](https://storj-s3.bcdn.literatehosting.net/original/2X/7/76befc8e0f323cbf91f8940baf0014b988b642dd.png)

---

<div class="post-metadata">

**Author:** ![Walter1](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/walter1/32/7825_2.png) [@Walter1](https://forum.storj.io/u/Walter1)\
**Post date:** [January 18, 2024, 7:13am UTC](https://forum.storj.io/t/quick-growth-of-piece-expiration-db/24965/9 "2024-01-18T07:13:03Z")

</div>

I habe also a high file on bandwith.wal:

 ![grafik](https://storj-s3.bcdn.literatehosting.net/original/2X/0/03fd7e9f3dee1087f40954bad601bca02371b126.png)

---

<div class="post-metadata">

**Author:** ![Walter1](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/walter1/32/7825_2.png) [@Walter1](https://forum.storj.io/u/Walter1)\
**Post date:** [January 18, 2024, 1:26pm UTC](https://forum.storj.io/t/quick-growth-of-piece-expiration-db/24965/10 "2024-01-18T13:26:29Z")

</div>

When the data-bases are moved to a SSD, then the problem is solved. Unfortunately it also happends to a single node on a 20 TB CMR Enterprise HDD.

---

<div class="post-metadata">

**Author:** ![Alexey](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/alexey/32/41_2.png) [@Alexey](https://forum.storj.io/u/Alexey)\
**Post date:** [January 19, 2024, 3:40am UTC](https://forum.storj.io/t/quick-growth-of-piece-expiration-db/24965/11 "2024-01-19T03:40:09Z")

</div>

> [@Walter1](#):
>
> I habe also a high file on bandwith.wal:

> **[Write-Ahead Logging](https://www.sqlite.org/wal.html)**

---

<div class="post-metadata">

**Author:** ![nerdatwork](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/nerdatwork/32/1135_2.png) [@nerdatwork](https://forum.storj.io/u/nerdatwork)\
**Post date:** [January 19, 2024, 3:46am UTC](https://forum.storj.io/t/quick-growth-of-piece-expiration-db/24965/12 "2024-01-19T03:46:14Z")

</div>

> [@nerdatwork](#):
>
> does it mean I have 435 pieces that got left behind from the year 2022 ?

@Alexey your opinion on this please.

---

<div class="post-metadata">

**Author:** ![Alexey](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/alexey/32/41_2.png) [@Alexey](https://forum.storj.io/u/Alexey)\
**Post date:** [January 19, 2024, 3:48am UTC](https://forum.storj.io/t/quick-growth-of-piece-expiration-db/24965/13 "2024-01-19T03:48:34Z")

</div>

I have no idea, I usually respond only when I think that I know the answer.

---

<div class="post-metadata">

**Author:** ![nerdatwork](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/nerdatwork/32/1135_2.png) [@nerdatwork](https://forum.storj.io/u/nerdatwork)\
**Post date:** [January 19, 2024, 3:50am UTC](https://forum.storj.io/t/quick-growth-of-piece-expiration-db/24965/14 "2024-01-19T03:50:55Z")

</div>

> [@Alexey](#):
>
> I usually respond only when I think that I know the answer.

I appreciate and admire that in you hence the title of Awesome 🙂

---

<div class="post-metadata">

**Author:** ![snorkel](https://storj.bcdn.literatehosting.com/letter_avatar/snorkel/32/6_449c533f586939b6e5f246632a46421f.png) [@snorkel](https://forum.storj.io/u/snorkel)\
**Post date:** [January 19, 2024, 6:02am UTC](https://forum.storj.io/t/quick-growth-of-piece-expiration-db/24965/15 "2024-01-19T06:02:47Z")

</div>

I thought it’s only me, and I was going to ignore it, but I also have that db way too big. I remember I checked all dbs a few monyhs ago and the biggest was the bandwidth. Now the piece…db is the biggest. I’ll edit with all the sizes from my nodes in an hours or so. So this si a recent thing. The FW was run on all nodes last week.  
How can I “vacuum” them? Is it dangerous? Do I have to stop the nodes?

---

<div class="post-metadata">

**Author:** ![Alexey](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/alexey/32/41_2.png) [@Alexey](https://forum.storj.io/u/Alexey)\
**Post date:** [January 19, 2024, 7:18am UTC](https://forum.storj.io/t/quick-growth-of-piece-expiration-db/24965/16 "2024-01-19T07:18:15Z")

</div>

You can vacuum them the usual way for SQLite:

1. Stop and remove the container (or stop the service in case of Windows/Linux GUI)
2. Run the command

> [@Vacuum databases in a ramdisk to reduce downtime](https://forum.storj.io/t/vacuum-databases-in-a-ramdisk-to-reduce-downtime/7066):
>
> I have a bunch of nodes and tested runtime of vacuuming the databases in-place against copying them to a ramdisk, vacuuming them there, and moving them back to the disk. The results are quite impressive. The orders database takes the overwhelming majority of the time so I’m including its size before and after. Both nodes run on the same host (two Docker containers). They run on different HDDs; both are exactly the same model. Node 1: 300MB orders database. Vacuuming all databases in-place t…

However

> [@Database is locked. What is the reason? What is the possible solution?](https://forum.storj.io/t/database-is-locked-what-is-the-reason-what-is-the-possible-solution/6806/30):
>
> Why do we have to call vacuum? That shouldn’t be needed. Autovacuum should also not be needed.

I think you should not fix what’s not broken.

---

<div class="post-metadata">

**Author:** ![snorkel](https://storj.bcdn.literatehosting.com/letter_avatar/snorkel/32/6_449c533f586939b6e5f246632a46421f.png) [@snorkel](https://forum.storj.io/u/snorkel)\
**Post date:** [January 19, 2024, 10:59am UTC](https://forum.storj.io/t/quick-growth-of-piece-expiration-db/24965/17 "2024-01-19T10:59:05Z")

</div>

I don’t know, but something fishy is happening. The piece\_expiration.db is growing FAST!  
These are all my nodes; take note that those small ones are started on 30-31 december, so I don’t even know if they are fully vetted, and have huge db for their age, too. The FW was run.  
Synology’s, Exos, ext4, no ssd.

Node: space occupied / piece\_expiration.db

node11: 12.3TB / 527MiB  
node12: 431GB / 77MiB

node21: 14.3TB / 512MiB  
node22: 426GB / 77MiB

node31: 6.8TB / 160MiB  
node32: 6.8TB / 255MiB

node41: 11.1TB / 522MiB  
node42: 425GB / 78MiB

node51: 12.8TB / 530MiB  
node52: 433GB / 78MiB

node61: 14.4TB / 418MiB

node71: 14.5TB / 324MiB  
node72: 654GB / 105MiB

node81: 14.4TB / 516MiB  
node82: 427GB / 78MiB

---

<div class="post-metadata">

**Author:** ![Walter1](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/walter1/32/7825_2.png) [@Walter1](https://forum.storj.io/u/Walter1)\
**Post date:** [January 19, 2024, 11:09am UTC](https://forum.storj.io/t/quick-growth-of-piece-expiration-db/24965/18 "2024-01-19T11:09:31Z")

</div>

I have also several 400-500 MB piece\_expiration DBs. Like I said the only solution was to move the DBs to a SSD and to change the --mount order, with this guide:

> [@How to move DB’s to SSD on Docker](https://forum.storj.io/t/how-to-move-db-s-to-ssd-on-docker/7064):
>
> How to move DB’s to SSD on Docker Before you beginning, please make sure that your SSD has good endurance (MLC is preferred), I personally recommend using SSD mirror. look into the [official documentation](https://documentation.storj.io/setup/cli/storage-node) and make sure that you are using –mount type=bind parameter in your docker run string Prepare a folder with mounted SSD outside of \<storage-dir\> from the official documentation. (it your folder with pieces) Add a new mont string to your docker run string: Now we have: docker run -d --restar…

Now the dashboard and the STORJ-Exporterlogs are fine after the DB-moving.

---

<div class="post-metadata">

**Author:** ![thepaul](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/thepaul/32/65_2.png) [@thepaul](https://forum.storj.io/u/thepaul)\
**Post date:** [January 19, 2024, 3:11pm UTC](https://forum.storj.io/t/quick-growth-of-piece-expiration-db/24965/19 "2024-01-19T15:11:47Z")

</div>

> [@nerdatwork](#):
>
> Considering above data, does it mean I have 435 pieces that got left behind from the year 2022 ?

It certainly seems like something is wrong. The collector service on your node should be continually trying to get rid of any entries older than the current time. My only theory is that it is experiencing some error trying to delete those files. If that’s the case, it would keep those entries around so it could try again later. Do you have any errors in the log relating to a service called “collector”?

Another thing to check would be whether the `deletion_failed_at` column is set for those entries:

```
SELECT
    satellite_id, piece_id, piece_expiration, deletion_failed_at
FROM
    piece_expirations
WHERE
    piece_expiration < '2023-01-01 00:00:00';
ORDER BY
    piece_expiration;

```

that would mark the last time the node failed in trying to delete those pieces. If it’s blank, something altogether different is going on.

---

<div class="post-metadata">

**Author:** ![nerdatwork](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/nerdatwork/32/1135_2.png) [@nerdatwork](https://forum.storj.io/u/nerdatwork)\
**Post date:** [January 19, 2024, 3:48pm UTC](https://forum.storj.io/t/quick-growth-of-piece-expiration-db/24965/20 "2024-01-19T15:48:25Z")

</div>

> [@thepaul](#):
>
> `'2023-01-01 00:00:00';`

The semi colon terminated the `where` clause so I edited it 🤓

 ![image](https://storj-s3.bcdn.literatehosting.net/original/2X/8/86fad96d3e6ff3da57827fb21a3a7431473e7418.png)

> [@thepaul](#):
>
> Do you have any errors in the log relating to a service called “collector”?

I will work on that and edit this post.

Edit:

```auto
2023-12-25T02:17:48Z ERROR collector unable to update piece info {"process": "storagenode", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Piece ID": "2XLSFR7RKZXCZZUBAATNNJRFNZG7DTSXQGKQ3UILLHTHO3AZ3FKQ", "error": "pieceexpirationdb:
database is locked", "errorVerbose": "pieceexpirationdb: database is locked\n\tstorj.io/storj/storagenode/storagenodedb.(*pieceExpirationDB).DeleteFailed:99\n\tstorj.io/storj/storagenode/pieces.(*Store).DeleteFailed:597\n\tstorj.io/storj/storagenode/collector.(*Servic
e).Collect:109\n\tstorj.io/storj/storagenode/collector.(*Service).Run.func1:57\n\tstorj.io/common/sync2.(*Cycle).Run:160\n\tstorj.io/storj/storagenode/collector.(*Service).Run:53\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:51\n\tsto
rj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75"}
2023-12-25T02:17:48Z ERROR collector unable to delete piece {"process": "storagenode", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Piece ID": "2XLSFR7RKZXCZZUBAATNNJRFNZG7DTSXQGKQ3UILLHTHO3AZ3FKQ", "error": "pieces error: database is
locked", "errorVerbose": "pieces error: database is locked\n\tstorj.io/storj/storagenode/pieces.(*Store).DeleteExpired:365\n\tstorj.io/storj/storagenode/pieces.(*Store).Delete:344\n\tstorj.io/storj/storagenode/collector.(*Service).Collect:97\n\tstorj.io/storj/storagen
ode/collector.(*Service).Run.func1:57\n\tstorj.io/common/sync2.(*Cycle).Run:160\n\tstorj.io/storj/storagenode/collector.(*Service).Run:53\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:51\n\tstorj.io/storj/private/lifecycle.(*Group).Ru
n.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75"}

```

[Next page](https://forum.storj.io/t/quick-growth-of-piece-expiration-db/24965.md?page=2)
