# V58 db migrate fail - piece\_expiration.db - 1.108.3

**URL:** <https://forum.storj.io/t/v58-db-migrate-fail-piece-expiration-db-1-108-3/27238>\
**Category:** troubleshooting\
**Tags:** database\
**Created:** [July 23, 2024, 6:13am UTC](https://forum.storj.io/t/v58-db-migrate-fail-piece-expiration-db-1-108-3/27238 "2024-07-23T06:13:30Z")\
**Posts on this page:** 7\
**Page:** 3

<div class="post-metadata">

**Author:** ![Mad\_Max](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/mad_max/32/1127_2.png) [@Mad\_Max](https://forum.storj.io/u/Mad_Max)\
**Post date:** [July 28, 2024, 6:42am UTC](https://forum.storj.io/t/v58-db-migrate-fail-piece-expiration-db-1-108-3/27238/41 "2024-07-28T06:42:34Z")

</div>

> [@Alexey](#):
>
> Yes, and now it should delete these records from the TTL database to do not do it again on the next run.

I checked the database and can assure you that on this particular node, all records from it were correctly deleted by the TTL collector, almost immediately (with few minutes delay at max) after deleting the corresponding files from the disk. And it was this way for many days before even before update from v 1.105 to v 1.108.  
You don’t seem to understand exactly what the problem with the TTL collector is in the current (version \<= 1.108) storage nodes, which the above-mentioned patch should fix.

This is not because the TTL collector now allegedly does not delete records from the database at all after deleting files. It actually deletes them too, but it does so only after completing each pass. And starts this work anew if it is interrupted before it finished.  
Which can cause serious problems on **large** nodes (and I see them too, but only on my other LARGE nodes). This will have to be fixed by the batching processing patch mentioned above.

However, if the TTL collector completes the pass successfully, it correctly deletes all processed records from the database at the end of each pass.  
Both in version 1.108.x and in version 1.105.x and in some previous versions too. And this is true for my small node from the example above, where the collector correctly completed its work every hour by deleting all records processed in each pass from the database.

> [@Alexey](#):
>
> But I would still suggest confirming the guess that GC moved them to the trash.

That’s where you were right. I randomly selected 3 pieces IDs from the collector warnings log and they were all found in the folder  
`\trash\ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa\2024-07-27\`

So it looks like it was actually deleted by GC. This latest one (it still running now, about 80% done):

```auto
2024-07-27T20:53:00+03:00	INFO	retain	Prepared to run a Retain request.	{"cachePath": "C:\\Program Files\\Storj\\Storage Node/retain", "Created Before": "2024-07-21T17:59:59Z", "Filter Size": 4624470, "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S"}

```

But this is not due to problems with the TTL collector, because GC now deletes files created no later than July 21 - as indicated in the Bloom Filter parameters. And the TTL collector on this node has already successfully processed and deleted all records from the database up to the beginning of July 28:  
There is a SQL request to count records in piece\_expirations.db which match criteria

```auto
sqlite> SELECT count(*) FROM piece_expirations WHERE piece_expiration < datetime('2024-07-28T04:00:00');
0
sqlite> SELECT count(*) FROM piece_expirations WHERE piece_expiration < datetime('2024-07-28T10:00:00');
5043

```

Although I just forgot that right now we don’t have any direct data deletes from the storagenodes after deleting them from the satellite at the command of the end client? And all the data deleted by the clients remains on the node until the garbage collector finds it sooner or later?

Then maybe that’s what’s happening here. If some large client recently(15-20 JUL) uploaded a large amount of data with TTL set on the US1 satellite, but then did not wait for the expiration of the specified period and deleted this data manually instead. This can lead to a situation where the GC deletes data earlier than the TTL collector, even with its completely correct and timely operation of both.

So maybe it was a false alarm. But then this is a need for further improvement. Because this is not a temporary and can be repeated on regular basis later (as long as the nodes do not process deletes directly and rely on GC).

I think I should create another suggestion on Github for an appropriate improvement: GC, when deleting files, should check whether the ID of the files it deletes exist in piece\_expiration.db and if it is delete records from the database by itself. So that the TTL collector does not try to delete them again later and does not pour out thousands of useless warnings (which will provoke operators to simply disable or filter out these warnings, which then you can skip an important problem).

---

<div class="post-metadata">

**Author:** ![Alexey](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/alexey/32/41_2.png) [@Alexey](https://forum.storj.io/u/Alexey)\
**Post date:** [July 28, 2024, 7:57am UTC](https://forum.storj.io/t/v58-db-migrate-fail-piece-expiration-db-1-108-3/27238/42 "2024-07-28T07:57:38Z")

</div>

> [@Mad\_Max](#):
>
> Although I just forgot that right now we don’t have any direct data deletes from the storagenodes after deleting them from the satellite at the command of the end client?

You are correct, we do not have a direct deletions anymore (too costly for the client).

> [@Mad\_Max](#):
>
> And all the data deleted by the clients remains on the node until the garbage collector finds it sooner or later?

exactly right.

> [@Mad\_Max](#):
>
> If some large client recently(15-20 JUL) uploaded a large amount of data with TTL set on the US1 satellite, but then did not wait for the expiration of the specified period and deleted this data manually instead.

exactly. However, not necessarily the same client, it could be several (unless you also bypass the /24 rule).

> [@Mad\_Max](#):
>
> But then this is a need for further improvement. Because this is not a temporary and can be repeated on regular basis later (as long as the nodes do not process deletes directly and rely on GC).

I do not think that it requires to spent developers’ time on it. Nothing damaged and works as expected.  
It could improve things a little bit, but I do not think it’s worth to fix it. If that could significantly improve something, sure - I would try to ping the team and increase the priority (as was for irregular BF). However, if the Community could submit a PR for this case - it would be very helpful!

---

<div class="post-metadata">

**Author:** ![mgonzalezm](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/mgonzalezm/32/1125_2.png) [@mgonzalezm](https://forum.storj.io/u/mgonzalezm)\
**Post date:** [July 29, 2024, 5:52pm UTC](https://forum.storj.io/t/v58-db-migrate-fail-piece-expiration-db-1-108-3/27238/43 "2024-07-29T17:52:41Z")

</div>

> [@tfoutfou](#):
>
> current ongoing issue , (maybe a way to reproduce the error)
> 
> do a proper “docker stop” to stop the node → the database seem to be succesfully closed with no error ( no -shm or -wal files in database folder)
> 
> > sqlite3 piece\_expiration.db “PRAGMA integrity\_check;”
> 
> report no error either
> 
> then “docker rm” the node , then launch it again with the same command line as usual ( as i have always used and working fine until now)
> 
> then i have the database error in an infinite loop in the log file and the node is never restarting  
> have to put a new empty piece\_expiration.db each time
> 
> (tested on 3 different node that had uptime around 56hours)

I can confirm. It seems that the procedure used for versions rollout is still plagued with problems with versions downgrades.

I recently had to recreate the container for a node that was already at version 1.108.3 and after node startup it is now at version 1.107.3.

Will this cause errors with the database after this node receive version 1.108.3 again ?

---

<div class="post-metadata">

**Author:** ![Stez](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/stez/32/1130_2.png) [@Stez](https://forum.storj.io/u/Stez)\
**Post date:** [July 29, 2024, 7:18pm UTC](https://forum.storj.io/t/v58-db-migrate-fail-piece-expiration-db-1-108-3/27238/44 "2024-07-29T19:18:04Z")

</div>

@Alexey seeing the above message and looking at [version.storj.io](http://version.storj.io); shouldn’t the minimum version be 1.108.3 instead of 1.107.3? Rollout for 1.1.09.2 has started. Lets say anyone has an issue in Docker, stops the node, removes it, reinstalls, they might and likely willl go from 1.108.3 to 1.107.3 leading to DB mismatch errors. This is preventable from Storj’s side.

---

<div class="post-metadata">

**Author:** ![Alexey](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/alexey/32/41_2.png) [@Alexey](https://forum.storj.io/u/Alexey)\
**Post date:** [July 30, 2024, 8:43am UTC](https://forum.storj.io/t/v58-db-migrate-fail-piece-expiration-db-1-108-3/27238/45 "2024-07-30T08:43:43Z")

</div>

> [@V1.107.3 as "minimum" on https://version.storj.io](https://forum.storj.io/t/v1-107-3-as-minimum-on-https-version-storj-io/27338/14):
>
> Looks like v1.108.3 is now minimum. Now that collector properly updates the deleted pieces after a run of used-space we should see far less disk usage discrepancy reports on the forum. Interesting this is not the case for me.

---

<div class="post-metadata">

**Author:** ![tfoutfou](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/tfoutfou/32/7813_2.png) [@tfoutfou](https://forum.storj.io/u/tfoutfou)\
**Post date:** [July 30, 2024, 12:59pm UTC](https://forum.storj.io/t/v58-db-migrate-fail-piece-expiration-db-1-108-3/27238/46 "2024-07-30T12:59:03Z")

</div>

that’s exactly my problem right there , i couldnt find an explanation , and this is in fact so simple

---

<div class="post-metadata">

**Author:** ![Alexey](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/alexey/32/41_2.png) [@Alexey](https://forum.storj.io/u/Alexey)\
**Post date:** [August 1, 2024, 8:30am UTC](https://forum.storj.io/t/v58-db-migrate-fail-piece-expiration-db-1-108-3/27238/47 "2024-08-01T08:30:09Z")

</div>

You may re-create the container one more time, it should be upgraded to v1.108.3, or wait until it would upgrade to 1.109.x

[Previous page](https://forum.storj.io/t/v58-db-migrate-fail-piece-expiration-db-1-108-3/27238.md?page=2)