Piece_expiration.db - no solution - HELP!

Node Operators troubleshooting
1

Good day to all!

I have a problem that I can’t find a solution on my own. I tried all available options.

There was a sudden shutdown of the node.
After the node was restored, frequent errors appeared:

2024-04-15T14:55:56+03:00 ERROR piecestore upload failed {“Piece ID”: “UKRYABIFNJ35D7KPHVC774QAD3GSQFPQK2LJ6SNSVQOFPWLU6AUA”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “Remote Address”: “79.127.219.33:34264”, “Size”: 121856, “error”: “pieceexpirationdb: database disk image is malformed”, “errorVerbose”: “pieceexpirationdb: database disk image is malformed\n\tstorj.io/storj/storagenode/storagenodedb.(*pieceExpirationDB).SetExpiration:67\n\tstorj.io/storj/storagenode/pieces.(*Store).SetExpiration:608\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload.func6:487\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:545\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:294\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:61\n\tstorj.io/common/experiment.(*Handler).HandleRPC:42\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:167\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:109\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:157\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35”}

After checking the data bases, corruption was detected:

piece_expiration.db *** in database main *** Tree 2 page 131350 cell 1: Rowid 4159861 out of order Tree 2 page 131350 cell 0: Rowid 4159830 out of order Tree 5 page 126064 cell 28: 2nd reference to page 130083 Tree 5 page 126064 cell 27: 2nd reference to page 129967 Tree 4 page 625 cell 54: 2nd reference to page 69459 Tree 4 page 625 cell 53: 2nd reference to page 69152 Tree 4 page 625 cell 52: 2nd reference to page 67816 Tree 4 page 625 cell 51: 2nd reference to page 67412 Tree 4 page 625 cell 50: 2nd reference to page 67362 Tree 4 page 625 cell 49: 2nd reference to page 66221 Tree 4 page 625 cell 48: 2nd reference to page 66082 Tree 4 page 625 cell 47: 2nd reference to page 9601 row 8868 missing from index idx_piece_expirations_deletion_failed_at row 8868 missing from index idx_piece_expirations_piece_expiration row 8869 missing from index idx_piece_expirations_deletion_failed_at row 8869 missing from index idx_piece_expirations_piece_expiration row 8869 missing from index sqlite_autoindex_piece_expirations_1 row 8870 missing from index idx_piece_expirations_deletion_failed_at row 8870 missing from index idx_piece_expirations_piece_expiration row 8870 missing from index sqlite_autoindex_piece_expirations_1 row 18514 missing from index sqlite_autoindex_piece_expirations_1 row 18926 missing from index sqlite_autoindex_piece_expirations_1 row 82562 missing from index sqlite_autoindex_piece_expirations_1 row 104083 missing from index sqlite_autoindex_piece_expirations_1 row 106653 missing from index sqlite_autoindex_piece_expirations_1 row 141043 missing from index sqlite_autoindex_piece_expirations_1 row 144261 missing from index sqlite_autoindex_piece_expirations_1 row 184251 missing from index sqlite_autoindex_piece_expirations_1 row 184638 missing from index sqlite_autoindex_piece_expirations_1 row 188743 missing from index sqlite_autoindex_piece_expirations_1 row 189520 missing from index sqlite_autoindex_piece_expirations_1 row 198386 missing from index sqlite_autoindex_piece_expirations_1 row 221164 missing from index sqlite_autoindex_piece_expirations_1 row 225413 missing from index sqlite_autoindex_piece_expirations_1 row 227654 missing from index sqlite_autoindex_piece_expirations_1 row 236832 missing from index sqlite_autoindex_piece_expirations_1 row 238844 missing from index sqlite_autoindex_piece_expirations_1 row 242400 missing from index sqlite_autoindex_piece_expirations_1 row 247027 missing from index sqlite_autoindex_piece_expirations_1 row 261589 missing from index sqlite_autoindex_piece_expirations_1 row 285856 missing from index sqlite_autoindex_piece_expirations_1 row 295295 missing from index sqlite_autoindex_piece_expirations_1 row 325082 missing from index sqlite_autoindex_piece_expirations_1 row 332086 missing from index sqlite_autoindex_piece_expirations_1 row 348828 missing from index sqlite_autoindex_piece_expirations_1 row 358845 missing from index sqlite_autoindex_piece_expirations_1 row 436435 missing from index sqlite_autoindex_piece_expirations_1 row 438431 missing from index sqlite_autoindex_piece_expirations_1 row 454916 missing from index sqlite_autoindex_piece_expirations_1 row 487849 missing from index sqlite_autoindex_piece_expirations_1 non-unique entry in index sqlite_autoindex_piece_expirations_1 row 503657 missing from index sqlite_autoindex_piece_expirations_1 row 519842 missing from index sqlite_autoindex_piece_expirations_1 row 581742 missing from index sqlite_autoindex_piece_expirations_1 row 582858 missing from index sqlite_autoindex_piece_expirations_1 row 598393 missing from index sqlite_autoindex_piece_expirations_1 row 602571 missing from index sqlite_autoindex_piece_expirations_1 row 614211 missing from index idx_piece_expirations_deletion_failed_at row 614212 missing from index idx_piece_expirations_deletion_failed_at row 614212 missing from index idx_piece_expirations_piece_expiration row 614213 missing from index idx_piece_expirations_deletion_failed_at row 614213 missing from index idx_piece_expirations_piece_expiration row 614214 missing from index idx_piece_expirations_deletion_failed_at row 614214 missing from index idx_piece_expirations_piece_expiration row 614215 missing from index idx_piece_expirations_deletion_failed_at row 614215 missing from index idx_piece_expirations_piece_expiration row 614216 missing from index idx_piece_expirations_deletion_failed_at row 614216 missing from index idx_piece_expirations_piece_expiration row 614217 missing from index idx_piece_expirations_deletion_failed_at row 614217 missing from index idx_piece_expirations_piece_expiration row 614218 missing from index idx_piece_expirations_deletion_failed_at row 614218 missing from index idx_piece_expirations_piece_expiration row 614219 missing from index idx_piece_expirations_deletion_failed_at row 614219 missing from index idx_piece_expirations_piece_expiration row 614220 missing from index idx_piece_expirations_deletion_failed_at row 614220 missing from index idx_piece_expirations_piece_expiration row 614221 missing from index idx_piece_expirations_deletion_failed_at row 614221 missing from index idx_piece_expirations_piece_expiration row 614222 missing from index idx_piece_expirations_deletion_failed_at row 614222 missing from index idx_piece_expirations_piece_expiration row 614223 missing from index idx_piece_expirations_deletion_failed_at row 614223 missing from index idx_piece_expirations_piece_expiration row 614224 missing from index idx_piece_expirations_deletion_failed_at row 614224 missing from index idx_piece_expirations_piece_expiration row 614225 missing from index idx_piece_expirations_deletion_failed_at row 614225 missing from index idx_piece_expirations_piece_expiration row 614226 missing from index sqlite_autoindex_piece_expirations_1 row 614227 missing from index sqlite_autoindex_piece_expirations_1 row 614228 missing from index sqlite_autoindex_piece_expirations_1 row 614229 missing from index sqlite_autoindex_piece_expirations_1 row 614230 missing from index sqlite_autoindex_piece_expirations_1 row 614231 missing from index sqlite_autoindex_piece_expirations_1 row 614232 missing from index sqlite_autoindex_piece_expirations_1 row 614233 missing from index sqlite_autoindex_piece_expirations_1 row 614234 missing from index sqlite_autoindex_piece_expirations_1 row 614235 missing from index sqlite_autoindex_piece_expirations_1 row 614236 missing from index sqlite_autoindex_piece_expirations_1 row 614237 missing from index sqlite_autoindex_piece_expirations_1 row 614238 missing from index sqlite_autoindex_piece_expirations_1 row 614239 missing from index sqlite_autoindex_piece_expirations_1

I did a database FIX. When checking the FIXED database it was - piece_expiration.db ok
But the node did not start:

2024-04-15T13:58:02+03:00 ERROR failure during run {“error”: “Error migrating tables for database on storagenode: migrate: v26: duplicate column name: trash\n\tstorj.io/storj/private/migrate.(*Migration).Run:212\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).MigrateToLatest:387\n\tmain.cmdRun:98\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.cleanup.func1.4:393\n\tstorj.io/common/process.cleanup.func1:411\n\tgithub.com/spf13/cobra.(*Command).execute:983\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1115\n\tgithub.com/spf13/cobra.(*Command).Execute:1039\n\tstorj.io/common/process.ExecWithCustomOptions:112\n\tstorj.io/common/process.ExecWithCustomConfigAndLogger:77\n\tstorj.io/common/process.ExecWithCustomConfig:72\n\tstorj.io/common/process.Exec:62\n\tmain.(*service).Execute.func1:107\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78”, “errorVerbose”: “Error migrating tables for database on storagenode: migrate: v26: duplicate column name: trash\n\tstorj.io/storj/private/migrate.(*Migration).Run:212\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).MigrateToLatest:387\n\tmain.cmdRun:98\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.cleanup.func1.4:393\n\tstorj.io/common/process.cleanup.func1:411\n\tgithub.com/spf13/cobra.(*Command).execute:983\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1115\n\tgithub.com/spf13/cobra.(*Command).Execute:1039\n\tstorj.io/common/process.ExecWithCustomOptions:112\n\tstorj.io/common/process.ExecWithCustomConfigAndLogger:77\n\tstorj.io/common/process.ExecWithCustomConfig:72\n\tstorj.io/common/process.Exec:62\n\tmain.(*service).Execute.func1:107\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78\n\tmain.cmdRun:100\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.cleanup.func1.4:393\n\tstorj.io/common/process.cleanup.func1:411\n\tgithub.com/spf13/cobra.(*Command).execute:983\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1115\n\tgithub.com/spf13/cobra.(*Command).Execute:1039\n\tstorj.io/common/process.ExecWithCustomOptions:112\n\tstorj.io/common/process.ExecWithCustomConfigAndLogger:77\n\tstorj.io/common/process.ExecWithCustomConfig:72\n\tstorj.io/common/process.Exec:62\n\tmain.(*service).Execute.func1:107\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78”}

Then I deleted the existing piece_expiration.db
When the node started, a new piece_expiration.db was created. But the node still won’t start:

2024-04-15T14:01:49+03:00 ERROR failure during run {“error”: “Error migrating tables for database on storagenode: migrate: v26: no such table: piece_expirations\n\tstorj.io/storj/private/migrate.(*Migration).Run:212\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).MigrateToLatest:387\n\tmain.cmdRun:98\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.cleanup.func1.4:393\n\tstorj.io/common/process.cleanup.func1:411\n\tgithub.com/spf13/cobra.(*Command).execute:983\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1115\n\tgithub.com/spf13/cobra.(*Command).Execute:1039\n\tstorj.io/common/process.ExecWithCustomOptions:112\n\tstorj.io/common/process.ExecWithCustomConfigAndLogger:77\n\tstorj.io/common/process.ExecWithCustomConfig:72\n\tstorj.io/common/process.Exec:62\n\tmain.(*service).Execute.func1:107\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78”, “errorVerbose”: “Error migrating tables for database on storagenode: migrate: v26: no such table: piece_expirations\n\tstorj.io/storj/private/migrate.(*Migration).Run:212\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).MigrateToLatest:387\n\tmain.cmdRun:98\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.cleanup.func1.4:393\n\tstorj.io/common/process.cleanup.func1:411\n\tgithub.com/spf13/cobra.(*Command).execute:983\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1115\n\tgithub.com/spf13/cobra.(*Command).Execute:1039\n\tstorj.io/common/process.ExecWithCustomOptions:112\n\tstorj.io/common/process.ExecWithCustomConfigAndLogger:77\n\tstorj.io/common/process.ExecWithCustomConfig:72\n\tstorj.io/common/process.Exec:62\n\tmain.(*service).Execute.func1:107\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78\n\tmain.cmdRun:100\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.cleanup.func1.4:393\n\tstorj.io/common/process.cleanup.func1:411\n\tgithub.com/spf13/cobra.(*Command).execute:983\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1115\n\tgithub.com/spf13/cobra.(*Command).Execute:1039\n\tstorj.io/common/process.ExecWithCustomOptions:112\n\tstorj.io/common/process.ExecWithCustomConfigAndLogger:77\n\tstorj.io/common/process.ExecWithCustomConfig:72\n\tstorj.io/common/process.Exec:62\n\tmain.(*service).Execute.func1:107\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78”}

How to solve this problem? Or how to manually create this empty table?

2

Why did you restore from backup ? There is a method to fix corrupt database whose link can be found in the forum.

If you still have the corrupt db then use above link to fix it and replace the corrupt db with fixed db.

3

All this has been done, but it doesn’t help.

4

You always have the option of just deleting all the .db files (while the node is stopped). The stats in the GUI will be wrong until you start the next month… but payouts are unaffected.

1 Like
5

All this has been done, but it doesn’t help.

I did not restore from backup. I fix DB from your link

6

Better option is to stop the node, move dbs to a backup folder, start the node, let it create the new dbs, stop the node, restore all but the corrupted db.

2 Likes
7

You chose the wrong word “restore”. Restoring is meant as replacing corrupt db from backup.

8

sorry
english not my own language

9

It’s safety for node?

10

is it not the same as delete only piece_expiration.db?

11

It’s not my main language either but when you talk in computer terms certain words can’t be replaced or misused. When you know English is not your first language then something like :point_down:

makes you sound harsh. FYI I did not feel offended. I was in your situation when dbs get corrupt and nothing works, frustration takes over. Just be patient and calm when trying to find a solution.

2 Likes
12

I think so but that was the recommended method from other posts. Since your node is not working you may try it anyway. :crossed_fingers:

13

yes, you are right. sorry for this

14

Thank you!
It’s helped!

2 Likes
15

Yes, but the node would try to recreate all databases only in case when there is no any. But yes, the result will be the same - you will re-create only the corrupted database.