"storage_usage": expected schema does not match actual

[chrispyduck]

Greetings, all. How do I recover from the following error?

2023-09-04T14:41:53.194Z	INFO	db.migration	Database Version	{"version": 53}
Error: Error during preflight check for storagenode databases: preflight: database "storage_usage": expected schema does not match actual:   &dbschema.Schema{
- 	Tables: []*dbschema.Table{
- 		(
- 			s"""
- 			Name: storage_usage
- 			Columns:
- 				Name: at_rest_total
- 				Type: REAL
- 				Nullable: false
- 				Default: ""
- 				Reference: nil
- 				Name: interval_start
- 				Type: TIMESTAMP
- 				Nullable: false
- 				Default: ""
- 				Reference: nil
- 				Name: satellite_id
- 				Type: BLOB
- 				Nullable: false
- 			... // 5 elided lines
- 			s"""
- 		),
- 	},
+ 	Tables:  nil,
  	Indexes: nil,
  }

	storj.io/storj/storagenode/storagenodedb.(*DB).preflight:406
	storj.io/storj/storagenode/storagenodedb.(*DB).Preflight:353
	main.cmdRun:208
	storj.io/private/process.cleanup.func1.4:363
	storj.io/private/process.cleanup.func1:381
	github.com/spf13/cobra.(*Command).execute:852
	github.com/spf13/cobra.(*Command).ExecuteC:960

I’ve found a few posts about this and similar db errors and I’ve tried:

1. restarting the daemon - no effect
2. running sqlite3’s PRAGMA integrity_check; (sqlite version 3.31.1) on all databases and restarting - all are ok, service started with same error
3. removing storage_usage.db and restarting the daemon - a new file is created and the same error is displayed. restoring the original file afterward has no effect.
4. removing all .db files, starting the node, stopping the node, restoring the original .db files - same error

A few posts recommend recreating the schema but they’re dated and I’m guessing the schema has changed since then.

This error is causing my node to crash.

[Alexey]

Hello @chrispyduck,
Welcome to the forum!

You need to recreate this database following this instruction:

How to fix database: file is not a database error

Only deleting this database will not trigger its creation, you need to move all databases to a backup folder, then start the node. Then stop the node when all databases will be re-created, now you may move back (with replace) all databases from the backup but this database.

[chrispyduck]

I thought that’s what I did with this step:

@chrispyduck

removing all .db files, starting the node, stopping the node, restoring the original .db files - same error

I removed every .db file, not just storage_usage. The node started and created new, empty databases. I stopped it, restored the original .db files (I tried with and without storage_usage.db), and started storj. I tried this twice yesterday and 3 times today. The node still raises the same error. Am I missing something?

[arrogantrabbit]

More than one database can be corrupted.

[chrispyduck]

The error has only referenced storage_usage

[arrogantrabbit]

Then either you have accidentally copied the corrupted database on top, or whatever caused original corruption happened again.

Did you find out what caused the corruption and prevented it (likely power issues but could also be storage, ram, or something else.).

[Alexey]

Did you restore databases when the node is stopped?
Please note, the databases are located in the storage subfolder of the data location.
I cannot reproduce it.

1. Stop and remove the container
2. Rename storage_usage.db

mv /mnt/storj/storagenode/storage/storage_usage.db /mnt/storj/storagenode/storage/storage_usage.db.bak

3. Move all databases to the another subfolder

mkdir /mnt/storj/storagenode/dbs
mv /mnt/storj/storagenode/storage/*.db /mnt/storj/storagenode/dbs/

4. Run the node with all your parameters, wait until all databases are created and node updated to the actual version
5. Stop and remove the container
6. Restore databases

mv /mnt/storj/storagenode/dbs/* /mnt/storj/storagenode/storage/

7. Run the node with all your parameters

[chrispyduck]

Yes, the node was stopped when I moved the databases.

With the exception of renaming the broken storage_usage.db after moving the files, I believe my process is the same. The only thing I’m not sure about is this:

wait until all databases are created and node updated to the actual version

I waited until the dashboard would load, plus a few seconds; was that sufficient? All databases appeared to have been created and the node was logging messages that didn’t appear to be part of the startup routine.

[Alexey]

You may check logs to see when databases are created and the node is updated. If it said that there is no new version and db migration is finished, then you may stop and remove the container to move to the next step.
However, I would recommend to check your backup folder to do not have this database there, thus why I recommended to rename a database, this will make a backup copy and will prevent the accidental recover of the broken version after databases creation.