Update to v0.33.4 causes error: Error during preflight check for storagenode databases

I updated my storagenode to v0.34.3 and my node fails to start throwing the below error. I found a couple of other threads about similar issues, but I got lost in the amount of suggestions, checks and manual changes. So thought it better to log a new item to chek up on it. Any suggestions what to do with the below?

2020-02-28T18:51:46.046Z INFO db.migration Database Version {“version”: 31}

Error: Error during preflight check for storagenode databases: storage node preflight database error: reputation: expected schema does not match actual: &dbschema.Schema{

  Tables: []*dbschema.Table{


  	&{Name: "reputation", Columns: []*dbschema.Column{&{Name: "audit_reputation_alpha", Type: "REAL"}, &{Name: "audit_reputation_beta", Type: "REAL"}, &{Name: "audit_reputation_score", Type: "REAL"}, &{Name: "audit_success_count", Type: "INTEGER"}, &{Name: "audit_total_count", Type: "INTEGER"}, &{Name: "disqualified", Type: "TIMESTAMP", IsNullable: true}, &{Name: "satellite_id", Type: "BLOB"}, &{Name: "updated_at", Type: "TIMESTAMP"}, &{Name: "uptime_reputation_alpha", Type: "REAL"}, &{Name: "uptime_reputation_beta", Type: "REAL"}, &{Name: "uptime_reputation_score", Type: "REAL"}, &{Name: "uptime_success_count", Type: "INTEGER"}, &{Name: "uptime_total_count", Type: "INTEGER"}}, PrimaryKey: []string{"satellite_id"}},

FWIW: All checks return OK

bash-4.3$ ls *.db | while read LINE

do
echo $LINE
sqlite3 $LINE “PRAGMA integrity_check”
done

bandwidth.db
ok

info.db
ok

notifications.db
ok

orders.db
ok

piece_expiration.db
ok

pieceinfo.db
ok

piece_spaced_used.db
ok

piecestore.db
ok

reputation.db
ok

satellites.db
ok

storage_usage.db
ok

used_serial.db
ok
bash-4.3$

Please, stop your storagenode and fix the problem:

sqlite3 reputation.db
drop table test_table;
.exit

Hello,

I tried (with stopped storage node), but the test_table does not exist:

bash-4.3$ sqlite3 reputation.db
SQLite version 3.10.2 2016-01-20 15:27:19
Enter “.help” for usage hints.
sqlite> drop table test_table;
Error: no such table: test_table
sqlite> .exit
bash-4.3$

I also tried to run the earnings.py script earlier and found another simlar error:

sqlite3.OperationalError: no such table: su.storage_usage

Looks like a proper borked system for some reason. Any remedy steps I could still try?

Make sure that you opened this database from the storage location.
If preflight check found this table it’s should be there.

The preflight check could remove this table though.
Try to start your storagenode and look into logs for the next error

lets start that 34.3 not yet relised.

I looked at the repeating errors that come by in the logs. It is rotating, I took out personal stuff and some other items, but below is the gist of it. Seems to be related to “reputation” and “storage_usage” databases.

2020-02-29T17:28:19.733Z INFO Configuration loaded from: /app/config/config.yaml

2020-02-29T17:28:21.255Z INFO version running on version v0.33.4
2020-02-29T17:28:21.892Z INFO db.migration Database Version {“version”: 31}
Error: Error during preflight check for storagenode databases: storage node preflight database error: reputation: expected schema does not match actual: &dbschema.Schema{

  • Tables: []*dbschema.Table{
    
  • 	&{
    
  • 		Name: "reputation",
    
  • 		Columns: []*dbschema.Column{
    
  • 			&{Name: "audit_reputation_alpha", Type: "REAL"},
    
  • 			&{Name: "audit_reputation_beta", Type: "REAL"},
    
  • 			&{Name: "audit_reputation_score", Type: "REAL"},
    
  • 			&{Name: "audit_success_count", Type: "INTEGER"},
    
  • 			&{Name: "audit_total_count", Type: "INTEGER"},
    
  • 			&{Name: "disqualified", Type: "TIMESTAMP", IsNullable: true},
    
  • 			&{Name: "satellite_id", Type: "BLOB"},
    
  • 			&{Name: "updated_at", Type: "TIMESTAMP"},
    
  • 			&{Name: "uptime_reputation_alpha", Type: "REAL"},
    
  • 			&{Name: "uptime_reputation_beta", Type: "REAL"},
    
  • 			&{Name: "uptime_reputation_score", Type: "REAL"},
    
  • 			&{Name: "uptime_success_count", Type: "INTEGER"},
    
  • 			&{Name: "uptime_total_count", Type: "INTEGER"},
    
  • 		},
    
  • 		PrimaryKey: []string{"satellite_id"},
    
  • 	},
    
  • },
    

2020-02-29T17:28:50.173Z INFO Configuration loaded from: /app/config/config.yaml

2020-02-29T17:28:51.902Z INFO version running on version v0.33.4
2020-02-29T17:28:51.916Z INFO db.migration Database Version {“version”: 31}
Error: Error during preflight check for storagenode databases: storage node preflight database error: storage_usage: expected schema does not match actual: &dbschema.Schema{

  • Tables: []*dbschema.Table{
    
  • 	&{
    
  • 		Name: "storage_usage",
    
  • 		Columns: []*dbschema.Column{
    
  • 			&{Name: "at_rest_total", Type: "REAL"},
    
  • 			&{Name: "interval_start", Type: "TIMESTAMP"},
    
  • 			&{Name: "satellite_id", Type: "BLOB"},
    
  • 		},
    
  • 		PrimaryKey: []string{"interval_start", "satellite_id"},
    
  • 	},
    
  • },
    

2020-02-29T17:29:05.086Z INFO Configuration loaded from: /app/config/config.yaml

2020-02-29T17:29:06.534Z INFO version running on version v0.33.4
2020-02-29T17:29:06.550Z INFO db.migration Database Version {“version”: 31}
Error: Error during preflight check for storagenode databases: storage node preflight database error: reputation: expected schema does not match actual: &dbschema.Schema{

2020-02-29T17:29:22.573Z INFO Configuration loaded from: /app/config/config.yaml

2020-02-29T17:29:24.553Z INFO version running on version v0.33.4
2020-02-29T17:29:24.568Z INFO db.migration Database Version {“version”: 31}
Error: Error during preflight check for storagenode databases: storage node preflight database error: satellites: expected schema does not match actual: &dbschema.Schema{
Tables: *dbschema.Table{
&{Name: “satellite_exit_progress”, Columns: *dbschema.Column{&{Name: “bytes_deleted”, Type: “INTEGER”}, &{Name: “completion_receipt”, Type: “BLOB”, IsNullable: true}, &{Name: “finished_at”, Type: “TIMESTAMP”, IsNullable: true}, &{Name: “initiated_at”, Type: “TIMESTAMP”, IsNullable: true}, &{Name: “satellite_id”, Type: “BLOB”}, &{Name: “starting_disk_usage”, Type: “INTEGER”}}, PrimaryKey: string{“satellite_id”}},
&{Name: “satellites”, Columns: *dbschema.Column{&{Name: “added_at”, Type: “TIMESTAMP”}, &{Name: “node_id”, Type: “BLOB”}, &{Name: “status”, Type: “INTEGER”}}, PrimaryKey: string{“node_id”}},

  • 	&{
    
  • 		Name: "test_table",
    
  • 		Columns: []*dbschema.Column{
    
  • 			&{Name: "id", Type: "int"},
    
  • 			&{Name: "name", Type: "varchar(30)", IsNullable: true},
    
  • 		},
    
  • 		PrimaryKey: []string{"id"},
    
  • 	},
    },
    Indexes: nil,
    

    }

    storj.io/storj/storagenode/storagenodedb.(*DB).Preflight:301

Yes, I stand corrected, it’s 33.4 indeed. Will update title.

Stop the storagenode.
Make sure, that you open database from the proper location:

sqlite3 satellites.db
drop table test_table;
.exit
sqlite3 storage_usage.db
CREATE TABLE storage_usage (
                                                satellite_id BLOB NOT NULL,
                                                at_rest_total REAL NOT NUll,
                                                interval_start TIMESTAMP NOT NULL,
                                                PRIMARY KEY (satellite_id, interval_start)
                                        );
.exit
sqlite3 reputation.db
CREATE TABLE reputation (
                                                satellite_id BLOB NOT NULL,
                                                uptime_success_count INTEGER NOT NULL,
                                                uptime_total_count INTEGER NOT NULL,
                                                uptime_reputation_alpha REAL NOT NULL,
                                                uptime_reputation_beta REAL NOT NULL,
                                                uptime_reputation_score REAL NOT NULL,
                                                audit_success_count INTEGER NOT NULL,
                                                audit_total_count INTEGER NOT NULL,
                                                audit_reputation_alpha REAL NOT NULL,
                                                audit_reputation_beta REAL NOT NULL,
                                                audit_reputation_score REAL NOT NULL,
                                                disqualified TIMESTAMP,
                                                updated_at TIMESTAMP NOT NULL,
                                                PRIMARY KEY (satellite_id)
                                        );
.exit

You have too many errors with databases. How is your HDD connected to the PC?
If it’s a network connected drive, then it will not work. Only local connected drives are supported.
The only working network protocol is iSCSI, but the latency will lead to lose the race for pieces. And with any network problem your node can be disqualified.

1 Like

Thank you Alexey, I am back online with those fixes.

I am running this node from a Synology NAS, from it’s internal storage with the storagenode in a docker container. Have been doing that for the old v2 and since early last year when v3 became available. All without major issues, until this week when I did a docker pull. Only thing I can think of is that I used the “alpha” tag in stead of the “beta” as that was in my notes.

1 Like

My fixes recreated lost databases. That is why I asked.

Sure, thanks again sir!