Update to v0.33.4 causes error: Error during preflight check for storagenode databases

therealmko · February 28, 2020, 7:34pm

I updated my storagenode to v0.34.3 and my node fails to start throwing the below error. I found a couple of other threads about similar issues, but I got lost in the amount of suggestions, checks and manual changes. So thought it better to log a new item to chek up on it. Any suggestions what to do with the below?

2020-02-28T18:51:46.046Z INFO db.migration Database Version {“version”: 31}

Error: Error during preflight check for storagenode databases: storage node preflight database error: reputation: expected schema does not match actual: &dbschema.Schema{
  Tables: []*dbschema.Table{


  	&{Name: "reputation", Columns: []*dbschema.Column{&{Name: "audit_reputation_alpha", Type: "REAL"}, &{Name: "audit_reputation_beta", Type: "REAL"}, &{Name: "audit_reputation_score", Type: "REAL"}, &{Name: "audit_success_count", Type: "INTEGER"}, &{Name: "audit_total_count", Type: "INTEGER"}, &{Name: "disqualified", Type: "TIMESTAMP", IsNullable: true}, &{Name: "satellite_id", Type: "BLOB"}, &{Name: "updated_at", Type: "TIMESTAMP"}, &{Name: "uptime_reputation_alpha", Type: "REAL"}, &{Name: "uptime_reputation_beta", Type: "REAL"}, &{Name: "uptime_reputation_score", Type: "REAL"}, &{Name: "uptime_success_count", Type: "INTEGER"}, &{Name: "uptime_total_count", Type: "INTEGER"}}, PrimaryKey: []string{"satellite_id"}},
	&{
		Name: "test_table",
		Columns: []*dbschema.Column{
			&{Name: "id", Type: "int"},
			&{Name: "name", Type: "varchar(30)", IsNullable: true},
		},
		PrimaryKey: []string{"id"},
	},


},


Indexes: nil,
}

storj.io/storj/storagenode/storagenodedb.(*DB).Preflight:301

main.cmdRun:198

storj.io/storj/pkg/process.cleanup.func1.2:307

storj.io/storj/pkg/process.cleanup.func1:325

github.com/spf13/cobra.(*Command).execute:826

github.com/spf13/cobra.(*Command).ExecuteC:914

github.com/spf13/cobra.(*Command).Execute:864

storj.io/storj/pkg/process.ExecWithCustomConfig:84

storj.io/storj/pkg/process.ExecCustomDebug:66

main.main:328

runtime.main:203

therealmko · February 28, 2020, 7:48pm

FWIW: All checks return OK

bash-4.3$ ls *.db | while read LINE

do
echo $LINE
sqlite3 $LINE “PRAGMA integrity_check”
done

bandwidth.db
ok

info.db
ok

notifications.db
ok

orders.db
ok

piece_expiration.db
ok

pieceinfo.db
ok

piece_spaced_used.db
ok

piecestore.db
ok

reputation.db
ok

satellites.db
ok

storage_usage.db
ok

used_serial.db
ok
bash-4.3$

Alexey · February 29, 2020, 4:31pm

Please, stop your storagenode and fix the problem:

sqlite3 reputation.db
drop table test_table;
.exit

therealmko · February 29, 2020, 4:36pm

Hello,

I tried (with stopped storage node), but the test_table does not exist:

bash-4.3$ sqlite3 reputation.db
SQLite version 3.10.2 2016-01-20 15:27:19
Enter “.help” for usage hints.
sqlite> drop table test_table;
Error: no such table: test_table
sqlite> .exit
bash-4.3$

I also tried to run the earnings.py script earlier and found another simlar error:

sqlite3.OperationalError: no such table: su.storage_usage

Looks like a proper borked system for some reason. Any remedy steps I could still try?

Alexey · February 29, 2020, 4:50pm

Make sure that you opened this database from the storage location.
If preflight check found this table it’s should be there.

The preflight check could remove this table though.
Try to start your storagenode and look into logs for the next error

Vadim · February 29, 2020, 4:55pm

lets start that 34.3 not yet relised.

therealmko · February 29, 2020, 5:38pm

I looked at the repeating errors that come by in the logs. It is rotating, I took out personal stuff and some other items, but below is the gist of it. Seems to be related to “reputation” and “storage_usage” databases.

2020-02-29T17:28:19.733Z INFO Configuration loaded from: /app/config/config.yaml
…
2020-02-29T17:28:21.255Z INFO version running on version v0.33.4
2020-02-29T17:28:21.892Z INFO db.migration Database Version {“version”: 31}
Error: Error during preflight check for storagenode databases: storage node preflight database error: reputation: expected schema does not match actual: &dbschema.Schema{
Tables: []*dbschema.Table{
	&{
		Name: "reputation",
		Columns: []*dbschema.Column{
			&{Name: "audit_reputation_alpha", Type: "REAL"},
			&{Name: "audit_reputation_beta", Type: "REAL"},
			&{Name: "audit_reputation_score", Type: "REAL"},
			&{Name: "audit_success_count", Type: "INTEGER"},
			&{Name: "audit_total_count", Type: "INTEGER"},
			&{Name: "disqualified", Type: "TIMESTAMP", IsNullable: true},
			&{Name: "satellite_id", Type: "BLOB"},
			&{Name: "updated_at", Type: "TIMESTAMP"},
			&{Name: "uptime_reputation_alpha", Type: "REAL"},
			&{Name: "uptime_reputation_beta", Type: "REAL"},
			&{Name: "uptime_reputation_score", Type: "REAL"},
			&{Name: "uptime_success_count", Type: "INTEGER"},
			&{Name: "uptime_total_count", Type: "INTEGER"},
		},
		PrimaryKey: []string{"satellite_id"},
	},
},
Tables:  nil,
Indexes: nil,
}

storj.io/storj/storagenode/storagenodedb.(*DB).Preflight:301
…
2020-02-29T17:28:50.173Z INFO Configuration loaded from: /app/config/config.yaml
…
2020-02-29T17:28:51.902Z INFO version running on version v0.33.4
2020-02-29T17:28:51.916Z INFO db.migration Database Version {“version”: 31}
Error: Error during preflight check for storagenode databases: storage node preflight database error: storage_usage: expected schema does not match actual: &dbschema.Schema{
Tables: []*dbschema.Table{
	&{
		Name: "storage_usage",
		Columns: []*dbschema.Column{
			&{Name: "at_rest_total", Type: "REAL"},
			&{Name: "interval_start", Type: "TIMESTAMP"},
			&{Name: "satellite_id", Type: "BLOB"},
		},
		PrimaryKey: []string{"interval_start", "satellite_id"},
	},
},
Tables:  nil,
Indexes: nil,
}

storj.io/storj/storagenode/storagenodedb.(*DB).Preflight:301
…
2020-02-29T17:29:05.086Z INFO Configuration loaded from: /app/config/config.yaml
…
2020-02-29T17:29:06.534Z INFO version running on version v0.33.4
2020-02-29T17:29:06.550Z INFO db.migration Database Version {“version”: 31}
Error: Error during preflight check for storagenode databases: storage node preflight database error: reputation: expected schema does not match actual: &dbschema.Schema{
…

2020-02-29T17:29:22.573Z INFO Configuration loaded from: /app/config/config.yaml
…
2020-02-29T17:29:24.553Z INFO version running on version v0.33.4
2020-02-29T17:29:24.568Z INFO db.migration Database Version {“version”: 31}
Error: Error during preflight check for storagenode databases: storage node preflight database error: satellites: expected schema does not match actual: &dbschema.Schema{
Tables: *dbschema.Table{
&{Name: “satellite_exit_progress”, Columns: *dbschema.Column{&{Name: “bytes_deleted”, Type: “INTEGER”}, &{Name: “completion_receipt”, Type: “BLOB”, IsNullable: true}, &{Name: “finished_at”, Type: “TIMESTAMP”, IsNullable: true}, &{Name: “initiated_at”, Type: “TIMESTAMP”, IsNullable: true}, &{Name: “satellite_id”, Type: “BLOB”}, &{Name: “starting_disk_usage”, Type: “INTEGER”}}, PrimaryKey: string{“satellite_id”}},
&{Name: “satellites”, Columns: *dbschema.Column{&{Name: “added_at”, Type: “TIMESTAMP”}, &{Name: “node_id”, Type: “BLOB”}, &{Name: “status”, Type: “INTEGER”}}, PrimaryKey: string{“node_id”}},
	&{
		Name: "test_table",
		Columns: []*dbschema.Column{
			&{Name: "id", Type: "int"},
			&{Name: "name", Type: "varchar(30)", IsNullable: true},
		},
		PrimaryKey: []string{"id"},
	},
},
Indexes: nil,
}

storj.io/storj/storagenode/storagenodedb.(*DB).Preflight:301
…

therealmko · February 29, 2020, 5:39pm

Yes, I stand corrected, it’s 33.4 indeed. Will update title.

Alexey · February 29, 2020, 6:13pm

Stop the storagenode.
Make sure, that you open database from the proper location:

sqlite3 satellites.db
drop table test_table;
.exit

sqlite3 storage_usage.db
CREATE TABLE storage_usage (
                                                satellite_id BLOB NOT NULL,
                                                at_rest_total REAL NOT NUll,
                                                interval_start TIMESTAMP NOT NULL,
                                                PRIMARY KEY (satellite_id, interval_start)
                                        );
.exit

sqlite3 reputation.db
CREATE TABLE reputation (
                                                satellite_id BLOB NOT NULL,
                                                uptime_success_count INTEGER NOT NULL,
                                                uptime_total_count INTEGER NOT NULL,
                                                uptime_reputation_alpha REAL NOT NULL,
                                                uptime_reputation_beta REAL NOT NULL,
                                                uptime_reputation_score REAL NOT NULL,
                                                audit_success_count INTEGER NOT NULL,
                                                audit_total_count INTEGER NOT NULL,
                                                audit_reputation_alpha REAL NOT NULL,
                                                audit_reputation_beta REAL NOT NULL,
                                                audit_reputation_score REAL NOT NULL,
                                                disqualified TIMESTAMP,
                                                updated_at TIMESTAMP NOT NULL,
                                                PRIMARY KEY (satellite_id)
                                        );
.exit

You have too many errors with databases. How is your HDD connected to the PC?
If it’s a network connected drive, then it will not work. Only local connected drives are supported.
The only working network protocol is iSCSI, but the latency will lead to lose the race for pieces. And with any network problem your node can be disqualified.

therealmko · February 29, 2020, 6:26pm

Thank you Alexey, I am back online with those fixes.

I am running this node from a Synology NAS, from it’s internal storage with the storagenode in a docker container. Have been doing that for the old v2 and since early last year when v3 became available. All without major issues, until this week when I did a docker pull. Only thing I can think of is that I used the “alpha” tag in stead of the “beta” as that was in my notes.

Alexey · February 29, 2020, 6:32pm

My fixes recreated lost databases. That is why I asked.

therealmko · February 29, 2020, 6:43pm

Sure, thanks again sir!