Node stuck restarting after v0.31.9 update

[S0litiare]

The docker node updated to v0.31.9 an hour ago since then the node just keeps restarting.

Only thing in the log-file is this repeatedly :

> 2020-01-30T10:43:58.080Z	INFO	Configuration loaded from: /app/config/config.yaml
> 2020-01-30T10:43:58.081Z	DEBUG	Found deprecated flag. Migrating value <removed>:28967 from kademlia.external-address to contact.external-address
> 2020-01-30T10:43:58.081Z	DEBUG	Found deprecated flag. Migrating value <removed> from kademlia.operator.wallet to operator.wallet
> 2020-01-30T10:43:58.081Z	DEBUG	Found deprecated flag. Migrating value <removed> from kademlia.operator.email to operator.email
> 2020-01-30T10:43:58.081Z	DEBUG	debug server listening on 127.0.0.1:34803
> 2020-01-30T10:43:58.097Z	INFO	Operator email: <removed>
> 2020-01-30T10:43:58.097Z	INFO	operator wallet: <removed>
> 2020-01-30T10:43:58.129Z	DEBUG	Binary Version: v0.31.9 with CommitHash d627034bb003d02d9219c169b7568856283badaf, built at 2020-01-30 01:34:57 +0000 UTC as Release true
> 2020-01-30T10:43:58.598Z	DEBUG	version	allowed minimum version from control server is: v0.30.0
> 2020-01-30T10:43:58.598Z	INFO	version	running on version v0.31.9
> 2020-01-30T10:43:58.599Z	DEBUG	telemetry	Initialized batcher with id = "19ENSdVFbhYHXS3Sz4gbfBy9vA9cF2QT3HUgqb3pxQVvYZb9gV"
> 2020-01-30T10:43:58.609Z	INFO	db.migration	Database Version	{"version": 29}
> 2020-01-30T10:43:58.612Z	DEBUG	Unrecoverable error	{"error": "Error during preflight check for storagenode databases: storage node preflight database error: used_serial: expected schema does not match actual:   &dbschema.Schema{\n  \tTables: []*dbschema.Table{&{Name: \"used_serial_\", Columns: []*dbschema.Column{&{Name: \"expiration\", Type: \"TIMESTAMP\"}, &{Name: \"satellite_id\", Type: \"BLOB\"}, &{Name: \"serial_number\", Type: \"BLOB\"}}}},\n  \tIndexes: []*dbschema.Index{\n  \t\t&{Name: \"idx_used_serial_\", Table: \"used_serial_\", Columns: []string{\"expiration\"}},\n- \t\t&{\n- \t\t\tName:    \"pk_used_serial_\",\n- \t\t\tTable:   \"used_serial_\",\n- \t\t\tColumns: []string{\"satellite_id\", \"serial_number\"},\n- \t\t},\n  \t},\n  }\n\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).Preflight:317\n\tmain.cmdRun:196\n\tstorj.io/storj/pkg/process.cleanup.func1.2:299\n\tstorj.io/storj/pkg/process.cleanup.func1:317\n\tgithub.com/spf13/cobra.(*Command).execute:826\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:914\n\tgithub.com/spf13/cobra.(*Command).Execute:864\n\tstorj.io/storj/pkg/process.ExecWithCustomConfig:79\n\tstorj.io/storj/pkg/process.Exec:61\n\tmain.main:326\n\truntime.main:203", "errorVerbose": "Error during preflight check for storagenode databases: storage node preflight database error: used_serial: expected schema does not match actual:   &dbschema.Schema{\n  \tTables: []*dbschema.Table{&{Name: \"used_serial_\", Columns: []*dbschema.Column{&{Name: \"expiration\", Type: \"TIMESTAMP\"}, &{Name: \"satellite_id\", Type: \"BLOB\"}, &{Name: \"serial_number\", Type: \"BLOB\"}}}},\n  \tIndexes: []*dbschema.Index{\n  \t\t&{Name: \"idx_used_serial_\", Table: \"used_serial_\", Columns: []string{\"expiration\"}},\n- \t\t&{\n- \t\t\tName:    \"pk_used_serial_\",\n- \t\t\tTable:   \"used_serial_\",\n- \t\t\tColumns: []string{\"satellite_id\", \"serial_number\"},\n- \t\t},\n  \t},\n  }\n\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).Preflight:317\n\tmain.cmdRun:196\n\tstorj.io/storj/pkg/process.cleanup.func1.2:299\n\tstorj.io/storj/pkg/process.cleanup.func1:317\n\tgithub.com/spf13/cobra.(*Command).execute:826\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:914\n\tgithub.com/spf13/cobra.(*Command).Execute:864\n\tstorj.io/storj/pkg/process.ExecWithCustomConfig:79\n\tstorj.io/storj/pkg/process.Exec:61\n\tmain.main:326\n\truntime.main:203\n\tmain.cmdRun:198\n\tstorj.io/storj/pkg/process.cleanup.func1.2:299\n\tstorj.io/storj/pkg/process.cleanup.func1:317\n\tgithub.com/spf13/cobra.(*Command).execute:826\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:914\n\tgithub.com/spf13/cobra.(*Command).Execute:864\n\tstorj.io/storj/pkg/process.ExecWithCustomConfig:79\n\tstorj.io/storj/pkg/process.Exec:61\n\tmain.main:326\n\truntime.main:203"}

I’ve done the usual integrity check of the databases and everything seems ok.
Anyone got any ideas?

[littleskunk]

Did you modify the database at some point? The schema is wrong.

[S0litiare]

Only thing was a “PRAGMA integrity_check” on the databases a month or so ago which fixed some errors.

[littleskunk]

  	Tables: []*dbschema.Table{&{Name: \"used_serial_\", Columns: []*dbschema.Column{&{Name: \"expiration\", Type: \"TIMESTAMP\"}, &{Name: \"satellite_id\", Type: \"BLOB\"}, &{Name: \"serial_number\", Type: \"BLOB\"}}}},
  	Indexes: []*dbschema.Index{
  		&{Name: \"idx_used_serial_\", Table: \"used_serial_\", Columns: []string{\"expiration\"}},
- 		&{
- 			Name:    \"pk_used_serial_\",
- 			Table:   \"used_serial_\",
- 			Columns: []string{\"satellite_id\", \"serial_number\"},
- 		},
  	},

Give me a moment to compare that against the original DB

[S0litiare]

We’re doing Troubleshooting in this thread it effects another user as well:

Bandwidth Error After Upgrade to 31.9

I’m getting the errors in 2 databases "orders.db and “used_serial.db” here are the schema for the 2 effected db’s in my node: used_serial.db: CREATE TABLE used_serial_ ( satellite_id BLOB NOT NULL, serial_number BLOB NOT NULL, expiration TIMESTAMP NOT NULL ); CREATE INDEX idx_used_serial_ ON used_serial_(expiration); orders.db: CREATE TABLE unsent_order ( satellite_id BLOB NOT NULL, serial_number BLOB NOT NULL, order_limit_serialized BLOB …

[ifraixedes]

Thanks @S0litiare

Let’s follow in that one because is the same issue despite that they are in different DBs across different users.