Error during preflight check for storagenode databases: storage node preflight database error: orders: expected schema does not match actual

I just tried to upgrade my STORJ setup from Docker to Windows on the same computer. After removing the docker STORJ and installing the Windows setup, the service always shows not started. I start it manually but it stops on its own.

Any ideas on what to do or should I just go back to Docker because that is what seems to work for me?

Event viewer doesn’t give much help… image

check your time, that it is synked, from this version it check db integrity and time sync, and it hapens on all platforms.

What do the logs say? It’s storagenode.log in… where ever you installed it.

I’m afraid the log doesn’t help much…

2020-01-31T12:39:59.310-0500 INFO Configuration loaded from: C:\Program Files\Storj\Storage Node\config.yaml
2020-01-31T12:39:59.336-0500 INFO Operator email: mail@mail.com
2020-01-31T12:39:59.336-0500 INFO operator wallet: XXX
2020-01-31T12:39:59.606-0500 INFO version running on version v0.31.12
2020-01-31T12:40:00.122-0500 INFO db.migration Database Version {“version”: 31}
2020-01-31T12:40:12.207-0500 INFO Configuration loaded from: C:\Program Files\Storj\Storage Node\config.yaml
2020-01-31T12:40:12.223-0500 INFO Operator email: mail@mail.com
2020-01-31T12:40:12.223-0500 INFO operator wallet: XXX
2020-01-31T12:40:12.391-0500 INFO version running on version v0.31.12
2020-01-31T12:40:12.407-0500 INFO db.migration Database Version {“version”: 31}
2020-01-31T12:40:19.757-0500 INFO Configuration loaded from: C:\Program Files\Storj\Storage Node\config.yaml
2020-01-31T12:40:19.773-0500 INFO Operator email: mail@mail.com
2020-01-31T12:40:19.773-0500 INFO operator wallet: XXX
2020-01-31T12:40:19.943-0500 INFO version running on version v0.31.12
2020-01-31T12:40:19.961-0500 INFO db.migration Database Version {“version”: 31}

Please, check your storage location. When migrate to the Windows GUI you should select the storage folder in your storage location.
Also, please, enable the debug level of logging (replace the info to debug), save the configuration file and restart the storagenode service.
Then check logs.

I can confirm the path in the config file:
# path to store data in
storage.path: D:\storj\storage\

The error that appears in the log is that the Schema is incorrect in my DB. How do I update it to the new version? Or does it need a repair?

This is the log file when I changed it to debug.
2020-02-01T01:20:25.547-0500 INFO Configuration loaded from: C:\Program Files\Storj\Storage Node\config.yaml
2020-02-01T01:20:25.551-0500 DEBUG debug server listening on 127.0.0.1:52062
2020-02-01T01:20:25.580-0500 INFO Operator email: mail@mail.com
2020-02-01T01:20:25.581-0500 INFO operator wallet: xxx
2020-02-01T01:20:25.610-0500 DEBUG Binary Version: v0.31.12 with CommitHash 8214de0e7ee6c55846e6f6a33e1309fc27c183c6, built at 2020-01-30 13:27:58 -0500 EST as Release true
2020-02-01T01:20:25.858-0500 DEBUG version allowed minimum version from control server is: v0.30.0
2020-02-01T01:20:25.858-0500 INFO version running on version v0.31.12
2020-02-01T01:20:25.859-0500 DEBUG telemetry Initialized batcher with id = “128oPJnQ5ZL6gZpeSjyC1yDUrBG7qxPYYMTGYzk2DnX8Dczq2Y4”
2020-02-01T01:20:54.515-0500 INFO db.migration Database Version {“version”: 31}
2020-02-01T01:20:57.415-0500 DEBUG Unrecoverable error {“error”: “Error during preflight check for storagenode databases: storage node preflight database error: orders: expected schema does not match actual: &dbschema.Schema{\n \tTables: *dbschema.Table{&{Name: “order_archive_”, Columns: *dbschema.Column{&{Name: “archived_at”, Type: “TIMESTAMP”}, &{Name: “order_limit_serialized”, Type: “BLOB”}, &{Name: “order_serialized”, Type: “BLOB”}, &{Name: “satellite_id”, Type: “BLOB”}, &{Name: “serial_number”, Type: “BLOB”}, &{Name: “status”, Type: “INTEGER”}, &{Name: “uplink_cert_id”, Type: “INTEGER”, Reference: &dbschema.Reference{Table: “certificate”, Column: “cert_id”}}}}, &{Name: “unsent_order”, Columns: *dbschema.Column{&{Name: “order_limit_expiration”, Type: “TIMESTAMP”}, &{Name: “order_limit_serialized”, Type: “BLOB”}, &{Name: “order_serialized”, Type: “BLOB”}, &{Name: “satellite_id”, Type: “BLOB”}, &{Name: “serial_number”, Type: “BLOB”}, &{Name: “uplink_cert_id”, Type: “INTEGER”, Reference: &dbschema.Reference{Table: “certificate”, Column: “cert_id”}}}}},\n \tIndexes: *dbschema.Index{\n \t\t&{Name: “idx_order_archived_at”, Table: “order_archive_”, Columns: string{“archived_at”}},\n- \t\t&{\n- \t\t\tName: “idx_orders”,\n- \t\t\tTable: “unsent_order”,\n- \t\t\tColumns: string{“satellite_id”, “serial_number”},\n- \t\t},\n \t},\n }\n\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).Preflight:317\n\tmain.cmdRun:196\n\tstorj.io/storj/pkg/process.cleanup.func1.2:299\n\tstorj.io/storj/pkg/process.cleanup.func1:317\n\tgithub.com/spf13/cobra.(*Command).execute:826\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:914\n\tgithub.com/spf13/cobra.(*Command).Execute:864\n\tstorj.io/storj/pkg/process.ExecWithCustomConfig:79\n\tstorj.io/storj/pkg/process.Exec:61\n\tmain.(*service).Execute.func1:66\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”, “errorVerbose”: “Error during preflight check for storagenode databases: storage node preflight database error: orders: expected schema does not match actual: &dbschema.Schema{\n \tTables: *dbschema.Table{&{Name: “order_archive_”, Columns: *dbschema.Column{&{Name: “archived_at”, Type: “TIMESTAMP”}, &{Name: “order_limit_serialized”, Type: “BLOB”}, &{Name: “order_serialized”, Type: “BLOB”}, &{Name: “satellite_id”, Type: “BLOB”}, &{Name: “serial_number”, Type: “BLOB”}, &{Name: “status”, Type: “INTEGER”}, &{Name: “uplink_cert_id”, Type: “INTEGER”, Reference: &dbschema.Reference{Table: “certificate”, Column: “cert_id”}}}}, &{Name: “unsent_order”, Columns: *dbschema.Column{&{Name: “order_limit_expiration”, Type: “TIMESTAMP”}, &{Name: “order_limit_serialized”, Type: “BLOB”}, &{Name: “order_serialized”, Type: “BLOB”}, &{Name: “satellite_id”, Type: “BLOB”}, &{Name: “serial_number”, Type: “BLOB”}, &{Name: “uplink_cert_id”, Type: “INTEGER”, Reference: &dbschema.Reference{Table: “certificate”, Column: “cert_id”}}}}},\n \tIndexes: *dbschema.Index{\n \t\t&{Name: “idx_order_archived_at”, Table: “order_archive_”, Columns: string{“archived_at”}},\n- \t\t&{\n- \t\t\tName: “idx_orders”,\n- \t\t\tTable: “unsent_order”,\n- \t\t\tColumns: string{“satellite_id”, “serial_number”},\n- \t\t},\n \t},\n }\n\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).Preflight:317\n\tmain.cmdRun:196\n\tstorj.io/storj/pkg/process.cleanup.func1.2:299\n\tstorj.io/storj/pkg/process.cleanup.func1:317\n\tgithub.com/spf13/cobra.(*Command).execute:826\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:914\n\tgithub.com/spf13/cobra.(*Command).Execute:864\n\tstorj.io/storj/pkg/process.ExecWithCustomConfig:79\n\tstorj.io/storj/pkg/process.Exec:61\n\tmain.(*service).Execute.func1:66\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57\n\tmain.cmdRun:198\n\tstorj.io/storj/pkg/process.cleanup.func1.2:299\n\tstorj.io/storj/pkg/process.cleanup.func1:317\n\tgithub.com/spf13/cobra.(*Command).execute:826\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:914\n\tgithub.com/spf13/cobra.(*Command).Execute:864\n\tstorj.io/storj/pkg/process.ExecWithCustomConfig:79\n\tstorj.io/storj/pkg/process.Exec:61\n\tmain.(*service).Execute.func1:66\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}

Your database orders.db have not migrated schema, perhaps it was restored from the backup, or was corrupted.

Please, stop the storagenode service either from the Services applet or from the elevated Powershell

Stop-Service storagenode

Then rename the orders.db to orders.db.bak, then start the storagenode service back, it will create an empty orders.db:

Start-Service storagenode

You will lose the last unsent orders with this action, but your node should survive.
Please, check that the error is gone: https://documentation.storj.io/resources/faq/check-logs

I killed orders and this is the new error:

2020-02-01T17:55:28.180-0500 DEBUG Unrecoverable error {“error”: “Error creating tables for master database on storagenode: migrate: no such table: main.order_archive_\n\tstorj.io/storj/private/migrate.(*Migration).Run:182\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).CreateTables:291\n\tmain.cmdRun:186\n\tstorj.io/storj/pkg/process.cleanup.func1.2:299\n\tstorj.io/storj/pkg/process.cleanup.func1:317\n\tgithub.com/spf13/cobra.(*Command).execute:826\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:914\n\tgithub.com/spf13/cobra.(*Command).Execute:864\n\tstorj.io/storj/pkg/process.ExecWithCustomConfig:79\n\tstorj.io/storj/pkg/process.Exec:61\n\tmain.(*service).Execute.func1:66\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”, “errorVerbose”: “Error creating tables for master database on storagenode: migrate: no such table: main.order_archive_\n\tstorj.io/storj/private/migrate.(*Migration).Run:182\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).CreateTables:291\n\tmain.cmdRun:186\n\tstorj.io/storj/pkg/process.cleanup.func1.2:299\n\tstorj.io/storj/pkg/process.cleanup.func1:317\n\tgithub.com/spf13/cobra.(*Command).execute:826\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:914\n\tgithub.com/spf13/cobra.(*Command).Execute:864\n\tstorj.io/storj/pkg/process.ExecWithCustomConfig:79\n\tstorj.io/storj/pkg/process.Exec:61\n\tmain.(*service).Execute.func1:66\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57\n\tmain.cmdRun:188\n\tstorj.io/storj/pkg/process.cleanup.func1.2:299\n\tstorj.io/storj/pkg/process.cleanup.func1:317\n\tgithub.com/spf13/cobra.(*Command).execute:826\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:914\n\tgithub.com/spf13/cobra.(*Command).Execute:864\n\tstorj.io/storj/pkg/process.ExecWithCustomConfig:79\n\tstorj.io/storj/pkg/process.Exec:61\n\tmain.(*service).Execute.func1:66\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}

I tried moving all the DB files out and everything works fine. I guess that would be a last resort as I don’t want to lose everything.

According to the log, you’re missing an index on the unsent_orders table. Another user reported the same issue, but on a different table. Something might be wrong with this preflight check system. We’ll look into it

I think I have something like this problem too :

Have tried repairing databases and now is the 3rd day the node is offline… don’t know what more I can try before I jump off this beta.

2020-02-03T09:39:33.975Z INFO version running on version v0.31.12
2020-02-03T09:39:34.033Z INFO db.migration Database Version {“version”: 31}
Error: Error during preflight check for storagenode databases: storage node preflight database error: orders: expected schema does not match actual: &dbschema.Schema{
Tables: *dbschema.Table{&{Name: “order_archive_”, Columns: *dbschema.Column{&{Name: “archived_at”, Type: “TIMESTAMP”}, &{Name: “order_limit_serialized”, Type: “BLOB”}, &{Name: “order_serialized”, Type: “BLOB”}, &{Name: “satellite_id”, Type: “BLOB”}, &{Name: “serial_number”, Type: “BLOB”}, &{Name: “status”, Type: “INTEGER”}, &{Name: “uplink_cert_id”, Type: “INTEGER”, Reference: &dbschema.Reference{Table: “certificate”, Column: “cert_id”}}}}, &{Name: “unsent_order”, Columns: *dbschema.Column{&{Name: “order_limit_expiration”, Type: “TIMESTAMP”}, &{Name: “order_limit_serialized”, Type: “BLOB”}, &{Name: “order_serialized”, Type: “BLOB”}, &{Name: “satellite_id”, Type: “BLOB”}, &{Name: “serial_number”, Type: “BLOB”}, &{Name: “uplink_cert_id”, Type: “INTEGER”, Reference: &dbschema.Reference{Table: “certificate”, Column: “cert_id”}}}}},
Indexes: *dbschema.Index{
&{Name: “idx_order_archived_at”, Table: “order_archive_”, Columns: string{“archived_at”}},

  •           &{
    
  •                   Name:    "idx_orders",
    
  •                   Table:   "unsent_order",
    
  •                   Columns: []string{"satellite_id", "serial_number"},
    
  •           },
      },
    

    }

      storj.io/storj/storagenode/storagenodedb.(*DB).Preflight:317
      main.cmdRun:196
      storj.io/storj/pkg/process.cleanup.func1.2:299
      storj.io/storj/pkg/process.cleanup.func1:317
      github.com/spf13/cobra.(*Command).execute:826
      github.com/spf13/cobra.(*Command).ExecuteC:914
      github.com/spf13/cobra.(*Command).Execute:864
      storj.io/storj/pkg/process.ExecWithCustomConfig:79
      storj.io/storj/pkg/process.Exec:61
      main.main:326
      runtime.main:203

Try to rename the orders.db in the storage folder and restart the storagenode

You mean rename so it creates a new one?

when I renamed it , i got this error :

2020-02-04T20:52:29.535Z INFO db.migration.27 Add index archived_at to ordersDB
Error: Error creating tables for master database on storagenode: migrate: no such table: main.order_archive_
storj.io/storj/private/migrate.(*Migration).Run:182
storj.io/storj/storagenode/storagenodedb.(*DB).CreateTables:291
main.cmdRun:186
storj.io/storj/pkg/process.cleanup.func1.2:299
storj.io/storj/pkg/process.cleanup.func1:317
github.com/spf13/cobra.(*Command).execute:826
github.com/spf13/cobra.(*Command).ExecuteC:914
github.com/spf13/cobra.(*Command).Execute:864
storj.io/storj/pkg/process.ExecWithCustomConfig:79
storj.io/storj/pkg/process.Exec:61
main.main:326
runtime.main:203

Yes, exactly. But you will have a backup in any case

Have you restarted the storagenode after rename?

Yes, twice now. There’s something strange with the updated version…

We will try to fix it. Stop the storagenode.
Restore the orders.db from the backup (keep the backup for a while).
Open the orders.db with a sqlite3 (you can install it or use the docker version).

sqlite3 orders.db

Apply this SQL script:

drop index idx_orders;
CREATE INDEX idx_orders ON unsent_order(satellite_id, serial_number);
.exit

Then try to start the storagenode

2 Likes

Finally Its back up :grin:

Thank you Alexey

@Alexey @cameron

Should I wipe my DB and start from scratch or wait for you guys to try debug the preflight?