Database Errors in bandwidth.db and orders.db

Node Operators troubleshooting
1

Hello,

I’m running my node for about 2 Weeks now.
Today i accessed the Web Dashboard and it didn’t show any Statistics.

Using the docker dashboard i got the error message:
Error: bandwidthdb error: database disk image is malformed

So i followed this guide https://support.storj.io/hc/en-us/articles/360029309111-How-to-fix-a-database-disk-image-is-malformed-?_ga=2.152107193.1211118873.1596472549-1474832515.1595094879
It found some Errors in bandwith.db and i also found some in orders.db.

Then i checked both files again with this command: sqlite3 /path/to/storage/bandwidth.db “PRAGMA integrity_check;”
It returns ok for both of them, but when i start the Node im still not able to access the Web Dashboard and Docker Dashboard returns this:
2020-08-03T18:31:15.349Z INFO Configuration loaded {“Location”: “/app/config/config.yaml”}
2020-08-03T18:31:15.372Z INFO Identity loaded. {“Node ID”: “removedmynodeidhere”}
Error: context canceled

Note: The Error message comes from alone after a few seconds, i didn’t cancel it.

Any Help would be apprecciated :slight_smile:

2

Please look for errors in your logs and post them here.
https://documentation.storj.io/resources/faq/check-logs

1 Like
3

You should replace the /path/to/storage/bandwidth.db to the actual path to the bandwidth.db
If database doesn’t exist for this path, the sqlite3 will just create an empty one and of course, the integrity check will be OK :slight_smile:

1 Like
4

I let the Node run overnight and now I’m getting this Error which repeats again and again.

2020-08-04T04:56:30.059Z        INFO    Configuration loaded    {"Location": "/app/config/config.yaml"}
2020-08-04T04:56:30.072Z        INFO    Operator email  {"Address": "emailhere"}
2020-08-04T04:56:30.072Z        INFO    Operator wallet {"Address": "0x37662033f6443BD538154F3890f1f40Ed3471c20"}
2020-08-04T04:56:32.779Z        INFO    Telemetry enabled
2020-08-04T04:56:32.799Z        INFO    db.migration    Database Version        {"version": 43}
Error: Error during preflight check for storagenode databases: storage node preflight database error: bandwidth: expected schema does not match actual:   &dbschema.Schema{
        Tables: []*dbschema.Table{
                &{Name: "bandwidth_usage", Columns: []*dbschema.Column{&{Name: "action", Type: "INTEGER"}, &{Name: "amount", Type: "BIGINT"}, &{Name: "created_at", Type: "TIMESTAMP"}, &{Name: "satellite_id", Type: "BLOB"}}},
-               s"Name: bandwidth_usage_rollups\nColumns:\n\tName: action\n\tType: INTEGER\n\tNullable: false\n\tDefault: \"\"\n\tReference: nil\n\tName: amount\n\tType: BIGINT\n\tNullable: false\n\tDefault: \"\"\n\tReference: nil\n\tName: interval_start\n\tType: TIMESTAMP\n\tNullable: false\n\tDefault: \"\"\n\tReference: nil\n\tName: satellite_id\n\tType: BLOB\n\tNullable: false\n\tDefault: \"\"\n\tReference: nil\nPrimaryKey: action interval_start satellite_id\nUniques:\n\t",
        },
-       Indexes: []*dbschema.Index{
-               s`Index<Table: bandwidth_usage, Name: idx_bandwidth_usage_created, Columns: created_at, Unique: false, Partial: "">`,
-               s`Index<Table: bandwidth_usage, Name: idx_bandwidth_usage_satellite, Columns: satellite_id, Unique: false, Partial: "">`,
-       },
+       Indexes: nil,
  }

        storj.io/storj/storagenode/storagenodedb.(*DB).Preflight:346
        main.cmdRun:193
        storj.io/private/process.cleanup.func1.4:353
        storj.io/private/process.cleanup.func1:371
        github.com/spf13/cobra.(*Command).execute:840
        github.com/spf13/cobra.(*Command).ExecuteC:945
        github.com/spf13/cobra.(*Command).Execute:885
        storj.io/private/process.ExecWithCustomConfig:88
        storj.io/private/process.ExecCustomDebug:70
        main.main:323
        runtime.main:203

You should replace the /path/to/storage/bandwidth.db to the actual path to the bandwidth.db

I did replace it with my path, just wanted to let you know which command i used.

After correcting the Erros by following the Guide and Checking that sqlites PRAGMA integrity_check returns ok i copied it back to the storage folder and startet the Node again but i get same Errors as before (which is the long block postet above)

5

Could you show the result of

sqlite3 /path/to/storage/bandwidth.db .schema

?

6

This is the Output when run against the “repaired” bandwith.db

CREATE TABLE bandwidth_usage (
                                                satellite_id  BLOB    NOT NULL,
                                                action        INTEGER NOT NULL,
                                                amount        BIGINT  NOT NULL,
                                                created_at    TIMESTAMP NOT NULL
                                        );
CREATE TABLE bandwidth_usage_rollups (
                                                                                interval_start  TIMESTAMP NOT NULL,
                                                                                satellite_id    BLOB    NOT NULL,
                                                                                action          INTEGER NOT NULL,
                                                                                amount          BIGINT  NOT NULL,
                                                                                PRIMARY KEY ( interval_start, satellite_id, action )
                                                                        );
CREATE INDEX idx_bandwidth_usage_satellite ON bandwidth_usage(satellite_id);
CREATE INDEX idx_bandwidth_usage_created   ON bandwidth_usage(created_at);

Just checked, when run against the old, damaged bandwith.db its exactly the same Output.

7

Yes, the schema won’t be changed when you run that repair. It just fixes issues with the actual data inside the tables. I just compared your schema with my db and it’s an exact match. My node isn’t throwing any schema errors. Are you running v1.9.5 yet? My node is.

8

Yes. I made the Update when the Dashboard didn’t show any Statistics. I thought it maybe had to do with me runing the older Version.

9

Then I have no idea how your node is refusing the same schema that my node is perfectly fine with.

You’re sure you’re still seeing this exact error regarding bandwidth.db? Some of the errors look pretty similar.

10

Yes i just compared them with the Notepad++ compare function to be sure and its exactly the same :frowning:

11

My Node is Online now and the Dashboard is reachable. I don’t know why its suddenly working again because I only stopped and started it a few times.

However my uptime is still on 100% and i didn’t even fail 1 audit, so it seems like my Node was back Online after fixing the dbs and only the Dashboard didn’t work :open_mouth:

Thank you for your help :grin: :+1: