istie
March 14, 2020, 12:43pm
1
Node keeps restarting. storagenode logs says:
2020-03-14T12:40:43.194Z INFO version running on version v0.34.6
2020-03-14T12:40:43.247Z INFO db.migration.27 Add index archived_at to ordersDB
Error: Error creating tables for master database on storagenode: migrate: no such table: main.order_archive_
storj.io/storj/private/dbutil/txutil.WithTx:47
storj.io/storj/private/migrate.(*Migration ).Run:170
storj.io/storj/storagenode/storagenodedb.(*DB ).CreateTables:275
main.cmdRun:188
storj.io/storj/pkg/process.cleanup.func1.2:312
storj.io/storj/pkg/process.cleanup.func1:330
github.com/spf13/cobra.(*Command ).execute:826
github.com/spf13/cobra.(*Command ).ExecuteC:914
github.com/spf13/cobra.(*Command ).Execute:864
storj.io/storj/pkg/process.ExecWithCustomConfig:84
storj.io/storj/pkg/process.ExecCustomDebug:66
main.main:328
runtime.main:203
Welcome to the forum @istie !
How is your HDD connected ? Is it a Windows or Linux node ?
istie
March 14, 2020, 3:25pm
3
Thx for your reply!
It’s a linux node (raspberry buster) with an external hdd of 2TB (external powered) mounted via fstab. Node is online since sept 2019
Can you show your docker run command and remove any personal details from it ?
istie
March 14, 2020, 4:13pm
5
docker run -d --restart unless-stopped -p 28967:28967 -p 14002:14002 -e WALLET=“0x****************************************” -e EMAIL="myemail@mail.com " -e ADDRESS=“my.address:28967” -e BANDWIDTH=“5TB” -e STORAGE=“1TB” --mount type=bind,source="/home/pi/.local/share/storj/identity/storagenode",destination=/app/identity --mount type=bind,source="/mnt/STORJ/_STORJ-DATA",destination=/app/config --name storagenode storjlabs/storagenode:beta
Alexey
March 14, 2020, 4:24pm
6
Try to fix the database:
Stop the storagenode
Make a backup of orders.db
database (for example by copying to orders.db.bak
).
Execute either with a local installed sqlite3 or with a docker container with sqlite3 :
sqlite3 orders.db
CREATE TABLE unsent_order (
satellite_id BLOB NOT NULL,
serial_number BLOB NOT NULL,
order_limit_serialized BLOB NOT NULL, -- serialized pb.OrderLimit
order_serialized BLOB NOT NULL, -- serialized pb.Order
order_limit_expiration TIMESTAMP NOT NULL, -- when is the deadline for sending it
uplink_cert_id INTEGER NOT NULL,
FOREIGN KEY(uplink_cert_id) REFERENCES certificate(cert_id)
);
CREATE TABLE order_archive_ (
satellite_id BLOB NOT NULL,
serial_number BLOB NOT NULL,
order_limit_serialized BLOB NOT NULL,
order_serialized BLOB NOT NULL,
uplink_cert_id INTEGER NOT NULL,
status INTEGER NOT NULL,
archived_at TIMESTAMP NOT NULL,
FOREIGN KEY(uplink_cert_id) REFERENCES certificate(cert_id)
);
CREATE UNIQUE INDEX idx_orders ON unsent_order(satellite_id, serial_number);
CREATE TABLE versions (version int, commited_at text);
CREATE INDEX idx_order_archived_at ON order_archive_(archived_at);
.exit
istie
March 14, 2020, 4:48pm
7
Executed commands with sqlite3 but still node keeps restarting
logs say:
2020-03-14T16:43:45.884Z INFO version running on version v0.34.6
2020-03-14T16:43:46.377Z INFO db.migration.27 Add index archived_at to ordersDB
Error: Error creating tables for master database on storagenode: migrate: index idx_order_archived_at already exists
storj.io/storj/private/dbutil/txutil.WithTx:47
storj.io/storj/private/migrate.(*Migration ).Run:170
storj.io/storj/storagenode/storagenodedb.(*DB ).CreateTables:275
main.cmdRun:188
storj.io/storj/pkg/process.cleanup.func1.2:312
storj.io/storj/pkg/process.cleanup.func1:330
github.com/spf13/cobra.(*Command ).execute:826
github.com/spf13/cobra.(*Command ).ExecuteC:914
github.com/spf13/cobra.(*Command ).Execute:864
storj.io/storj/pkg/process.ExecWithCustomConfig:84
storj.io/storj/pkg/process.ExecCustomDebug:66
main.main:328
runtime.main:203
istie
March 14, 2020, 4:58pm
8
Is there no tool or script which can repair a broken database, Or build it correct again?
Or am I asking for something really stupid?
Alexey
March 14, 2020, 4:59pm
9
Stop the storagenode
execute the script:
sqlite3 orders.db
drop index idx_order_archived_at;
.exit
Start the storagenode
Yes, you can try to remove the database and lose the statistic and some unsent orders (they will not be paid), the storagenode should recreate a database. But it may not work. So, better to use a provided fix.
istie
March 14, 2020, 7:28pm
10
Thx for your answers, tried your suggestion but still not ok.
2020-03-14T19:22:58.405Z INFO version running on version v0.34.6
2020-03-14T19:22:58.474Z INFO db.migration Database Version {“version”: 31}
Error: Error during preflight check for storagenode databases: storage node preflight database error: piece_expiration: expected schema does not match actual: &dbschema.Schema{
Tables: *dbschema.Table{
&{Name: “piece_expirations”, Columns: *dbschema.Column{&{Name: “deletion_failed_at”, Type: “TIMESTAMP”, IsNullable: true}, &{Name: “piece_expiration”, Type: “TIMESTAMP”}, &{Name: “piece_id”, Type: “BLOB”}, &{Name: “satellite_id”, Type: “BLOB”}, &{Name: “trash”, Type: “INTEGER”}}, PrimaryKey: string{“piece_id”, “satellite_id”}},
&{
Name: "test_table",
Columns: []*dbschema.Column{
&{Name: "id", Type: "int"},
&{Name: "name", Type: "varchar(30)", IsNullable: true},
},
PrimaryKey: []string{"id"},
},
},
Indexes: *dbschema.Index{&{Name: “idx_piece_expirations_deletion_failed_at”, Table: “piece_expirations”, Columns: string{“deletion_failed_at”}}, &{Name: “idx_piece_expirations_piece_expiration”, Table: “piece_expirations”, Columns: string{“piece_expiration”}}, &{Name: “idx_piece_expirations_trashed”, Table: “piece_expirations”, Columns: string{“satellite_id”, “trash”}, Partial: “trash = 1”}},
}
storj.io/storj/storagenode/storagenodedb.(*DB ).Preflight:301
main.cmdRun:198
storj.io/storj/pkg/process.cleanup.func1.2:312
storj.io/storj/pkg/process.cleanup.func1:330
github.com/spf13/cobra.(*Command ).execute:826
github.com/spf13/cobra.(*Command ).ExecuteC:914
github.com/spf13/cobra.(*Command ).Execute:864
storj.io/storj/pkg/process.ExecWithCustomConfig:84
storj.io/storj/pkg/process.ExecCustomDebug:66
main.main:328
runtime.main:203
Looks like more databases are corrupt isn’t it.
You say: “better use a provided fix” so deleting database is not an option…?
But If I delete a database, which ones should be saved and which ones I can safely delete?
Alexey
March 14, 2020, 9:38pm
11
Please, stop the node and execute:
sqlite3 piece_expiration.db
drop test_table;
.exit
And try to start the node
istie
March 16, 2020, 11:17pm
12
Sorry it’s still restarting.
And with the command: “drop test_table;”
I get:
sqlite> drop test_table;
Error: near “test_table”: syntax error
logs:
2020-03-16T23:15:39.097Z INFO version running on version v0.34.6
2020-03-16T23:15:39.176Z INFO db.migration Database Version {“version”: 31}
2020-03-16T23:15:40.761Z INFO Got a signal from the OS: “terminated”
Error: Error during preflight check for storagenode databases: storage node preflight database error: piece_spaced_used: expected schema does not match actual: &dbschema.Schema{
Tables: *dbschema.Table{
&{Name: “piece_space_used”, Columns: *dbschema.Column{&{Name: “content_size”, Type: “INTEGER”}, &{Name: “satellite_id”, Type: “BLOB”, IsNullable: true}, &{Name: “total”, Type: “INTEGER”}}},
&{
Name: "test_table",
Columns: []*dbschema.Column{
&{Name: "id", Type: "int"},
&{Name: "name", Type: "varchar(30)", IsNullable: true},
},
PrimaryKey: []string{"id"},
},
},
Indexes: *dbschema.Index{&{Name: “idx_piece_space_used_satellite_id”, Table: “piece_space_used”, Columns: string{“satellite_id”}}},
}
storj.io/storj/storagenode/storagenodedb.(*DB ).Preflight:301
main.cmdRun:198
storj.io/storj/pkg/process.cleanup.func1.2:312
storj.io/storj/pkg/process.cleanup.func1:330
github.com/spf13/cobra.(*Command ).execute:826
github.com/spf13/cobra.(*Command ).ExecuteC:914
github.com/spf13/cobra.(*Command ).Execute:864
storj.io/storj/pkg/process.ExecWithCustomConfig:84
storj.io/storj/pkg/process.ExecCustomDebug:66
main.main:328
runtime.main:203
Alexey
March 17, 2020, 6:24pm
13
Stop the storagenode
execute
sqlite3 piece_space_used.db
drop table test_table;
.exit
istie
March 17, 2020, 8:24pm
14
Thanks for all your replies but is still not working
command:
sqlite> drop table test_table;
Error: no such table: test_table
sqlite> .exit
logs: (much longer this time…??):
2020-03-17T20:14:51.479Z INFO Configuration loaded from: /app/config/config.yaml
2020-03-17T20:14:51.484Z INFO Operator email: ***************
2020-03-17T20:14:51.485Z INFO operator wallet: **************************************
2020-03-17T20:14:52.658Z INFO version running on version v0.34.6
2020-03-17T20:14:52.766Z INFO db.migration Database Version {“version”: 31}
Error: Error during preflight check for storagenode databases: storage node preflight database error: piece_spaced_used: expected schema does not match actual: &dbschema.Schema{
Tables: *dbschema.Table{
&{Name: “piece_space_used”, Columns: *dbschema.Column{&{Name: “content_size”, Type: “INTEGER”}, &{Name: “satellite_id”, Type: “BLOB”, IsNullable: true}, &{Name: “total”, Type: “INTEGER”}}},
&{
Name: "test_table",
Columns: []*dbschema.Column{
&{Name: "id", Type: "int"},
&{Name: "name", Type: "varchar(30)", IsNullable: true},
},
PrimaryKey: []string{"id"},
},
},
Indexes: *dbschema.Index{&{Name: “idx_piece_space_used_satellite_id”, Table: “piece_space_used”, Columns: string{“satellite_id”}}},
}
storj.io/storj/storagenode/storagenodedb.(*DB).Preflight:301
main.cmdRun:198
storj.io/storj/pkg/process.cleanup.func1.2:312
storj.io/storj/pkg/process.cleanup.func1:330
github.com/spf13/cobra.(*Command).execute:826
github.com/spf13/cobra.(*Command).ExecuteC:914
github.com/spf13/cobra.(*Command).Execute:864
storj.io/storj/pkg/process.ExecWithCustomConfig:84
storj.io/storj/pkg/process.ExecCustomDebug:66
main.main:328
runtime.main:203
2020-03-17T20:14:58.224Z INFO Configuration loaded from: /app/config/config.yaml
2020-03-17T20:14:58.229Z INFO Operator email: ***************
2020-03-17T20:14:58.229Z INFO operator wallet: **************************************
2020-03-17T20:14:59.876Z INFO version running on version v0.34.6
2020-03-17T20:14:59.989Z INFO db.migration Database Version {“version”: 31}
Error: Error during preflight check for storagenode databases: storage node preflight database error: storage_usage: expected schema does not match actual: &dbschema.Schema{
Tables: *dbschema.Table{
&{Name: “storage_usage”, Columns: *dbschema.Column{&{Name: “at_rest_total”, Type: “REAL”}, &{Name: “interval_start”, Type: “TIMESTAMP”}, &{Name: “satellite_id”, Type: “BLOB”}}, PrimaryKey: string{“interval_start”, “satellite_id”}},
&{
Name: "test_table",
Columns: []*dbschema.Column{
&{Name: "id", Type: "int"},
&{Name: "name", Type: "varchar(30)", IsNullable: true},
},
PrimaryKey: []string{"id"},
},
},
Indexes: nil,
}
storj.io/storj/storagenode/storagenodedb.(*DB).Preflight:301
main.cmdRun:198
storj.io/storj/pkg/process.cleanup.func1.2:312
storj.io/storj/pkg/process.cleanup.func1:330
github.com/spf13/cobra.(*Command).execute:826
github.com/spf13/cobra.(*Command).ExecuteC:914
github.com/spf13/cobra.(*Command).Execute:864
storj.io/storj/pkg/process.ExecWithCustomConfig:84
storj.io/storj/pkg/process.ExecCustomDebug:66
main.main:328
runtime.main:203
2020-03-17T20:15:03.316Z INFO Configuration loaded from: /app/config/config.yaml
2020-03-17T20:15:03.321Z INFO Operator email: ****************
2020-03-17T20:15:03.322Z INFO operator wallet: **************************************
2020-03-17T20:15:05.264Z INFO version running on version v0.34.6
2020-03-17T20:15:05.372Z INFO db.migration Database Version {“version”: 31}
Error: Error during preflight check for storagenode databases: storage node preflight database error: used_serial: expected schema does not match actual: &dbschema.Schema{
Tables: *dbschema.Table{
&{
Name: "used_serial_",
Columns: []*dbschema.Column{
&{Name: "expiration", Type: "TIMESTAMP"},
&{Name: "satellite_id", Type: "BLOB"},
&{Name: "serial_number", Type: "BLOB"},
},
},
},
Indexes: *dbschema.Index{
&{Name: "idx_used_serial_", Table: "used_serial_", Columns: []string{"expiration"}},
&{
Name: "pk_used_serial_",
Table: "used_serial_",
Columns: []string{"satellite_id", "serial_number"},
},
},
Tables: *dbschema.Table{
&{
Name: "used_serial_",
Columns: []*dbschema.Column{
&{Name: "expiration", Type: "TIMESTAMP"},
&{Name: "satellite_id", Type: "BLOB"},
&{Name: "serial_number", Type: "BLOB"},
},
},
},
Indexes: *dbschema.Index{
&{Name: "idx_used_serial_", Table: "used_serial_", Columns: []string{"expiration"}},
&{
Name: "pk_used_serial_",
Table: "used_serial_",
Columns: []string{"satellite_id", "serial_number"},
},
},
Indexes: nil,
}
storj.io/storj/storagenode/storagenodedb.(*DB).Preflight:301
main.cmdRun:198
storj.io/storj/pkg/process.cleanup.func1.2:312
storj.io/storj/pkg/process.cleanup.func1:330
github.com/spf13/cobra.(*Command).execute:826
github.com/spf13/cobra.(*Command).ExecuteC:914
github.com/spf13/cobra.(*Command).Execute:864
storj.io/storj/pkg/process.ExecWithCustomConfig:84
storj.io/storj/pkg/process.ExecCustomDebug:66
main.main:328
runtime.main:203
2020-03-17T20:15:16.144Z INFO Configuration loaded from: /app/config/config.yaml
2020-03-17T20:15:16.149Z INFO Operator email: *************
2020-03-17T20:15:16.149Z INFO operator wallet: **************************************
2020-03-17T20:15:17.695Z INFO version running on version v0.34.6
2020-03-17T20:15:17.813Z INFO db.migration Database Version {“version”: 31}
Error: Error during preflight check for storagenode databases: storage node preflight database error: piece_spaced_used: expected schema does not match actual: &dbschema.Schema{
Tables: *dbschema.Table{
&{Name: “piece_space_used”, Columns: *dbschema.Column{&{Name: “content_size”, Type: “INTEGER”}, &{Name: “satellite_id”, Type: “BLOB”, IsNullable: true}, &{Name: “total”, Type: “INTEGER”}}},
&{
Name: "test_table",
Columns: []*dbschema.Column{
&{Name: "id", Type: "int"},
&{Name: "name", Type: "varchar(30)", IsNullable: true},
},
PrimaryKey: []string{"id"},
},
},
Indexes: *dbschema.Index{&{Name: “idx_piece_space_used_satellite_id”, Table: “piece_space_used”, Columns: string{“satellite_id”}}},
}
storj.io/storj/storagenode/storagenodedb.(*DB).Preflight:301
main.cmdRun:198
storj.io/storj/pkg/process.cleanup.func1.2:312
storj.io/storj/pkg/process.cleanup.func1:330
github.com/spf13/cobra.(*Command).execute:826
github.com/spf13/cobra.(*Command).ExecuteC:914
github.com/spf13/cobra.(*Command).Execute:864
storj.io/storj/pkg/process.ExecWithCustomConfig:84
storj.io/storj/pkg/process.ExecCustomDebug:66
main.main:328
runtime.main:203
2020-03-17T20:15:23.102Z INFO Configuration loaded from: /app/config/config.yaml
2020-03-17T20:15:23.108Z INFO Operator email: ************
2020-03-17T20:15:23.109Z INFO operator wallet: **************************************
2020-03-17T20:15:23.679Z INFO version running on version v0.34.6
2020-03-17T20:15:23.763Z INFO db.migration Database Version {“version”: 31}
Error: Error during preflight check for storagenode databases: storage node preflight database error: piece_expiration: expected schema does not match actual: &dbschema.Schema{
Tables: *dbschema.Table{
&{Name: “piece_expirations”, Columns: *dbschema.Column{&{Name: “deletion_failed_at”, Type: “TIMESTAMP”, IsNullable: true}, &{Name: “piece_expiration”, Type: “TIMESTAMP”}, &{Name: “piece_id”, Type: “BLOB”}, &{Name: “satellite_id”, Type: “BLOB”}, &{Name: “trash”, Type: “INTEGER”}}, PrimaryKey: string{“piece_id”, “satellite_id”}},
&{
Name: "test_table",
Columns: []*dbschema.Column{
&{Name: "id", Type: "int"},
&{Name: "name", Type: "varchar(30)", IsNullable: true},
},
PrimaryKey: []string{"id"},
},
},
Indexes: *dbschema.Index{&{Name: “idx_piece_expirations_deletion_failed_at”, Table: “piece_expirations”, Columns: string{“deletion_failed_at”}}, &{Name: “idx_piece_expirations_piece_expiration”, Table: “piece_expirations”, Columns: string{“piece_expiration”}}, &{Name: “idx_piece_expirations_trashed”, Table: “piece_expirations”, Columns: string{“satellite_id”, “trash”}, Partial: “trash = 1”}},
}
storj.io/storj/storagenode/storagenodedb.(*DB).Preflight:301
main.cmdRun:198
storj.io/storj/pkg/process.cleanup.func1.2:312
storj.io/storj/pkg/process.cleanup.func1:330
github.com/spf13/cobra.(*Command).execute:826
github.com/spf13/cobra.(*Command).ExecuteC:914
github.com/spf13/cobra.(*Command).Execute:864
storj.io/storj/pkg/process.ExecWithCustomConfig:84
storj.io/storj/pkg/process.ExecCustomDebug:66
main.main:328
runtime.main:203
2020-03-17T20:15:31.476Z INFO Configuration loaded from: /app/config/config.yaml
2020-03-17T20:15:31.481Z INFO Operator email: **************
2020-03-17T20:15:31.481Z INFO operator wallet: **************************************
2020-03-17T20:15:32.957Z INFO version running on version v0.34.6
2020-03-17T20:15:33.086Z INFO db.migration Database Version {“version”: 31}
Error: Error during preflight check for storagenode databases: storage node preflight database error: info: expected schema does not match actual: &dbschema.Schema{
Tables: *dbschema.Table{
&{
Name: "test_table",
Columns: []*dbschema.Column{
&{Name: "id", Type: "int"},
&{Name: "name", Type: "varchar(30)", IsNullable: true},
},
PrimaryKey: []string{"id"},
},
},
Indexes: nil,
}
storj.io/storj/storagenode/storagenodedb.(*DB).Preflight:301
main.cmdRun:198
storj.io/storj/pkg/process.cleanup.func1.2:312
storj.io/storj/pkg/process.cleanup.func1:330
github.com/spf13/cobra.(*Command).execute:826
github.com/spf13/cobra.(*Command).ExecuteC:914
github.com/spf13/cobra.(*Command).Execute:864
storj.io/storj/pkg/process.ExecWithCustomConfig:84
storj.io/storj/pkg/process.ExecCustomDebug:66
main.main:328
runtime.main:203
2020-03-17T20:15:41.619Z INFO Configuration loaded from: /app/config/config.yaml
2020-03-17T20:15:41.624Z INFO Operator email: *************
2020-03-17T20:15:41.626Z INFO operator wallet: **************************************
2020-03-17T20:15:43.884Z INFO version running on version v0.34.6
2020-03-17T20:15:44.009Z INFO db.migration Database Version {“version”: 31}
Error: Error during preflight check for storagenode databases: storage node preflight database error: reputation: expected schema does not match actual: &dbschema.Schema{
Tables: *dbschema.Table{
&{Name: “reputation”, Columns: *dbschema.Column{&{Name: “audit_reputation_alpha”, Type: “REAL”}, &{Name: “audit_reputation_beta”, Type: “REAL”}, &{Name: “audit_reputation_score”, Type: “REAL”}, &{Name: “audit_success_count”, Type: “INTEGER”}, &{Name: “audit_total_count”, Type: “INTEGER”}, &{Name: “disqualified”, Type: “TIMESTAMP”, IsNullable: true}, &{Name: “satellite_id”, Type: “BLOB”}, &{Name: “updated_at”, Type: “TIMESTAMP”}, &{Name: “uptime_reputation_alpha”, Type: “REAL”}, &{Name: “uptime_reputation_beta”, Type: “REAL”}, &{Name: “uptime_reputation_score”, Type: “REAL”}, &{Name: “uptime_success_count”, Type: “INTEGER”}, &{Name: “uptime_total_count”, Type: “INTEGER”}}, PrimaryKey: string{“satellite_id”}},
&{
Name: "test_table",
Columns: []*dbschema.Column{
&{Name: "id", Type: "int"},
&{Name: "name", Type: "varchar(30)", IsNullable: true},
},
PrimaryKey: []string{"id"},
},
},
Indexes: nil,
}
storj.io/storj/storagenode/storagenodedb.(*DB).Preflight:301
main.cmdRun:198
storj.io/storj/pkg/process.cleanup.func1.2:312
storj.io/storj/pkg/process.cleanup.func1:330
github.com/spf13/cobra.(*Command).execute:826
github.com/spf13/cobra.(*Command).ExecuteC:914
github.com/spf13/cobra.(*Command).Execute:864
storj.io/storj/pkg/process.ExecWithCustomConfig:84
storj.io/storj/pkg/process.ExecCustomDebug:66
main.main:328
runtime.main:203
2020-03-17T20:16:01.646Z INFO Configuration loaded from: /app/config/config.yaml
2020-03-17T20:16:01.650Z INFO Operator email: **************
2020-03-17T20:16:01.651Z INFO operator wallet: **************************************
2020-03-17T20:16:02.251Z INFO version running on version v0.34.6
2020-03-17T20:16:02.306Z INFO db.migration Database Version {“version”: 31}
Error: Error during preflight check for storagenode databases: storage node preflight database error: piece_expiration: expected schema does not match actual: &dbschema.Schema{
Tables: *dbschema.Table{
&{Name: “piece_expirations”, Columns: *dbschema.Column{&{Name: “deletion_failed_at”, Type: “TIMESTAMP”, IsNullable: true}, &{Name: “piece_expiration”, Type: “TIMESTAMP”}, &{Name: “piece_id”, Type: “BLOB”}, &{Name: “satellite_id”, Type: “BLOB”}, &{Name: “trash”, Type: “INTEGER”}}, PrimaryKey: string{“piece_id”, “satellite_id”}},
&{
Name: "test_table",
Columns: []*dbschema.Column{
&{Name: "id", Type: "int"},
&{Name: "name", Type: "varchar(30)", IsNullable: true},
},
PrimaryKey: []string{"id"},
},
},
Indexes: *dbschema.Index{&{Name: “idx_piece_expirations_deletion_failed_at”, Table: “piece_expirations”, Columns: string{“deletion_failed_at”}}, &{Name: “idx_piece_expirations_piece_expiration”, Table: “piece_expirations”, Columns: string{“piece_expiration”}}, &{Name: “idx_piece_expirations_trashed”, Table: “piece_expirations”, Columns: string{“satellite_id”, “trash”}, Partial: “trash = 1”}},
}
storj.io/storj/storagenode/storagenodedb.(*DB).Preflight:301
main.cmdRun:198
storj.io/storj/pkg/process.cleanup.func1.2:312
storj.io/storj/pkg/process.cleanup.func1:330
github.com/spf13/cobra.(*Command).execute:826
github.com/spf13/cobra.(*Command).ExecuteC:914
github.com/spf13/cobra.(*Command).Execute:864
storj.io/storj/pkg/process.ExecWithCustomConfig:84
storj.io/storj/pkg/process.ExecCustomDebug:66
main.main:328
runtime.main:203
2020-03-17T20:16:37.454Z INFO Configuration loaded from: /app/config/config.yaml
2020-03-17T20:16:37.461Z INFO Operator email: *************
2020-03-17T20:16:37.462Z INFO operator wallet: **************************************
2020-03-17T20:16:38.628Z INFO version running on version v0.34.6
2020-03-17T20:16:38.768Z INFO db.migration Database Version {“version”: 31}
Error: Error during preflight check for storagenode databases: storage node preflight database error: piece_spaced_used: expected schema does not match actual: &dbschema.Schema{
Tables: *dbschema.Table{
&{Name: “piece_space_used”, Columns: *dbschema.Column{&{Name: “content_size”, Type: “INTEGER”}, &{Name: “satellite_id”, Type: “BLOB”, IsNullable: true}, &{Name: “total”, Type: “INTEGER”}}},
&{
Name: "test_table",
Columns: []*dbschema.Column{
&{Name: "id", Type: "int"},
&{Name: "name", Type: "varchar(30)", IsNullable: true},
},
PrimaryKey: []string{"id"},
},
},
Indexes: *dbschema.Index{&{Name: “idx_piece_space_used_satellite_id”, Table: “piece_space_used”, Columns: string{“satellite_id”}}},
}
storj.io/storj/storagenode/storagenodedb.(*DB).Preflight:301
main.cmdRun:198
storj.io/storj/pkg/process.cleanup.func1.2:312
storj.io/storj/pkg/process.cleanup.func1:330
github.com/spf13/cobra.(*Command).execute:826
github.com/spf13/cobra.(*Command).ExecuteC:914
github.com/spf13/cobra.(*Command).Execute:864
storj.io/storj/pkg/process.ExecWithCustomConfig:84
storj.io/storj/pkg/process.ExecCustomDebug:66
main.main:328
runtime.main:203
Unfortunately this db file is misspelled in the installation. So the command should be.
sqlite3 piece_spaced_used.db
drop table test_table;
.exit
And actually a few more have the same issue.
sqlite3 storage_usage.db
drop table test_table;
.exit
sqlite3 piece_expiration.db
drop table test_table;
.exit
sqlite3 reputation.db
drop table test_table;
.exit
There is an issue with used_serial
as well, but I’m not entirely sure what.
I really wonder what happened to this node to have so many different database issues…
istie
March 17, 2020, 11:03pm
16
Again Thanks for all the help. The node had no problems from sept. 2019
I’ve been on vacation when the node went offline.
Mabybe a power outtage which corrupted my databases? I really do not know…
Is it easier to startup a new node?
Can I use the same identity (email and authorisation code)?
Can I take my reputaion with me?
Thx again!
Commands I tried this time on your request:
pi@gekkojoris:/mnt/STORJ/_STORJ-DATA/storage sqlite3 piece_spaced_used.db
SQLite version 3.27.2 2019-02-25 16:06:06
Enter ".help" for usage hints.
sqlite> drop table test_table;
sqlite> .exit
pi@gekkojoris:/mnt/STORJ/_STORJ-DATA/storage sqlite3 storage_usage.db
SQLite version 3.27.2 2019-02-25 16:06:06
Enter “.help” for usage hints.
sqlite> drop table test_table;
sqlite> .exit
pi@gekkojoris:/mnt/STORJ/_STORJ-DATA/storage sqlite3 piece_expiration.db
SQLite version 3.27.2 2019-02-25 16:06:06
Enter ".help" for usage hints.
sqlite> drop table test_table;
sqlite> .exit
pi@gekkojoris:/mnt/STORJ/_STORJ-DATA/storage sqlite3 reputation.db
SQLite version 3.27.2 2019-02-25 16:06:06
Enter “.help” for usage hints.
sqlite> drop table test_table;
sqlite> .exit
pi@gekkojoris:/mnt/STORJ/_STORJ-DATA/storage $
Status:
pi@gekkojoris:/mnt/STORJ/_STORJ-DATA/storage sudo docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
500c4f0a40c3 storjlabs/storagenode:beta "/entrypoint" 41 seconds ago Up 6 seconds 0.0.0.0:14002->14002/tcp, 0.0.0.0:28967->28967/tcp storagenode
pi@gekkojoris:/mnt/STORJ/_STORJ-DATA/storage sudo docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
2b0b2cc04e5e storjlabs/storagenode:beta “/entrypoint” 2 minutes ago Restarting (1) 42 seconds ago storagenode
pi@gekkojoris:/mnt/STORJ/_STORJ-DATA/storage $
Again lots news in logs:
2020-03-17T22:56:01.326Z INFO Configuration loaded from: /app/config/config.yaml
2020-03-17T22:56:01.331Z INFO Operator email: **************
2020-03-17T22:56:01.331Z INFO operator wallet: ***************************************
2020-03-17T22:56:01.896Z INFO version running on version v0.34.6
2020-03-17T22:56:01.962Z INFO db.migration Database Version {“version”: 31}
Error: Error during preflight check for storagenode databases: storage node preflight database error: info: expected schema does not match actual: &dbschema.Schema{
Tables: *dbschema.Table{
&{
Name: "test_table",
Columns: []*dbschema.Column{
&{Name: "id", Type: "int"},
&{Name: "name", Type: "varchar(30)", IsNullable: true},
},
PrimaryKey: []string{"id"},
},
},
Indexes: nil,
}
storj.io/storj/storagenode/storagenodedb.(*DB ).Preflight:301
main.cmdRun:198
storj.io/storj/pkg/process.cleanup.func1.2:312
storj.io/storj/pkg/process.cleanup.func1:330
github.com/spf13/cobra.(*Command ).execute:826
github.com/spf13/cobra.(*Command ).ExecuteC:914
github.com/spf13/cobra.(*Command ).Execute:864
storj.io/storj/pkg/process.ExecWithCustomConfig:84
storj.io/storj/pkg/process.ExecCustomDebug:66
main.main:328
runtime.main:203
2020-03-17T22:56:12.832Z INFO Configuration loaded from: /app/config/config.yaml
2020-03-17T22:56:12.837Z INFO Operator email: **************
2020-03-17T22:56:12.837Z INFO operator wallet: ***************************************
2020-03-17T22:56:14.367Z INFO version running on version v0.34.6
2020-03-17T22:56:14.460Z INFO db.migration Database Version {“version”: 31}
Error: Error during preflight check for storagenode databases: storage node preflight database error: used_serial: expected schema does not match actual: &dbschema.Schema{
Tables: *dbschema.Table{
&{
Name: "used_serial_",
Columns: []*dbschema.Column{
&{Name: "expiration", Type: "TIMESTAMP"},
&{Name: "satellite_id", Type: "BLOB"},
&{Name: "serial_number", Type: "BLOB"},
},
},
},
Indexes: *dbschema.Index{
&{Name: "idx_used_serial_", Table: "used_serial_", Columns: []string{"expiration"}},
&{
Name: "pk_used_serial_",
Table: "used_serial_",
Columns: []string{"satellite_id", "serial_number"},
},
},
Indexes: nil,
}
storj.io/storj/storagenode/storagenodedb.(*DB ).Preflight:301
main.cmdRun:198
storj.io/storj/pkg/process.cleanup.func1.2:312
storj.io/storj/pkg/process.cleanup.func1:330
github.com/spf13/cobra.(*Command ).execute:826
github.com/spf13/cobra.(*Command ).ExecuteC:914
github.com/spf13/cobra.(*Command ).Execute:864
storj.io/storj/pkg/process.ExecWithCustomConfig:84
storj.io/storj/pkg/process.ExecCustomDebug:66
main.main:328
runtime.main:203
2020-03-17T22:56:20.321Z INFO Configuration loaded from: /app/config/config.yaml
2020-03-17T22:56:20.325Z INFO Operator email: **************
2020-03-17T22:56:20.326Z INFO operator wallet: ***************************************
2020-03-17T22:56:21.812Z INFO version running on version v0.34.6
2020-03-17T22:56:21.915Z INFO db.migration Database Version {“version”: 31}
Error: Error during preflight check for storagenode databases: storage node preflight database error: satellites: expected schema does not match actual: &dbschema.Schema{
Tables: *dbschema.Table{
&{Name: “satellite_exit_progress”, Columns: *dbschema.Column{&{Name: “bytes_deleted”, Type: “INTEGER”}, &{Name: “completion_receipt”, Type: “BLOB”, IsNullable: true}, &{Name: “finished_at”, Type: “TIMESTAMP”, IsNullable: true}, &{Name: “initiated_at”, Type: “TIMESTAMP”, IsNullable: true}, &{Name: “satellite_id”, Type: “BLOB”}, &{Name: “starting_disk_usage”, Type: “INTEGER”}}, PrimaryKey: string{“satellite_id”}},
&{Name: “satellites”, Columns: *dbschema.Column{&{Name: “added_at”, Type: “TIMESTAMP”}, &{Name: “node_id”, Type: “BLOB”}, &{Name: “status”, Type: “INTEGER”}}, PrimaryKey: string{“node_id”}},
&{
Name: "test_table",
Columns: []*dbschema.Column{
&{Name: "id", Type: "int"},
&{Name: "name", Type: "varchar(30)", IsNullable: true},
},
PrimaryKey: []string{"id"},
},
},
Indexes: nil,
}
storj.io/storj/storagenode/storagenodedb.(*DB ).Preflight:301
main.cmdRun:198
storj.io/storj/pkg/process.cleanup.func1.2:312
storj.io/storj/pkg/process.cleanup.func1:330
github.com/spf13/cobra.(*Command ).execute:826
github.com/spf13/cobra.(*Command ).ExecuteC:914
github.com/spf13/cobra.(*Command ).Execute:864
storj.io/storj/pkg/process.ExecWithCustomConfig:84
storj.io/storj/pkg/process.ExecCustomDebug:66
main.main:328
runtime.main:203
2020-03-17T22:56:25.334Z INFO Configuration loaded from: /app/config/config.yaml
2020-03-17T22:56:25.341Z INFO Operator email: **************
2020-03-17T22:56:25.342Z INFO operator wallet: ***************************************
2020-03-17T22:56:27.062Z INFO version running on version v0.34.6
2020-03-17T22:56:27.165Z INFO db.migration Database Version {“version”: 31}
Error: Error during preflight check for storagenode databases: storage node preflight database error: satellites: expected schema does not match actual: &dbschema.Schema{
Tables: *dbschema.Table{
&{Name: “satellite_exit_progress”, Columns: *dbschema.Column{&{Name: “bytes_deleted”, Type: “INTEGER”}, &{Name: “completion_receipt”, Type: “BLOB”, IsNullable: true}, &{Name: “finished_at”, Type: “TIMESTAMP”, IsNullable: true}, &{Name: “initiated_at”, Type: “TIMESTAMP”, IsNullable: true}, &{Name: “satellite_id”, Type: “BLOB”}, &{Name: “starting_disk_usage”, Type: “INTEGER”}}, PrimaryKey: string{“satellite_id”}},
&{Name: “satellites”, Columns: *dbschema.Column{&{Name: “added_at”, Type: “TIMESTAMP”}, &{Name: “node_id”, Type: “BLOB”}, &{Name: “status”, Type: “INTEGER”}}, PrimaryKey: string{“node_id”}},
&{
Name: "test_table",
Columns: []*dbschema.Column{
&{Name: "id", Type: "int"},
&{Name: "name", Type: "varchar(30)", IsNullable: true},
},
PrimaryKey: []string{"id"},
},
},
Indexes: nil,
}
storj.io/storj/storagenode/storagenodedb.(*DB ).Preflight:301
main.cmdRun:198
storj.io/storj/pkg/process.cleanup.func1.2:312
storj.io/storj/pkg/process.cleanup.func1:330
github.com/spf13/cobra.(*Command ).execute:826
github.com/spf13/cobra.(*Command ).ExecuteC:914
github.com/spf13/cobra.(*Command ).Execute:864
storj.io/storj/pkg/process.ExecWithCustomConfig:84
storj.io/storj/pkg/process.ExecCustomDebug:66
main.main:328
runtime.main:203
2020-03-17T22:56:32.437Z INFO Configuration loaded from: /app/config/config.yaml
2020-03-17T22:56:32.442Z INFO Operator email: **************
2020-03-17T22:56:32.442Z INFO operator wallet: ***************************************
2020-03-17T22:56:33.214Z INFO version running on version v0.34.6
2020-03-17T22:56:33.269Z INFO db.migration Database Version {“version”: 31}
Error: Error during preflight check for storagenode databases: storage node preflight database error: used_serial: expected schema does not match actual: &dbschema.Schema{
Tables: *dbschema.Table{
&{
Name: "used_serial_",
Columns: []*dbschema.Column{
&{Name: "expiration", Type: "TIMESTAMP"},
&{Name: "satellite_id", Type: "BLOB"},
&{Name: "serial_number", Type: "BLOB"},
},
},
},
Indexes: *dbschema.Index{
&{Name: "idx_used_serial_", Table: "used_serial_", Columns: []string{"expiration"}},
&{
Name: "pk_used_serial_",
Table: "used_serial_",
Columns: []string{"satellite_id", "serial_number"},
},
},
Tables: *dbschema.Table{
&{
Name: "used_serial_",
Columns: []*dbschema.Column{
&{Name: "expiration", Type: "TIMESTAMP"},
&{Name: "satellite_id", Type: "BLOB"},
&{Name: "serial_number", Type: "BLOB"},
},
},
},
Indexes: *dbschema.Index{
&{Name: "idx_used_serial_", Table: "used_serial_", Columns: []string{"expiration"}},
&{
Name: "pk_used_serial_",
Table: "used_serial_",
Columns: []string{"satellite_id", "serial_number"},
},
},
Indexes: nil,
}
storj.io/storj/storagenode/storagenodedb.(*DB ).Preflight:301
main.cmdRun:198
storj.io/storj/pkg/process.cleanup.func1.2:312
storj.io/storj/pkg/process.cleanup.func1:330
github.com/spf13/cobra.(*Command ).execute:826
github.com/spf13/cobra.(*Command ).ExecuteC:914
github.com/spf13/cobra.(*Command ).Execute:864
storj.io/storj/pkg/process.ExecWithCustomConfig:84
storj.io/storj/pkg/process.ExecCustomDebug:66
main.main:328
runtime.main:203
2020-03-17T22:56:49.503Z INFO Configuration loaded from: /app/config/config.yaml
2020-03-17T22:56:49.507Z INFO Operator email: **************
2020-03-17T22:56:49.507Z INFO operator wallet: ***************************************
2020-03-17T22:56:51.275Z INFO version running on version v0.34.6
2020-03-17T22:56:51.377Z INFO db.migration Database Version {“version”: 31}
Error: Error during preflight check for storagenode databases: storage node preflight database error: satellites: expected schema does not match actual: &dbschema.Schema{
Tables: *dbschema.Table{
&{Name: “satellite_exit_progress”, Columns: *dbschema.Column{&{Name: “bytes_deleted”, Type: “INTEGER”}, &{Name: “completion_receipt”, Type: “BLOB”, IsNullable: true}, &{Name: “finished_at”, Type: “TIMESTAMP”, IsNullable: true}, &{Name: “initiated_at”, Type: “TIMESTAMP”, IsNullable: true}, &{Name: “satellite_id”, Type: “BLOB”}, &{Name: “starting_disk_usage”, Type: “INTEGER”}}, PrimaryKey: string{“satellite_id”}},
&{Name: “satellites”, Columns: *dbschema.Column{&{Name: “added_at”, Type: “TIMESTAMP”}, &{Name: “node_id”, Type: “BLOB”}, &{Name: “status”, Type: “INTEGER”}}, PrimaryKey: string{“node_id”}},
&{
Name: "test_table",
Columns: []*dbschema.Column{
&{Name: "id", Type: "int"},
&{Name: "name", Type: "varchar(30)", IsNullable: true},
},
PrimaryKey: []string{"id"},
},
},
Indexes: nil,
}
storj.io/storj/storagenode/storagenodedb.(*DB ).Preflight:301
main.cmdRun:198
storj.io/storj/pkg/process.cleanup.func1.2:312
storj.io/storj/pkg/process.cleanup.func1:330
github.com/spf13/cobra.(*Command ).execute:826
github.com/spf13/cobra.(*Command ).ExecuteC:914
github.com/spf13/cobra.(*Command ).Execute:864
storj.io/storj/pkg/process.ExecWithCustomConfig:84
storj.io/storj/pkg/process.ExecCustomDebug:66
main.main:328
runtime.main:203
2020-03-17T22:57:06.884Z INFO Configuration loaded from: /app/config/config.yaml
2020-03-17T22:57:06.889Z INFO Operator email: **************
2020-03-17T22:57:06.892Z INFO operator wallet: ***************************************
2020-03-17T22:57:07.495Z INFO version running on version v0.34.6
2020-03-17T22:57:07.555Z INFO db.migration Database Version {“version”: 31}
Error: Error during preflight check for storagenode databases: storage node preflight database error: satellites: expected schema does not match actual: &dbschema.Schema{
Tables: *dbschema.Table{
&{Name: “satellite_exit_progress”, Columns: *dbschema.Column{&{Name: “bytes_deleted”, Type: “INTEGER”}, &{Name: “completion_receipt”, Type: “BLOB”, IsNullable: true}, &{Name: “finished_at”, Type: “TIMESTAMP”, IsNullable: true}, &{Name: “initiated_at”, Type: “TIMESTAMP”, IsNullable: true}, &{Name: “satellite_id”, Type: “BLOB”}, &{Name: “starting_disk_usage”, Type: “INTEGER”}}, PrimaryKey: string{“satellite_id”}},
&{Name: “satellites”, Columns: *dbschema.Column{&{Name: “added_at”, Type: “TIMESTAMP”}, &{Name: “node_id”, Type: “BLOB”}, &{Name: “status”, Type: “INTEGER”}}, PrimaryKey: string{“node_id”}},
&{
Name: "test_table",
Columns: []*dbschema.Column{
&{Name: "id", Type: "int"},
&{Name: "name", Type: "varchar(30)", IsNullable: true},
},
PrimaryKey: []string{"id"},
},
},
Indexes: nil,
}
storj.io/storj/storagenode/storagenodedb.(*DB ).Preflight:301
main.cmdRun:198
storj.io/storj/pkg/process.cleanup.func1.2:312
storj.io/storj/pkg/process.cleanup.func1:330
github.com/spf13/cobra.(*Command ).execute:826
github.com/spf13/cobra.(*Command ).ExecuteC:914
github.com/spf13/cobra.(*Command ).Execute:864
storj.io/storj/pkg/process.ExecWithCustomConfig:84
storj.io/storj/pkg/process.ExecCustomDebug:66
main.main:328
runtime.main:203
2020-03-17T22:57:28.461Z INFO Configuration loaded from: /app/config/config.yaml
2020-03-17T22:57:28.466Z INFO Operator email: **************
2020-03-17T22:57:28.467Z INFO operator wallet: ***************************************
2020-03-17T22:57:29.045Z INFO version running on version v0.34.6
2020-03-17T22:57:29.149Z INFO db.migration Database Version {“version”: 31}
Error: Error during preflight check for storagenode databases: storage node preflight database error: info: expected schema does not match actual: &dbschema.Schema{
Tables: *dbschema.Table{
&{
Name: "test_table",
Columns: []*dbschema.Column{
&{Name: "id", Type: "int"},
&{Name: "name", Type: "varchar(30)", IsNullable: true},
},
PrimaryKey: []string{"id"},
},
},
Indexes: nil,
}
storj.io/storj/storagenode/storagenodedb.(*DB ).Preflight:301
main.cmdRun:198
storj.io/storj/pkg/process.cleanup.func1.2:312
storj.io/storj/pkg/process.cleanup.func1:330
github.com/spf13/cobra.(*Command ).execute:826
github.com/spf13/cobra.(*Command ).ExecuteC:914
github.com/spf13/cobra.(*Command ).Execute:864
storj.io/storj/pkg/process.ExecWithCustomConfig:84
storj.io/storj/pkg/process.ExecCustomDebug:66
main.main:328
runtime.main:203
2020-03-17T22:57:56.858Z INFO Configuration loaded from: /app/config/config.yaml
2020-03-17T22:57:56.862Z INFO Operator email: **************
2020-03-17T22:57:56.863Z INFO operator wallet: ***************************************
2020-03-17T22:57:59.070Z INFO version running on version v0.34.6
2020-03-17T22:57:59.196Z INFO db.migration Database Version {“version”: 31}
Error: Error during preflight check for storagenode databases: storage node preflight database error: info: expected schema does not match actual: &dbschema.Schema{
Tables: *dbschema.Table{
&{
Name: "test_table",
Columns: []*dbschema.Column{
&{Name: "id", Type: "int"},
&{Name: "name", Type: "varchar(30)", IsNullable: true},
},
PrimaryKey: []string{"id"},
},
},
Indexes: nil,
}
storj.io/storj/storagenode/storagenodedb.(*DB ).Preflight:301
main.cmdRun:198
storj.io/storj/pkg/process.cleanup.func1.2:312
storj.io/storj/pkg/process.cleanup.func1:330
github.com/spf13/cobra.(*Command ).execute:826
github.com/spf13/cobra.(*Command ).ExecuteC:914
github.com/spf13/cobra.(*Command ).Execute:864
storj.io/storj/pkg/process.ExecWithCustomConfig:84
storj.io/storj/pkg/process.ExecCustomDebug:66
main.main:328
runtime.main:203
2020-03-17T22:58:52.553Z INFO Configuration loaded from: /app/config/config.yaml
2020-03-17T22:58:52.558Z INFO Operator email: **************
2020-03-17T22:58:52.558Z INFO operator wallet: ***************************************
2020-03-17T22:58:53.689Z INFO version running on version v0.34.6
2020-03-17T22:58:53.792Z INFO db.migration Database Version {“version”: 31}
Error: Error during preflight check for storagenode databases: storage node preflight database error: used_serial: expected schema does not match actual: &dbschema.Schema{
Tables: *dbschema.Table{
&{
Name: "used_serial_",
Columns: []*dbschema.Column{
&{Name: "expiration", Type: "TIMESTAMP"},
&{Name: "satellite_id", Type: "BLOB"},
&{Name: "serial_number", Type: "BLOB"},
},
},
},
Indexes: *dbschema.Index{
&{Name: "idx_used_serial_", Table: "used_serial_", Columns: []string{"expiration"}},
&{
Name: "pk_used_serial_",
Table: "used_serial_",
Columns: []string{"satellite_id", "serial_number"},
},
},
Alexey
March 17, 2020, 11:10pm
17
sqlite3 used_serial.db
CREATE TABLE used_serial_ (
satellite_id BLOB NOT NULL,
serial_number BLOB NOT NULL,
expiration TIMESTAMP NOT NULL
);
CREATE UNIQUE INDEX pk_used_serial_ ON used_serial_(satellite_id, serial_number);
CREATE INDEX idx_used_serial_ ON used_serial_(expiration);
.exit
Looks like a huge damage to the databases.
Have you had a power outage?
istie
March 17, 2020, 11:50pm
18
See my previous answer and some questions…
Alexey
March 18, 2020, 8:07am
19
With a new node you will lose the gained reputation and held amount, so better to fix it.
If you want to start over, then you need to generate a new identity, request a new authorization token and start with clean data.
Also satellites.db seems to have this test table as well.
sqlite3 satellites.db
drop table test_table;
.exit