Node restarting, malformed database

Node Operators troubleshooting
1

hey :slight_smile:
I had transferred all my data to other pc.And attempted to start it back up again.
When I did, my node kept restarting, and in logs it has shown that my database is malformed.
After checking my databases , I have found that piecestore.db is throwing these errors.
storj data

I’ll try to unload and load the data again.

Update 1:
Unloading and loading is done.
Here is the error that i get.Not sure if it’s even done cause of it.

storj data error
storj data error1050×295 68.5 KB

2

Anything I can do at this point?Does this mean it’s unsalvagable?I ran this as well on the newly created pieceinfo.db and it said ok

data storage
data storage709×105 26 KB

.But I’m not sure if this means anything at all.All in all newly created pieceinfo is tiny and not sure it’s even complete by judging by that error that occured when creating it from dump_all_notrans.sql

3

@Alexey any suggestions on what to do? :slight_smile:
Besides what I have done already.

4

That error can be safely ignored. It’s normal that the database is much smaller as it will have gotten rid of a lot of no longer needed data (similar to when you vacuum it). If your db now tests ok, it should be good to go now.

2 Likes
5 
storj.io/private/process.ExecWithCustomConfig:88
        storj.io/private/process.ExecCustomDebug:70
        main.main:320
        runtime.main:203
2020-07-01T08:15:39.464Z        INFO    Configuration loaded    {"Location": "/app/config/config.yaml"}
2020-07-01T08:15:39.486Z        INFO    Operator email  {"Address": "//"}
2020-07-01T08:15:39.487Z        INFO    Operator wallet {"Address": "//"}
2020-07-01T08:15:40.366Z        INFO    Telemetry enabled
2020-07-01T08:15:40.383Z        INFO    db.migration    Database Version        {"version": 39}
Error: Error during preflight check for storagenode databases: storage node preflight database error: pieceinfo: expected schema does not match actual:   &dbschema.Schema{
        Tables: []*dbschema.Table{&{Name: "pieceinfo_", Columns: []*dbschema.Column{&{Name: "deletion_failed_at", Type: "TIMESTAMP", IsNullable: true}, &{Name: "order_limit", Type: "BLOB"}, &{Name: "piece_creation", Type: "TIMESTAMP"}, &{Name: "piece_expiration", Type: "TIMESTAMP", IsNullable: true}, &{Name: "piece_id", Type: "BLOB"}, &{Name: "piece_size", Type: "BIGINT"}, &{Name: "satellite_id", Type: "BLOB"}, &{Name: "uplink_cert_id", Type: "INTEGER", Reference: &dbschema.Reference{Table: "certificate", Column: "cert_id"}}, &{Name: "uplink_piece_hash", Type: "BLOB"}}}},
        Indexes: []*dbschema.Index{
                &{Name: "idx_pieceinfo__expiration", Table: "pieceinfo_", Columns: []string{"piece_expiration"}, Partial: "piece_expiration IS NOT NULL"},
-               s`Index<Table: pieceinfo_, Name: pk_pieceinfo_, Columns: satellite_id piece_id, Unique: false, Partial: "">`,
        },
  }

        storj.io/storj/storagenode/storagenodedb.(*DB).Preflight:323
        main.cmdRun:190
        storj.io/private/process.cleanup.func1.4:359
        storj.io/private/process.cleanup.func1:377
        github.com/spf13/cobra.(*Command).execute:840
        github.com/spf13/cobra.(*Command).ExecuteC:945
        github.com/spf13/cobra.(*Command).Execute:885
        storj.io/private/process.ExecWithCustomConfig:88
        storj.io/private/process.ExecCustomDebug:70
        main.main:320
        runtime.main:203

I have started my node using that smaller db :slight_smile: , and it’s restarting again, and this is what i get in the logs.I have removed email and the wallet part.

6

Please,

  1. Stop the storagenode
  2. make a backup of the pieceinfo.db
  3. execute:
sudo docker run -it --rm --mount type=bind,source=/mnt/data/Storagenode/storage,destination=/data sstc/sqlite3 sqlite3 pieceinfo.db

delete from pieceinfo_ as pi 
where exists(select 1 from pieceinfo_ 
        where pi.satellite_id = satellite_id and pi.piece_id = piece_id
        group by satellite_id, piece_id
        having count(*) > 1
    );

CREATE UNIQUE INDEX pk_pieceinfo_ ON pieceinfo_(satellite_id, piece_id);
CREATE INDEX idx_pieceinfo__expiration ON pieceinfo_(piece_expiration) WHERE piece_expiration IS NOT NULL;
.exit
  1. Try to start the storagenode, if no errors
2 Likes
7

It is working, thank you very much.Sorry for having to write all of that by hand.
It seems that node now sees only 313gb of data, whilst it had around 540gb.But it’s okay, at least it’s working now :slight_smile: .Was so worried it was a lost cause.