Storage node failure

Node Operators troubleshooting
1

My storage has stopped for some reason and I am unable to restart. Replaced docker images that dint helped. HDD perfectly mounted and accessible. Any ideas on how to fix it?

docker logs -t storagenode:
2021-03-07T11:53:54.724397185Z 2021-03-07T11:53:54.723Z INFO Configuration loaded {“Location”: “/app/config/config.yaml”}
2021-03-07T11:53:54.728374590Z 2021-03-07T11:53:54.727Z WARN Operator email address isn’t specified.
2021-03-07T11:53:54.729227947Z 2021-03-07T11:53:54.728Z INFO Operator wallet {“Address”: “0xcF733882A842655f15F1b0409E3b88----------”}
2021-03-07T11:53:54.743466852Z Error: Error starting master database on storagenode: database: file is not a database
2021-03-07T11:53:54.743673608Z storj.io/storj/storagenode/storagenodedb.(*DB).openDatabase:323
2021-03-07T11:53:54.743722734Z storj.io/storj/storagenode/storagenodedb.(*DB).openExistingDatabase:305
2021-03-07T11:53:54.743765569Z storj.io/storj/storagenode/storagenodedb.(*DB).openDatabases:281
2021-03-07T11:53:54.743809487Z storj.io/storj/storagenode/storagenodedb.OpenExisting:248
2021-03-07T11:53:54.743849821Z main.cmdRun:160
2021-03-07T11:53:54.743890614Z storj.io/private/process.cleanup.func1.4:363
2021-03-07T11:53:54.743931698Z storj.io/private/process.cleanup.func1:381
2021-03-07T11:53:54.743972575Z github.com/spf13/cobra.(*Command).execute:842
2021-03-07T11:53:54.744012242Z github.com/spf13/cobra.(*Command).ExecuteC:950
2021-03-07T11:53:54.744054535Z github.com/spf13/cobra.(*Command).Execute:887
2021-03-07T11:53:54.744095745Z storj.io/private/process.ExecWithCustomConfig:88
2021-03-07T11:53:54.744136329Z storj.io/private/process.ExecCustomDebug:70
2021-03-07T11:53:54.744176414Z main.main:385
2021-03-07T11:53:54.744216748Z runtime.main:204

2

Welcome to the forum @rumsas1!

Try the steps mentioned here and post the result.

2 Likes
3

Have tried the steps mentioned above. Fixed several corrupted db. Now when I try to start my storage node i am getting:

2021-03-07T15:06:55.222Z INFO db.migration.25 Remove address from satellites table
Error: Error creating tables for master database on storagenode: migrate: no such table: satellites
storj.io/storj/private/migrate.SQL.Run:292
storj.io/storj/private/migrate.(*Migration).Run.func1:197
storj.io/storj/private/dbutil/txutil.withTxOnce:75
storj.io/storj/private/dbutil/txutil.WithTx:36
storj.io/storj/private/migrate.(*Migration).Run:196
storj.io/storj/storagenode/storagenodedb.(*DB).MigrateToLatest:346
main.cmdRun:193
storj.io/private/process.cleanup.func1.4:363
storj.io/private/process.cleanup.func1:381
github.com/spf13/cobra.(*Command).execute:842
github.com/spf13/cobra.(*Command).ExecuteC:950
github.com/spf13/cobra.(*Command).Execute:887
storj.io/private/process.ExecWithCustomConfig:88
storj.io/private/process.ExecCustomDebug:70
main.main:385
runtime.main:204

4

Time for awesome @Alexey :slight_smile:

5

Not sure if it’s related, but just checked in on my 6 nodes (all docker running on linux) via prometheus/grafana and found one was randomly offline. Tried to connect remotely, and couldn’t connect via VNC, and when checking the hardware, it looked like the the boot/root ssd (USB) wasn’t reading (indicator light was off). I power cycled the raspberry pi, and all has started the node back up again just fine so far.

6
  1. Stop the storagenode
  2. Remove satellites.db and all other which was “not a database” during the check
  3. Move all remained databases (*.db) to a different place
  4. Start the storagenode
  5. Stop the storagenode
  6. Move backed up databases back with overwrite
  7. Start the storagenode
  8. Check your logs
2 Likes
7

Thank you Alexey!

Node is fixed. It’s up and running again.

2 Likes