Master database corrupt after watchtower upgrades storagenode

Node Operators troubleshooting
,
1

My Storj node won’t start. Is anything wrong with my run command? (I ran the setup instructions many months ago)

docker run -d --restart unless-stopped --stop-timeout 300 \ -p 28967:28967 \ -p 127.0.0.1:14002:14002 \ -e WALLET="<wallet>" \ -e EMAIL="jmetaj@hotmail.com" \ -e ADDRESS="amtubu.ddns.net:28967" \ -e STORAGE="9.5TB" \ --mount type=bind,source="/home/jb/.local/share/storj/identity/storagenode",destination=/app/identity \ --mount type=bind,source="/mnt/hangten/storjFarm/node1",destination=/app/config \ --name storagenode storjlabs/storagenode

yields

Error: Error starting master database on storagenode: database: file is not a database storj.io/storj/storagenode/storagenodedb.(*DB).openDatabase:323 storj.io/storj/storagenode/storagenodedb.(*DB).openExistingDatabase:305 storj.io/storj/storagenode/storagenodedb.(*DB).openDatabases:281 storj.io/storj/storagenode/storagenodedb.OpenExisting:248 main.cmdRun:160 storj.io/private/process.cleanup.func1.4:363 storj.io/private/process.cleanup.func1:381 github.com/spf13/cobra.(*Command).execute:842 github.com/spf13/cobra.(*Command).ExecuteC:950 github.com/spf13/cobra.(*Command).Execute:887 storj.io/private/process.ExecWithCustomConfig:88 storj.io/private/process.ExecCustomDebug:70 main.main:385 runtime.main:204

Nothing has changed on this machine. It’s been running fine for months. Watchtower logs show

time="2021-02-07T04:35:30Z" level=info msg="Found new storjlabs/storagenode:latest image (sha256:426ced3a1c89ed400d1db44437295f38d85c448221710bf50f15a76f93b8bcf5)" time="2021-02-07T04:35:31Z" level=info msg="Stopping /storagenode (d9ff0e4b1a27f5b2af88312c0361c162bfd00ec2a0b62b815429ee0c43ec3d66) with SIGTERM" time="2021-02-07T04:35:36Z" level=info msg="Creating /storagenode" time="2021-02-07T04:35:43Z" level=info msg="Removing image sha256:7d88b04aa4ed02967871ae4f5d378c87ce6db59ea22d33b839f89f90becb62cc" time="2021-02-10T01:44:30Z" level=info msg="Waiting for running update to be finished..."

2

Try steps from here and update the post.

3

I’ve completed the steps for fixing malformed dbs, but several resulted in empty db files (size=0). Are there any other recovery steps, or must I wipe the node data and start over?

4

What databases was corrupted?
If you give me a name, I can give you a script to recover them. You can remove all zeroed databases - they are unrecoverable.
The other easy way to restore databases would be:

  1. Stop the storagenode
  2. Move all healthy databases to temporary folder
  3. Start your node and check logs when databases got created: https://documentation.storj.io/resources/faq/check-logs
  4. Stop the node when databases are created
  5. Move backed up databases to the original place and replace created ones
  6. Start storagenode again - now it should not complain anymore.