Unrecoverable error :: error creating tables for master database

Node Operators troubleshooting
1

just updated my node and in the log is this unrecoverable error (below). What should I do?

2019-09-19T20:01:54.453Z INFO Configuration loaded from: /app/config/config.yaml
2019-09-19T20:01:54.465Z INFO Operator email:
2019-09-19T20:01:54.465Z INFO operator wallet: 0xxxxxx
2019-09-19T20:01:55.630Z INFO version running on version v0.21.1
2019-09-19T20:01:55.832Z FATAL Unrecoverable error {“error”: “Error creating tables for master database on storagenode: migrate: creating version table failed: migrate: file is not a database\n\tstorj/storj/internal/migrate.(*Migration).Run:112\n\tstorj/storj/storagenode/storagenodedb.(*DB).CreateTables:222\n\tmain.cmdRun:167\n\tstorj/storj/pkg/process.cleanup.func1.2:264\n\tstorj/storj/pkg/process.cleanup.func1:282\n\tgithub/spf13/cobra.(*Command).execute:762\n\tgithub/spf13/cobra.(*Command).ExecuteC:852\n\tgithub/spf13/cobra.(*Command).Execute:800\n\tstorj/storj/pkg/process.Exec:73\n\tmain.main:296\n\truntime.main:203”, “errorVerbose”: “Error creating tables for master database on storagenode: migrate: creating version table failed: migrate: file is not a database\n\tstorj/storj/internal/migrate.(*Migration).Run:112\n\tstorj/storj/storagenode/storagenodedb.(*DB).CreateTables:222\n\tmain.cmdRun:167\n\tstorj/storj/pkg/process.cleanup.func1.2:264\n\tstorj/storj/pkg/process.cleanup.func1:282\n\tgithub/spf13/cobra.(*Command).execute:762\n\tgithub/spf13/cobra.(*Command).ExecuteC:852\n\tgithub/spf13/cobra.(*Command).Execute:800\n\tstorj/storj/pkg/process.Exec:73\n\tmain.main:296\n\truntime.main:203\n\tmain.cmdRun:169\n\tstorj.io/storj/pkg/process.cleanup.func1.2:264\n\tstorj/storj/pkg/process.cleanup.func1:282\n\tgithub/spf13/cobra.(*Command).execute:762\n\tgithub/spf13/cobra.(*Command).ExecuteC:852\n\tgithub/spf13/cobra.(*Command).Execute:800\n\tstorj/storj/pkg/process.Exec:73\n\tmain.main:296\n\truntime.main:203”}

2

Hey @mastrip2 and welcome!

There’s a possibility your database could be corrupt. Can first you try re-starting your container?

docker stop -t 300 storagenode
docker rm storagenode

Then execute your docker run command

If that doesn’t fix it check out this article: https://support.storj.io/hc/en-us/articles/360029309111-How-to-fix-a-database-disk-image-is-malformed-

3

thanks for the reply. I recovered the database but im getting this error now and definitly unsure what to do. Maybe the bandwidth files are also corrupt.

2019-09-20T14:12:41.939Z ERROR orders listing orders {“error”: “ordersdb error: disk I/O error”, “errorVerbose”: “ordersdb error: disk I/O error\n\tstorj.io/storj/storagenode/storagenodedb.(*ordersDB).ListUnsentBySatellite:140\n\tstorj.io/storj/storagenode/orders.(*Service).sendOrders:153\n\tstorj.io/storj/internal/sync2.(*Cycle).Run:87\n\tstorj.io/storj/internal/sync2.(*Cycle).Start.func1:68\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}
2019-09-20T14:12:42.109Z ERROR piecestore:cacheUpdate error getting current space used calculation: {“error”: “context canceled”}
2019-09-20T14:12:42.109Z ERROR piecestore:cacheUpdate error persisting cache totals to the database: {“error”: “piece space used error: context canceled”, “errorVerbose”: “piece space used error: context canceled\n\tstorj.io/storj/storagenode/storagenodedb.(*pieceSpaceUsedDB).UpdateTotal:121\n\tstorj.io/storj/storagenode/pieces.(*CacheService).PersistCacheTotals:82\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run.func1:68\n\tstorj.io/storj/internal/sync2.(*Cycle).Run:87\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run:63\n\tstorj.io/storj/storagenode.(*Peer).Run.func7:439\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}
2019-09-20T14:12:42.147Z ERROR bandwidth Could not rollup bandwidth usage {“error”: “bandwidthdb error: disk I/O error”, “errorVerbose”: “bandwidthdb error: disk I/O error\n\tstorj.io/storj/storagenode/storagenodedb.(*bandwidthDB).Rollup:236\n\tstorj.io/storj/storagenode/bandwidth.(*Service).Rollup:53\n\tstorj.io/storj/internal/sync2.(*Cycle).Run:87\n\tstorj.io/storj/storagenode/bandwidth.(*Service).Run:45\n\tstorj.io/storj/storagenode.(*Peer).Run.func9:446\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}
2019-09-20T14:12:42.186Z ERROR orders cleaning archive {“error”: “ordersdb error: disk I/O error”, “errorVerbose”: “ordersdb error: disk I/O error\n\tstorj.io/storj/storagenode/storagenodedb.(*ordersDB).CleanArchive:326\n\tstorj.io/storj/storagenode/orders.(*Service).cleanArchive:137\n\tstorj.io/storj/internal/sync2.(*Cycle).Run:87\n\tstorj.io/storj/internal/sync2.(*Cycle).Start.func1:68\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}
2019-09-20T14:12:44.241Z FATAL Unrecoverable error {“error”: “bandwidthdb error: disk I/O error”, “errorVerbose”: “bandwidthdb error: disk I/O error\n\tstorj.io/storj/storagenode/storagenodedb.(*bandwidthDB).Summary:112\n\tstorj.io/storj/storagenode/storagenodedb.(*bandwidthDB).MonthSummary:79\n\tstorj.io/storj/storagenode/monitor.(*Service).usedBandwidth:174\n\tstorj.io/storj/storagenode/monitor.(*Service).Run:83\n\tstorj.io/storj/storagenode.(*Peer).Run.func6:436\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}

5

Your database is unrecoverable corrupted.
You have only one option - to remove the info.db and restart the node.
You will lost some unsent orders and stat, your disk and bandwidth usage will not match the real usage.
To start the node after removing the info.db database you should have at least 500GB of free space.

6

This sounds like something more than just the database being corrupted, unless the corruption is causing an out-of-bounds read (which still wouldn’t be an I/O error unless sqlite is just erroneously reporting it that way). You may want to check to make sure the disk isn’t failing.

I’ve also seen this sqlite error when the volume is full or currently mounted read-only.

7

This message is a final for the database. The mentioned article will not help to recover such corruption.
And we seeing the result of the attempt:

But I could be wrong and better to check this:

8

I was able to fix the db. I think I had to basically delete the database (I have backup on hand) and let the node start up new so it would make a new one. Then I stopped the node and dumped the database and browsed thru it and compared both databases so I could perform the db upgrade manually to my corrupted one. I then followed the tutorial from step 11 and restarted the node. and everything seemed to worked as normal. no real issues going forward.

9

Its worth noting I just so happen to have a copy of the db before all this started that wasnt to old. maybe it was only about 24 hrs old. When you get that error “file is not a database” you might as well delete everything and start fresh. I dont think there anything you can do

10

You should follow the guide, the sqlite will create a db, if it doesn’t exist.
Allowing create a db to storagenode could harm much more than prepare the DB before the start of the storagenode.
Of course the prepare is possible only if database file could be read by sqlite