Bandwidthdb error: file is not a database

Node Operators troubleshooting
1

image
image950×61 23 KB

Anyone seen it?

2

It looks like your bandwidth db is either missing or corrupted

3

is it critical? should I worry about it? I haven’t really done anything with my system except for ‘apt update/upgrade’ from time to time?
How would I go about fixing it?

(mostly I’m concerned because I just crested a TB of storage data and would hate to mess it up somehow)

4

HIi @kuprov try checking this thread Unrecoverable error :: error creating tables for master database

5

to repaired a malformed database, please see https://support.storj.io/hc/en-us/articles/360029309111-How-to-fix-a-database-disk-image-is-malformed-

6

Alright, thank you. I’ll look into it tonight. All audits are still passing so there’s no performance problem as of now, it seems.

1 Like
7

Unfortunately the only fix of such error is replacing the destroyed database with an empty one. As a consequence - you will lose the statistics for your bandwidth usage.

Here is an empty database: https://alpha.transfer.sh/FQKvJ/bandwidth.db

8

Alexy, I got the same problem with the orders.db file.

2019-12-11T23:36:46.400Z ERROR orders archiving orders {“error”: “ordersdb error: database disk image is malformed”, “errorVerbose”: “ordersdb error: database disk image is malformed\n\tstorj.io/storj/storagenode/storagenodedb.(*ordersDB).archiveOne:238\n\tstorj.io/storj/storagenode/storagenodedb.(*ordersDB).Archive:202\n\tstorj.io/storj/storagenode/orders.(*Service).handleBatches.func2:213\n\tstorj.io/storj/storagenode/orders.(*Service).handleBatches:230\n\tstorj.io/storj/storagenode/orders.(*Service).sendOrders.func1:164\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}

9

to repaired a malformed database, please see https://support.storj.io/hc/en-us/articles/360029309111-How-to-fix-a-database-disk-image-is-malformed-

10

Bad news for me. Got the following message after watchtower upgraded to 27.1:

2019-12-11T22:53:17.575Z FATAL Unrecoverable error {“error”: “bandwidthdb error: file is not a database”, “errorVerbose”: “bandwidthdb error: file is not a database\n\tstorj.io/storj/storagenode/storagenodedb.(*bandwidthDB).getSummary:153\n\tstorj.io/storj/storagenode/storagenodedb.(*bandwidthDB).Summary:111\n\tstorj.io/storj/storagenode/storagenodedb.(*bandwidthDB).MonthSummary:73\n\tstorj.io/storj/storagenode/monitor.(*Service).usedBandwidth:174\n\tstorj.io/storj/storagenode/monitor.(*Service).Run:83\n\tstorj.io/storj/storagenode.(*Peer).Run.func2:433\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}

11

@mitchellds another user also had a similar problem :frowning:

Alexey uploaded an empty bandwidth.db to get you back up and running:

1 Like
12

Yea, that didn’t work. So here is my new, very brief, error log:

2019-12-11T23:08:28.932Z ERROR bandwidth Could not rollup bandwidth usage {“error”: “bandwidthdb error: file is not a database”, “errorVerbose”: “bandwidthdb error: file is not a database\n\tstorj.io/storj/storagenode/storagenodedb.(*bandwidthDB).Rollup:300\n\tstorj.io/storj/storagenode/bandwidth.(*Service).Rollup:53\n\tstorj.io/storj/private/sync2.(*Cycle).Run:87\n\tstorj.io/storj/storagenode/bandwidth.(*Service).Run:45\n\tstorj.io/storj/storagenode.(*Peer).Run.func9:454\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}
2019-12-11T23:08:28.933Z ERROR version Failed to do periodic version check: version control client error: Get https://version.storj.io: context canceled
2019-12-11T23:08:28.939Z ERROR piecestore:cacheUpdate error getting current space used calculation: {“error”: “context canceled”}
2019-12-11T23:08:28.940Z ERROR piecestore:cacheUpdate error persisting cache totals to the database: {“error”: “piece space used error: context canceled”, “errorVerbose”: “piece space used error: context canceled\n\tstorj.io/storj/storagenode/storagenodedb.(*pieceSpaceUsedDB).UpdateTotal:115\n\tstorj.io/storj/storagenode/pieces.(*CacheService).PersistCacheTotals:82\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run.func1:68\n\tstorj.io/storj/private/sync2.(*Cycle).Run:87\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run:63\n\tstorj.io/storj/storagenode.(*Peer).Run.func6:445\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}
2019-12-11T23:08:29.001Z WARN orders DB contains invalid marshalled orders {“error”: “ordersdb error: database disk image is malformed”, “errorVerbose”: “ordersdb error: database disk image is malformed\n\tstorj.io/storj/storagenode/storagenodedb.(*ordersDB).ListUnsentBySatellite:169\n\tstorj.io/storj/storagenode/orders.(*Service).sendOrders:152\n\tstorj.io/storj/private/sync2.(*Cycle).Run:87\n\tstorj.io/storj/private/sync2.(*Cycle).Start.func1:68\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}
2019-12-11T23:08:29.001Z INFO orders.118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW sending {“count”: 387}
2019-12-11T23:08:29.001Z INFO orders.12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S sending {“count”: 411}
2019-12-11T23:08:29.002Z INFO orders.118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW finished
2019-12-11T23:08:29.002Z ERROR orders.118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW failed to settle orders {“error”: “order: unable to connect to the satellite: rpccompat: context canceled”, “errorVerbose”: “order: unable to connect to the satellite: rpccompat: context canceled\n\tstorj.io/storj/storagenode/orders.(*Service).settle:256\n\tstorj.io/storj/storagenode/orders.(*Service).Settle:195\n\tstorj.io/storj/storagenode/orders.(*Service).sendOrders.func2:174\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}
2019-12-11T23:08:29.002Z INFO orders.121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6 sending {“count”: 481}
2019-12-11T23:08:29.002Z INFO orders.12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs sending {“count”: 349}
2019-12-11T23:08:29.002Z INFO orders.12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S finished
2019-12-11T23:08:29.003Z INFO orders.121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6 finished
2019-12-11T23:08:29.003Z ERROR orders.121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6 failed to settle orders {“error”: “order: unable to connect to the satellite: rpccompat: context canceled”, “errorVerbose”: “order: unable to connect to the satellite: rpccompat: context canceled\n\tstorj.io/storj/storagenode/orders.(*Service).settle:256\n\tstorj.io/storj/storagenode/orders.(*Service).Settle:195\n\tstorj.io/storj/storagenode/orders.(*Service).sendOrders.func2:174\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}
2019-12-11T23:08:29.003Z ERROR orders.12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S failed to settle orders {“error”: “order: unable to connect to the satellite: rpccompat: context canceled”, “errorVerbose”: “order: unable to connect to the satellite: rpccompat: context canceled\n\tstorj.io/storj/storagenode/orders.(*Service).settle:256\n\tstorj.io/storj/storagenode/orders.(*Service).Settle:195\n\tstorj.io/storj/storagenode/orders.(*Service).sendOrders.func2:174\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}
2019-12-11T23:08:29.003Z INFO orders.12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs finished
2019-12-11T23:08:29.003Z ERROR orders.12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs failed to settle orders {“error”: “order: unable to connect to the satellite: rpccompat: context canceled”, “errorVerbose”: “order: unable to connect to the satellite: rpccompat: context canceled\n\tstorj.io/storj/storagenode/orders.(*Service).settle:256\n\tstorj.io/storj/storagenode/orders.(*Service).Settle:195\n\tstorj.io/storj/storagenode/orders.(*Service).sendOrders.func2:174\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}
2019-12-11T23:08:30.982Z DEBUG orders cleanup finished {“items deleted”: 0}
2019-12-11T23:08:31.045Z FATAL Unrecoverable error {“error”: “bandwidthdb error: file is not a database”, “errorVerbose”: “bandwidthdb error: file is not a database\n\tstorj.io/storj/storagenode/storagenodedb.(*bandwidthDB).getSummary:153\n\tstorj.io/storj/storagenode/storagenodedb.(*bandwidthDB).Summary:111\n\tstorj.io/storj/storagenode/storagenodedb.(*bandwidthDB).MonthSummary:73\n\tstorj.io/storj/storagenode/monitor.(*Service).usedBandwidth:174\n\tstorj.io/storj/storagenode/monitor.(*Service).Run:83\n\tstorj.io/storj/storagenode.(*Peer).Run.func2:433\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}

13

…and I was on a VERY GOOD ROLL before the update. Everything was working perfectly. This is a big bummer.

14

The file that Alexy posted…and I subsequently used… returned this error:

2019-12-11T23:11:14.757Z FATAL Unrecoverable error {“error”: “bandwidthdb error: file is not a database”, “errorVerbose”: “bandwidthdb error: file is not a database\n\tstorj.io/storj/storagenode/storagenodedb.(*bandwidthDB).getSummary:153\n\tstorj.io/storj/storagenode/storagenodedb.(*bandwidthDB).Summary:111\n\tstorj.io/storj/storagenode/storagenodedb.(*bandwidthDB).MonthSummary:73\n\tstorj.io/storj/storagenode/monitor.(*Service).usedBandwidth:174\n\tstorj.io/storj/storagenode/monitor.(*Service).Run:83\n\tstorj.io/storj/storagenode.(*Peer).Run.func2:433\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}

15

Another thing to maybe try -

Stopping the container docker stop storagenode -t 300
Removing the container docker rm storagenode
Then rebooting the system sudo reboot

Worth a shot.

16

Hey Alexey, I also now get this error. However, my node is still passing audits. The only thing I did was change --name storagenode storjlabs/storagenode:alpha to --name storagenode storjlabs/storagenode:beta and I reduced my storage from 10 TB to 3.5 TB, when my node was at 3.1 TB. Did this cause the error?

17

I’m following https://support.storj.io/hc/en-us/articles/360029309111-How-to-fix-a-database-disk-image-is-malformed- because indeed my database is malformed, somehow. Do I follow this procedure for every .db file in the folder?Malformed

E.g., info.db, satellites.db ect?

18

Yes you should check all your databases.

19

Thanks… I did. It seams that bandwidth.db fails on the last 10 kb or something. I check again with docker run --rm -it --mount type=bind,source=x:\storagenode\storage\bandwidth.db,destination=/bandwidth.db sstc/sqlite3 sqlite3 /bandwidth.db “PRAGMA integrity_check;” and it still says malformed. I will start my node again with the backup i made of bandwidth.db since maybe the network can restore it (no clue). All i know is that the backup i made on another drive is completely intact (from yesterday actually). Very strange.

20

The data in this db won’t be restored, but I’m pretty sure it’s non essential. It is however used to calculate values on the dashboards both web and cli. So keep in mind that numbers displayed there may be wrong. In addition to that community tools like the earnings calculator and anything that uses the dashboard API will likely show bad data as well. New data from this point on should be correct again though if you use an old db file.

This doesn’t necessarily go for all db’s but it think with this one you’re fine with an old backup.