Node crashes after restart

Node Operators troubleshooting
1

Hello there’s one node that keeps restarting! the problem has happened one of these days. Here are part of the log! I’ve run a health check and restarted it again!

2024-05-23T21:57:46Z	INFO	Telemetry enabled	{"Process": "storagenode", "instance ID": "1PYi5kCo4kYx1DKd816QfULhmAyMDK7rtUjVZnD4iYEoKSsXoS"}
2024-05-23T21:57:46Z	INFO	Event collection enabled	{"Process": "storagenode", "instance ID": "1PYi5kCo4kYx1DKd816QfULhmAyMDK7rtUjVZnD4iYEoKSsXoS"}
2024-05-23T21:57:46Z	INFO	db.migration.57	Create new bandwidth_usage table, backfilling data from bandwidth_usage_rollups and bandwidth_usage tables, and dropping the old tables.	{"Process": "storagenode"}
2024-05-23T21:57:46Z	ERROR	failure during run	{"Process": "storagenode", "error": "Error migrating tables for database on storagenode: migrate: v57: database disk image is malformed\n\tstorj.io/storj/private/migrate.(*Migration).Run:212\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).MigrateToLatest:387\n\tmain.cmdRun:98\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.cleanup.func1.4:393\n\tstorj.io/common/process.cleanup.func1:411\n\tgithub.com/spf13/cobra.(*Command).execute:983\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1115\n\tgithub.com/spf13/cobra.(*Command).Execute:1039\n\tstorj.io/common/process.ExecWithCustomOptions:112\n\tmain.main:34\n\truntime.main:267", "errorVerbose": "Error migrating tables for database on storagenode: migrate: v57: database disk image is malformed\n\tstorj.io/storj/private/migrate.(*Migration).Run:212\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).MigrateToLatest:387\n\tmain.cmdRun:98\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.cleanup.func1.4:393\n\tstorj.io/common/process.cleanup.func1:411\n\tgithub.com/spf13/cobra.(*Command).execute:983\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1115\n\tgithub.com/spf13/cobra.(*Command).Execute:1039\n\tstorj.io/common/process.ExecWithCustomOptions:112\n\tmain.main:34\n\truntime.main:267\n\tmain.cmdRun:100\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.cleanup.func1.4:393\n\tstorj.io/common/process.cleanup.func1:411\n\tgithub.com/spf13/cobra.(*Command).execute:983\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1115\n\tgithub.com/spf13/cobra.(*Command).Execute:1039\n\tstorj.io/common/process.ExecWithCustomOptions:112\n\tmain.main:34\n\truntime.main:267"}
Error: Error migrating tables for database on storagenode: migrate: v57: database disk image is malformed
	storj.io/storj/private/migrate.(*Migration).Run:212
	storj.io/storj/storagenode/storagenodedb.(*DB).MigrateToLatest:387
	main.cmdRun:98
	main.newRunCmd.func1:33
	storj.io/common/process.cleanup.func1.4:393
	storj.io/common/process.cleanup.func1:411
	github.com/spf13/cobra.(*Command).execute:983
	github.com/spf13/cobra.(*Command).ExecuteC:1115
	github.com/spf13/cobra.(*Command).Execute:1039
	storj.io/common/process.ExecWithCustomOptions:112
	main.main:34
	runtime.main:267
2024-05-23 21:57:46,243 INFO stopped: storagenode (exit status 1)
2024-05-23 21:57:46,245 INFO stopped: processes-exit-eventlistener (terminated by SIGTERM)
2

it doesn’t solve… created new container!

3

Hello @toma,
Welcome back!

You need either fix it:

or re-create (you will lose a bandwidth usage history in this case, however, it will not affect your payout, but the bandwidth usage on the dashboard will be wrong, and estimation likely too):

1 Like
4

I’m getting ‘storagewars@raspberrypi:~ $ sudo docker run --rm -it --mount type=bind,source=/home/storagewars/Hds/HD2/storage,destination=/storage sstc/sqlite3 sh
Unable to find image ‘sstc/sqlite3:latest’ locally
latest: Pulling from sstc/sqlite3
docker: no matching manifest for linux/arm/v8 in the manifest list entries’

5

how do i re-create it?

6

Just stop node and delete all databases. The node creates new databases on next start.

7

Then you need to use a binary sqlite3, please install it using your package manager (for Debian based it’s apt-get).

8

all databases in what file? or arquive file? all .db?

9 
2024-05-25T14:50:21Z	INFO	Configuration loaded	{"Process": "storagenode-updater", "Location": "/app/config/config.yaml"}
2024-05-25T14:50:21Z	INFO	Invalid configuration file key	{"Process": "storagenode-updater", "Key": "operator.wallet-features"}
2024-05-25T14:50:21Z	INFO	Invalid configuration file key	{"Process": "storagenode-updater", "Key": "healthcheck.enabled"}
2024-05-25T14:50:21Z	INFO	Invalid configuration file key	{"Process": "storagenode-updater", "Key": "operator.email"}
2024-05-25T14:50:21Z	INFO	Invalid configuration file key	{"Process": "storagenode-updater", "Key": "server.address"}
2024-05-25T14:50:21Z	INFO	Invalid configuration file key	{"Process": "storagenode-updater", "Key": "operator.wallet"}
2024-05-25T14:50:21Z	INFO	Invalid configuration file key	{"Process": "storagenode-updater", "Key": "healthcheck.details"}
2024-05-25T14:50:21Z	INFO	Invalid configuration file key	{"Process": "storagenode-updater", "Key": "console.address"}
2024-05-25T14:50:21Z	INFO	Invalid configuration file key	{"Process": "storagenode-updater", "Key": "contact.external-address"}
2024-05-25T14:50:21Z	INFO	Invalid configuration file key	{"Process": "storagenode-updater", "Key": "server.private-address"}
2024-05-25T14:50:21Z	INFO	Invalid configuration file key	{"Process": "storagenode-updater", "Key": "storage.allocated-bandwidth"}
2024-05-25T14:50:21Z	INFO	Invalid configuration file key	{"Process": "storagenode-updater", "Key": "storage.allocated-disk-space"}
10

It says : storagewars@raspberrypi:~/Hds/HD2/storage $ sudo apt-get install sqlite3 Reading package lists... Done Building dependency tree... Done Reading state information... Done sqlite3 is already the newest version (3.34.1-3). 0 upgraded, 0 newly installed, 0 to remove and 109 not upgraded.

when trying to run storagewars@raspberrypi:~/Hds/HD2/storage $ sudo docker run --rm -it --mount type=bind,source=/home/storagewars/Hds/HD2/storage,destination=/storage sstc/sqlite3 sh Unable to find image 'sstc/sqlite3:latest' locally latest: Pulling from sstc/sqlite3 docker: no matching manifest for linux/arm/v8 in the manifest list entries. See 'docker run --help'.

11

all *.db. Please use this manual:

1 Like
12 
storagewars@raspberrypi:~/Hds/HD2/storage $ sudo apt-get install sqlite3 
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Suggested packages:
  sqlite3-doc
The following NEW packages will be installed:
  sqlite3
0 upgraded, 1 newly installed, 0 to remove and 109 not upgraded.
Need to get 1,054 kB of archives.
After this operation, 2,729 kB of additional disk space will be used.
Get:1 http://mirrors.fe.up.pt/raspbian/raspbian bullseye/main armhf sqlite3 armhf 3.34.1-3 [1,054 kB]
Fetched 1,054 kB in 11s (92.1 kB/s)                                                                                               
Selecting previously unselected package sqlite3.
(Reading database ... 178253 files and directories currently installed.)
Preparing to unpack .../sqlite3_3.34.1-3_armhf.deb ...
Unpacking sqlite3 (3.34.1-3) ...
Setting up sqlite3 (3.34.1-3) ...
Processing triggers for man-db (2.9.4-2) ...
storagewars@raspberrypi:~/Hds/HD2/storage $ sudo docker stop storagenode22
storagenode22
storagewars@raspberrypi:~/Hds/HD2/storage $ sudo docker run --rm -it --mount type=bind,source=/home/storagewars/Hds/HD2/storage,destination=/home/storagewars/Hds/HD2/storage sstc/sqlite3 sh
Unable to find image 'sstc/sqlite3:latest' locally
latest: Pulling from sstc/sqlite3
docker: no matching manifest for linux/arm/v8 in the manifest list entries.
See 'docker run --help'.

:upside_down_face:

13

Should docker be updated?

14

You installed SQLite directly so you need to follow direct installation instructions on that page.

This should be the command

find /path/to/storage/ -maxdepth 1 -iname "*.db" -print0 -exec sqlite3 '{}' 'PRAGMA integrity_check;' ';'
2 Likes
15

here is the output : storagewars@raspberrypi:// $ find /home/storagewars/Hds/HD2/storage -maxdepth 1 -iname "*.db" -print0 -exec sqlite3 '{}' 'PRAGMA integrity_check;' ';' /home/storagewars/Hds/HD2/storage/piece_spaced_used.dbok /home/storagewars/Hds/HD2/storage/orders.dbok /home/storagewars/Hds/HD2/storage/notifications.dbok /home/storagewars/Hds/HD2/storage/heldamount.dbok /home/storagewars/Hds/HD2/storage/pieceinfo.dbok /home/storagewars/Hds/HD2/storage/used_serial.dbok /home/storagewars/Hds/HD2/storage/reputation.dbok /home/storagewars/Hds/HD2/storage/info.dbok /home/storagewars/Hds/HD2/storage/bandwidth.dbok /home/storagewars/Hds/HD2/storage/secret.dbok /home/storagewars/Hds/HD2/storage/used_space_per_prefix.dbok /home/storagewars/Hds/HD2/storage/piece_expiration.dbok /home/storagewars/Hds/HD2/storage/storage_usage.dbok /home/storagewars/Hds/HD2/storage/garbage_collection_filewalker_progress.dbok /home/storagewars/Hds/HD2/storage/satellites.dbok /home/storagewars/Hds/HD2/storage/pricing.dbok

16

Strange, I would have expected an error for bandwidth.db. You can try starting your node. If it still fails, stop the node again and try following the repair steps for bandwidth.db starting at step 9.

17

got this now: /home/storagewars/Hds/HD2/storage/bandwidth.dbError: database disk image is malformed

18

When doing what? Little more context would be helpful.

19 
storagewars@raspberrypi:~/Hds/HD2/storage $ sqlite3 bandwidth.db
SQLite version 3.34.1 2021-01-20 14:10:07
Enter ".help" for usage hints.
sqlite> .mode insert
sqlite> .output /storage/dump_all.sql
Error: cannot open "/storage/dump_all.sql"
Error: cannot write to "/storage/dump_all.sql"
sqlite> .output dump_all.sql
sqlite> .dump
sqlite> .exit
storagewars@raspberrypi:~/Hds/HD2/storage $ cat /storage/dump_all.sql | grep -v TRANSACTION | grep -v ROLLBACK | grep -v COMMIT >/storage/dump_all_notrans.sql
cat: /storage/dump_all.sqlbash: /storage/dump_all_notrans.sql: No such file or directory
: No such file or directory
20

same command, differ output!