Disk crash and can't recover data (error in storj container)

Hi all,

since past wednesday my disk failed, I bought another and tried to recover as many data as I can.

The fact is I can’t even setup and start the node with my ID.

What I can do?

Here is whta appears in logs:

2022-10-04 11:05:06,798 INFO spawned: ‘storagenode’ with pid 49
2022-10-04 11:05:06,799 WARN received SIGQUIT indicating exit request
2022-10-04 11:05:06,799 INFO waiting for storagenode, processes-exit-eventlistener, storagenode-updater to die
2022-10-04T11:05:06.800Z INFO Got a signal from the OS: “terminated” {“Process”: “storagenode-updater”}
2022-10-04 11:05:06,806 INFO stopped: storagenode-updater (exit status 0)
2022-10-04 11:05:06,808 INFO stopped: storagenode (terminated by SIGTERM)
2022-10-04 11:05:06,809 INFO stopped: processes-exit-eventlistener (terminated by SIGTERM)
2022-10-04 11:05:10,355 INFO Set uid to user 0 succeeded
2022-10-04 11:05:10,362 INFO RPC interface ‘supervisor’ initialized
2022-10-04 11:05:10,363 INFO supervisord started with pid 1
2022-10-04 11:05:11,370 INFO spawned: ‘processes-exit-eventlistener’ with pid 11
2022-10-04 11:05:11,373 INFO spawned: ‘storagenode’ with pid 12
2022-10-04 11:05:11,375 INFO spawned: ‘storagenode-updater’ with pid 13
2022-10-04T11:05:11.427Z INFO Configuration loaded {“Process”: “storagenode-updater”, “Location”: “/app/config/config.yaml”}
2022-10-04T11:05:11.427Z INFO Invalid configuration file key {“Process”: “storagenode-updater”, “Key”: “operator.wallet-features.0”}
2022-10-04T11:05:11.427Z INFO Invalid configuration file key {“Process”: “storagenode-updater”, “Key”: “storage.allocated-bandwidth”}
2022-10-04T11:05:11.427Z INFO Invalid configuration file key {“Process”: “storagenode-updater”, “Key”: “operator.email”}
2022-10-04T11:05:11.427Z INFO Invalid configuration file key {“Process”: “storagenode-updater”, “Key”: “storage.allocated-disk-space”}
2022-10-04T11:05:11.427Z INFO Invalid configuration file key {“Process”: “storagenode-updater”, “Key”: “healthcheck.details”}
2022-10-04T11:05:11.427Z INFO Invalid configuration file key {“Process”: “storagenode-updater”, “Key”: “server.private-address”}
2022-10-04T11:05:11.427Z INFO Invalid configuration file key {“Process”: “storagenode-updater”, “Key”: “contact.external-address”}
2022-10-04T11:05:11.427Z INFO Invalid configuration file key {“Process”: “storagenode-updater”, “Key”: “server.address”}
2022-10-04T11:05:11.427Z INFO Invalid configuration file key {“Process”: “storagenode-updater”, “Key”: “console.address”}
2022-10-04T11:05:11.427Z INFO Invalid configuration file key {“Process”: “storagenode-updater”, “Key”: “operator.wallet”}
2022-10-04T11:05:11.427Z INFO Invalid configuration file key {“Process”: “storagenode-updater”, “Key”: “healthcheck.enabled”}
2022-10-04T11:05:11.427Z INFO Invalid configuration file value for key {“Process”: “storagenode-updater”, “Key”: “log.level”}
2022-10-04T11:05:11.446Z INFO Configuration loaded {“Process”: “storagenode”, “Location”: “/app/config/config.yaml”}
2022-10-04T11:05:11.496Z INFO Anonymized tracing enabled {“Process”: “storagenode”}
2022-10-04T11:05:11.496Z INFO Anonymized tracing enabled {“Process”: “storagenode-updater”}
2022-10-04T11:05:11.508Z INFO Operator email {“Process”: “storagenode”, “Address”: “nicola.lomartire@mac.com”}
2022-10-04T11:05:11.508Z INFO Running on version {“Process”: “storagenode-updater”, “Service”: “storagenode-updater”, “Version”: “v1.64.1”}
2022-10-04T11:05:11.508Z INFO Downloading versions. {“Process”: “storagenode-updater”, “Server Address”: “https://version.storj.io”}
2022-10-04T11:05:11.509Z INFO Operator wallet {“Process”: “storagenode”, “Address”: “0xbe57fF93a1f3C39126A30625FFa5DD9AD2aeB909”}
2022-10-04T11:05:12.017Z INFO Current binary version {“Process”: “storagenode-updater”, “Service”: “storagenode”, “Version”: “v1.64.1”}
2022-10-04T11:05:12.017Z INFO Version is up to date {“Process”: “storagenode-updater”, “Service”: “storagenode”}
2022-10-04T11:05:12.038Z INFO Current binary version {“Process”: “storagenode-updater”, “Service”: “storagenode-updater”, “Version”: “v1.64.1”}
2022-10-04T11:05:12.039Z INFO Version is up to date {“Process”: “storagenode-updater”, “Service”: “storagenode-updater”}
2022-10-04T11:05:12.425Z INFO Telemetry enabled {“Process”: “storagenode”, “instance ID”: “12a5yXMvcnYtdHqwgCdSju8vcq23Ad2y8XsFy2wwWRemetGp1ve”}
2022-10-04 11:05:12,426 INFO success: processes-exit-eventlistener entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2022-10-04 11:05:12,426 INFO success: storagenode entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2022-10-04 11:05:12,427 INFO success: storagenode-updater entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
Error: Error creating tables for master database on storagenode: migrate: database disk image is malformed
storj.io/private/dbutil/txutil.WithTx:53
storj.io/storj/private/migrate.(*Migration).getLatestVersion:249
storj.io/storj/private/migrate.(*Migration).Run:179
storj.io/storj/storagenode/storagenodedb.(*DB).MigrateToLatest:347
main.cmdRun:226
storj.io/private/process.cleanup.func1.4:378
storj.io/private/process.cleanup.func1:396
github.com/spf13/cobra.(*Command).execute:852
github.com/spf13/cobra.(*Command).ExecuteC:960
github.com/spf13/cobra.(*Command).Execute:897
storj.io/private/process.ExecWithCustomConfigAndLogger:93
main.main:478
runtime.main:255
2022-10-04 11:05:12,774 INFO exited: storagenode (exit status 1; not expected)

You can delete all *.db files und start the node again.
Your Statistics will be lost but the data will remain.
Make a backup from all db files before you delete :wink:

And now that’s is the result after the node started.

What I gonna do with this?

Now it’s working with different scores:

Looks like you’ve got it working. It’ll take some time to get the Suspension scores back to 100%, but otherwise it looks as though you’re good to go.

TBH I don’t know how it (re)started working with this scores (the yellow ones); but, if it works, let it work! :smiley:

I was a step from totally give up (without grace exit, just shut down and farewell)

I’m persevering just for respect of the work you make and the community

For sure it’s not for money: this node give me around 10$/Mo, just in Juanuary I spent around 200 € to buy two new dedicated HDD :smiley:

3 Likes

The node is desqualified… So sorry, but so long and farewell :sleepy:

I just setup another node (with new identity, of course)

3 Likes