Which database is this? Node not starting

jarridgraham · July 19, 2024, 1:57am

Trying to figure out what happened, all was well then bam.

Thanks

2024-07-19T01:49:26Z    INFO    db.migration.58 Remove unused trash column      {"Process": "storagenode"}
2024-07-19T01:49:26Z    ERROR   failure during run      {"Process": "storagenode", "error": "Error migrating tables for database on storagenode: migrate: v58: database disk image is malformed\n\tstorj.io/storj/private/migrate.(*Migration).Run:212\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).MigrateToLatest:425\n\tmain.cmdRun:100\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.cleanup.func1.4:393\n\tstorj.io/common/process.cleanup.func1:411\n\tgithub.com/spf13/cobra.(*Command).execute:983\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1115\n\tgithub.com/spf13/cobra.(*Command).Execute:1039\n\tstorj.io/common/process.ExecWithCustomOptions:112\n\tmain.main:34\n\truntime.main:271", "errorVerbose": "Error migrating tables for database on storagenode: migrate: v58: database disk image is malformed\n\tstorj.io/storj/private/migrate.(*Migration).Run:212\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).MigrateToLatest:425\n\tmain.cmdRun:100\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.cleanup.func1.4:393\n\tstorj.io/common/process.cleanup.func1:411\n\tgithub.com/spf13/cobra.(*Command).execute:983\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1115\n\tgithub.com/spf13/cobra.(*Command).Execute:1039\n\tstorj.io/common/process.ExecWithCustomOptions:112\n\tmain.main:34\n\truntime.main:271\n\tmain.cmdRun:102\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.cleanup.func1.4:393\n\tstorj.io/common/process.cleanup.func1:411\n\tgithub.com/spf13/cobra.(*Command).execute:983\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1115\n\tgithub.com/spf13/cobra.(*Command).Execute:1039\n\tstorj.io/common/process.ExecWithCustomOptions:112\n\tmain.main:34\n\truntime.main:271"}
2024-07-19T01:49:26Z    FATAL   Unrecoverable error     {"Process": "storagenode", "error": "Error migrating tables for database on storagenode: migrate: v58: database disk image is malformed\n\tstorj.io/storj/private/migrate.(*Migration).Run:212\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).MigrateToLatest:425\n\tmain.cmdRun:100\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.cleanup.func1.4:393\n\tstorj.io/common/process.cleanup.func1:411\n\tgithub.com/spf13/cobra.(*Command).execute:983\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1115\n\tgithub.com/spf13/cobra.(*Command).Execute:1039\n\tstorj.io/common/process.ExecWithCustomOptions:112\n\tmain.main:34\n\truntime.main:271", "errorVerbose": "Error migrating tables for database on storagenode: migrate: v58: database disk image is malformed\n\tstorj.io/storj/private/migrate.(*Migration).Run:212\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).MigrateToLatest:425\n\tmain.cmdRun:100\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.cleanup.func1.4:393\n\tstorj.io/common/process.cleanup.func1:411\n\tgithub.com/spf13/cobra.(*Command).execute:983\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1115\n\tgithub.com/spf13/cobra.(*Command).Execute:1039\n\tstorj.io/common/process.ExecWithCustomOptions:112\n\tmain.main:34\n\truntime.main:271\n\tmain.cmdRun:102\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.cleanup.func1.4:393\n\tstorj.io/common/process.cleanup.func1:411\n\tgithub.com/spf13/cobra.(*Command).execute:983\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1115\n\tgithub.com/spf13/cobra.(*Command).Execute:1039\n\tstorj.io/common/process.ExecWithCustomOptions:112\n\tmain.main:34\n\truntime.main:271"}

arrogantrabbit · July 19, 2024, 1:59am

Your system allowed databases to get corrupted.

You need to check each database and recreate corrupted ones. But then fix the root cause of the corruption.

jarridgraham · July 19, 2024, 2:11am

OK are there more databases to check than the ones in the storage dir? I copied them to a temp dir and ran this on them. They all seem to be OK. Unless that does not detect the issue in this case.

Thanks

for i in $(ls) ; do echo -n "$i "; sqlite3 $i "PRAGMA integrity_check" ; done
bandwidth.db ok
heldamount.db ok
info.db ok
notifications.db ok
orders.db ok
piece_expiration.db ok
pieceinfo.db ok
piece_spaced_used.db ok
pricing.db ok
reputation.db ok
satellites.db ok
secret.db ok
storage_usage.db ok
used_serial.db ok

jarridgraham · July 19, 2024, 2:52am

It looks like when I start the node it downloads v1.105.4, then downloads v1.108.3 then starts and it looks like maybe it is trying to update a database somewhere and it craps out.

nerdatwork · July 19, 2024, 2:54am

Try to change log level from info to debug to get more details in your log. You can do this in the config.yaml file.

jarridgraham · July 19, 2024, 3:08am

Yea I did that, here is what I get from.

docker logs storagenode
there is a bunch of download stuff I cut off.

2024-07-19 02:52:12,567 INFO Set uid to user 0 succeeded
2024-07-19 02:52:12,575 INFO RPC interface 'supervisor' initialized
2024-07-19 02:52:12,576 INFO supervisord started with pid 1
2024-07-19 02:52:13,577 INFO spawned: 'processes-exit-eventlistener' with pid 64
2024-07-19 02:52:13,579 INFO spawned: 'storagenode' with pid 65
2024-07-19 02:52:13,581 INFO spawned: 'storagenode-updater' with pid 66
2024-07-19T02:52:13Z    INFO    Configuration loaded    {"Process": "storagenode-updater", "Location": "/app/config/config.yaml"}
2024-07-19T02:52:13Z    INFO    Anonymized tracing enabled      {"Process": "storagenode-updater"}
2024-07-19T02:52:13Z    INFO    Running on version      {"Process": "storagenode-updater", "Service": "storagenode-updater", "Version": "v1.108.3"}
2024-07-19T02:52:13Z    INFO    Downloading versions.   {"Process": "storagenode-updater", "Server Address": "https://version.storj.io"}
2024-07-19T02:52:13Z    INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode", "Version": "v1.108.3"}
2024-07-19T02:52:13Z    INFO    Version is up to date   {"Process": "storagenode-updater", "Service": "storagenode"}
2024-07-19T02:52:13Z    INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode-updater", "Version": "v1.108.3"}
2024-07-19T02:52:13Z    INFO    Version is up to date   {"Process": "storagenode-updater", "Service": "storagenode-updater"}
2024-07-19 02:52:14,786 INFO success: processes-exit-eventlistener entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2024-07-19 02:52:14,786 INFO success: storagenode entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2024-07-19 02:52:14,786 INFO success: storagenode-updater entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2024-07-19 02:52:15,037 INFO exited: storagenode (exit status 1; not expected)
2024-07-19 02:52:16,040 INFO spawned: 'storagenode' with pid 112
2024-07-19 02:52:16,041 WARN received SIGQUIT indicating exit request
2024-07-19 02:52:16,041 INFO waiting for storagenode, processes-exit-eventlistener, storagenode-updater to die
2024-07-19T02:52:16Z    INFO    Got a signal from the OS: "terminated"  {"Process": "storagenode-updater"}
2024-07-19 02:52:16,043 INFO stopped: storagenode-updater (exit status 0)
2024-07-19 02:52:16,044 INFO stopped: storagenode (terminated by SIGTERM)
2024-07-19 02:52:16,044 INFO stopped: processes-exit-eventlistener (terminated by SIGTERM)

I have it save to external logs as well and I get this in there.

2024-07-19T02:52:25Z    INFO    Configuration loaded    {"Process": "storagenode", "Location": "/app/config/config.yaml"}
2024-07-19T02:52:25Z    INFO    Anonymized tracing enabled      {"Process": "storagenode"}
2024-07-19T02:52:25Z    DEBUG   tracing collector       started {"Process": "storagenode"}
2024-07-19T02:52:25Z    DEBUG   debug server listening on 127.0.0.1:45325       {"Process": "storagenode"}
2024-07-19T02:52:25Z    INFO    Operator email  {"Process": "storagenode", "Address": "..."}
2024-07-19T02:52:25Z    INFO    Operator wallet {"Process": "storagenode", "Address": "..."}
2024-07-19T02:52:25Z    DEBUG   Version info    {"Process": "storagenode", "Version": "1.108.3", "Commit Hash": "318a54287f2ebd59a5cd9ce3320cee798007a51c", "Build Timestamp": "2024-07-09 18:27:20 +0000 UTC", "Release Build": true}
2024-07-19T02:52:25Z    INFO    server  kernel support for server-side tcp fast open remains disabled.  {"Process": "storagenode"}
2024-07-19T02:52:25Z    INFO    server  enable with: sysctl -w net.ipv4.tcp_fastopen=3  {"Process": "storagenode"}
2024-07-19T02:52:26Z    DEBUG   version Allowed minimum version from control server.    {"Process": "storagenode", "Minimum Version": "1.105.4"}
2024-07-19T02:52:26Z    DEBUG   version Running on allowed version.     {"Process": "storagenode", "Version": "1.108.3"}
2024-07-19T02:52:26Z    INFO    Telemetry enabled       {"Process": "storagenode", "instance ID": "..."}
2024-07-19T02:52:26Z    INFO    Event collection enabled        {"Process": "storagenode", "instance ID": "..."}
2024-07-19T02:52:26Z    INFO    db.migration.58 Remove unused trash column      {"Process": "storagenode"}
2024-07-19T02:52:26Z    ERROR   failure during run      {"Process": "storagenode", "error": "Error migrating tables for database on storagenode: migrate: v58: database disk image is malformed\n\tstorj.io/storj/private/migrate.(*Migration).Run:212\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).MigrateToLatest:425\n\tmain.cmdRun:100\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.cleanup.func1.4:393\n\tstorj.io/common/process.cleanup.func1:411\n\tgithub.com/spf13/cobra.(*Command).execute:983\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1115\n\tgithub.com/spf13/cobra.(*Command).Execute:1039\n\tstorj.io/common/process.ExecWithCustomOptions:112\n\tmain.main:34\n\truntime.main:271", "errorVerbose": "Error migrating tables for database on storagenode: migrate: v58: database disk image is malformed\n\tstorj.io/storj/private/migrate.(*Migration).Run:212\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).MigrateToLatest:425\n\tmain.cmdRun:100\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.cleanup.func1.4:393\n\tstorj.io/common/process.cleanup.func1:411\n\tgithub.com/spf13/cobra.(*Command).execute:983\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1115\n\tgithub.com/spf13/cobra.(*Command).Execute:1039\n\tstorj.io/common/process.ExecWithCustomOptions:112\n\tmain.main:34\n\truntime.main:271\n\tmain.cmdRun:102\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.cleanup.func1.4:393\n\tstorj.io/common/process.cleanup.func1:411\n\tgithub.com/spf13/cobra.(*Command).execute:983\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1115\n\tgithub.com/spf13/cobra.(*Command).Execute:1039\n\tstorj.io/common/process.ExecWithCustomOptions:112\n\tmain.main:34\n\truntime.main:271"}
2024-07-19T02:52:26Z    FATAL   Unrecoverable error     {"Process": "storagenode", "error": "Error migrating tables for database on storagenode: migrate: v58: database disk image is malformed\n\tstorj.io/storj/private/migrate.(*Migration).Run:212\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).MigrateToLatest:425\n\tmain.cmdRun:100\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.cleanup.func1.4:393\n\tstorj.io/common/process.cleanup.func1:411\n\tgithub.com/spf13/cobra.(*Command).execute:983\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1115\n\tgithub.com/spf13/cobra.(*Command).Execute:1039\n\tstorj.io/common/process.ExecWithCustomOptions:112\n\tmain.main:34\n\truntime.main:271", "errorVerbose": "Error migrating tables for database on storagenode: migrate: v58: database disk image is malformed\n\tstorj.io/storj/private/migrate.(*Migration).Run:212\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).MigrateToLatest:425\n\tmain.cmdRun:100\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.cleanup.func1.4:393\n\tstorj.io/common/process.cleanup.func1:411\n\tgithub.com/spf13/cobra.(*Command).execute:983\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1115\n\tgithub.com/spf13/cobra.(*Command).Execute:1039\n\tstorj.io/common/process.ExecWithCustomOptions:112\n\tmain.main:34\n\truntime.main:271\n\tmain.cmdRun:102\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.cleanup.func1.4:393\n\tstorj.io/common/process.cleanup.func1:411\n\tgithub.com/spf13/cobra.(*Command).execute:983\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1115\n\tgithub.com/spf13/cobra.(*Command).Execute:1039\n\tstorj.io/common/process.ExecWithCustomOptions:112\n\tmain.main:34\n\truntime.main:271"}

nerdatwork · July 19, 2024, 3:24am

Try searching for out of memory (OOM) error by executing journalctl | grep -i oom | tail

jarridgraham · July 19, 2024, 3:28am

That did not yield anything there was 15 G free before start and went to 17 G after.

nerdatwork · July 19, 2024, 3:40am

Did you check the journal for the event at the exact time when it crashed your node ?

jarridgraham · July 19, 2024, 3:50am

I do not really see anything to odd in the journal , when it started crashing I see the network interface for docker doing up and down, I supsect it does that when a container starts. Kind of a bummer , may just try to repair all the databases anyway and see what that yeilds.

I was kind of hoping there would be more data on which database it was working on to point to a direction at least.

nerdatwork · July 19, 2024, 4:39am

IIRC the trash column is being removed from piece_expiration.db so if you can check whether its been removed or not, it might help.

jarridgraham · July 19, 2024, 5:15am

ok here is what I see in that file, if I am doing this right, been a while.

sqlite3 piece_expiration.db
SQLite version 3.40.1 2022-12-28 14:03:47
Enter ".help" for usage hints.
sqlite> .tables
piece_expirations  versions
sqlite> .schema piece_expirations
CREATE TABLE piece_expirations (
                                        satellite_id       BLOB      NOT NULL,
                                        piece_id           BLOB      NOT NULL,
                                        piece_expiration   TIMESTAMP NOT NULL, -- date when it can be deleted
                                        deletion_failed_at TIMESTAMP, trash INTEGER NOT NULL DEFAULT 0,
                                        PRIMARY KEY (satellite_id, piece_id)
                                );
CREATE INDEX idx_piece_expirations_piece_expiration ON piece_expirations(piece_expiration);
CREATE INDEX idx_piece_expirations_deletion_failed_at ON piece_expirations(deletion_failed_at);
CREATE INDEX idx_piece_expirations_trashed
                                        ON piece_expirations(satellite_id, trash)
                                        WHERE trash = 1;


sqlite> PRAGMA table_info(piece_expirations);
0|satellite_id|BLOB|1||1
1|piece_id|BLOB|1||2
2|piece_expiration|TIMESTAMP|1||0
3|deletion_failed_at|TIMESTAMP|0||0
4|trash|INTEGER|1|0|0
sqlite>

I have one node that is alrady on the latest and working, that database still has the trash column in it so I don’t know.

Alexey · July 19, 2024, 5:34am

Please make sure that you checking the correct storage directory. For the docker setup all databases are under storage subfolder in the data location.
It should apply a migration without issues (I checked my docker node and it’s successfully migrated to 1.108.3 today).

2024-07-13T04:22:50Z    INFO    db.migration    Database Version        {"Process": "storagenode", "version": 57}
2024-07-13T04:24:16Z    INFO    db.migration    Database Version        {"Process": "storagenode", "version": 57}
2024-07-19T01:39:34Z    INFO    db.migration.58 Remove unused trash column      {"Process": "storagenode"}
2024-07-19T01:40:03Z    INFO    db.migration.59 Remove unused deletion_failed_at column {"Process": "storagenode"}
2024-07-19T01:40:15Z    INFO    db.migration.60 Overhaul piece_expirations      {"Process": "storagenode"}
2024-07-19T01:40:53Z    INFO    db.migration    Database Version        {"Process": "storagenode", "version": 60}

So, it’s something with a database on this node.
You need to stop the node before the check.

agente · July 19, 2024, 7:32am

I have the same issue in one node today. Can I delete *.db and restart? I’m too lazy for repair

pangolin · July 19, 2024, 12:25pm

As far as I remember, you can only delete all databases, deleting one did not work when I tried.

I never wasting time with repairing databases. I always stop node and delete all databases when there is some corrupted database.

jarridgraham · July 19, 2024, 12:46pm

Humm, now that you mention it I believe I have been a doofus this whole time. I was in the wrong dir, I had moved them a long time ago to an ssd , there was a bad db file and now it is at least running, it seems to be very angry about trying to delete files that don’t exist and being misconfigured but will let it run a while and see what it setteles on.

Thanks again to all

Everything is finally back up, somehow in the flury of working on the wrong thing I had messed up the config.yaml where it wasn’t setting the port right but it is back up.

Whatever I have done though now shows no storage used and like it started over.

Alexey · July 20, 2024, 6:25am

Yes, you can, but I would suggest to use this guide to have a less loses:

blanaru · July 26, 2024, 9:10am

Hi Alexey,

here is the error log I got from another node

Blockquote2024-07-26T02:05:36-07:00 FATAL Unrecoverable error {“error”: “Error migrating tables for database on storag
enode: migr
ate: v58: database disk image is malformed\n\tstorj.io/storj/private/migrate.(*Migration).Run:212\n\tstorj.io/storj/sto
ragenode/storagenodedb.(*DB).MigrateToLatest:425\n\tmain.cmdRun:100\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/proce
ss.cleanup.func1.4:393\n\tstorj.io/common/process.cleanup.func1:411\n\tgithub.com/spf13/cobra.(*Command).execute:983\n
tgithub.com/spf13/cobra.(*Command).ExecuteC:1115\n\tgithub.com/spf13/cobra.(*Command).Execute:1039\n\tstorj.io/common/p
rocess.ExecWithCustomOptions:112\n\tstorj.io/common/process.ExecWithCustomConfigAndLogger:77\n\tstorj.io/common/process
.ExecWithCustomConfig:72\n\tstorj.io/common/process.Exec:62\n\tmain.(*service).Execute.func1:107\n\tgolang.org/x/sync/e
rrgroup.(*Group).Go.func1:78”, “errorVerbose”: “Error migrating tables for database on storagenode: migrate: v58: datab
ase disk image is malformed\n\tstorj.io/storj/private/migrate.(*Migration).Run:212\n\tstorj.io/storj/storagenode/storag
enodedb.(*DB).MigrateToLatest:425\n\tmain.cmdRun:100\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.cleanup.func
1.4:393\n\tstorj.io/common/process.cleanup.func1:411\n\tgithub.com/spf13/cobra.(*Command).execute:983\n\tgithub.com/spf
13/cobra.(*Command).ExecuteC:1115\n\tgithub.com/spf13/cobra.(*Command).Execute:1039\n\tstorj.io/common/process.ExecWith
CustomOptions:112\n\tstorj.io/common/process.ExecWithCustomConfigAndLogger:77\n\tstorj.io/common/process.ExecWithCustom
Config:72\n\tstorj.io/common/process.Exec:62\n\tmain.(*service).Execute.func1:107\n\tgolang.org/x/sync/errgroup.(*Group
).Go.func1:78\n\tmain.cmdRun:102\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.cleanup.func1.4:393\n\tstorj.io/
common/process.cleanup.func1:411\n\tgithub.com/spf13/cobra.(*Command).execute:983\n\tgithub.com/spf13/cobra.(*Command).
ExecuteC:1115\n\tgithub.com/spf13/cobra.(*Command).Execute:1039\n\tstorj.io/common/process.ExecWithCustomOptions:112\n
tstorj.io/common/process.ExecWithCustomConfigAndLogger:77\n\tstorj.io/common/process.ExecWithCustomConfig:72\n\tstorj.i
o/common/process.Exec:62\n\tmain.(*service).Execute.func1:107\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78”}
2024-07-26T02:06:39-07:00 FATAL Unrecoverable error {“error”: “Error migrating tables for database on storag
enode: migr
ate: v58: database disk image is malformed\n\tstorj.io/storj/private/migrate.(*Migration).Run:212\n\tstorj.io/storj/sto
ragenode/storagenodedb.(*DB).MigrateToLatest:425\n\tmain.cmdRun:100\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/proce
ss.cleanup.func1.4:393\n\tstorj.io/common/process.cleanup.func1:411\n\tgithub.com/spf13/cobra.(*Command).execute:983\n
tgithub.com/spf13/cobra.(*Command).ExecuteC:1115\n\tgithub.com/spf13/cobra.(*Command).Execute:1039\n\tstorj.io/common/p
rocess.ExecWithCustomOptions:112\n\tstorj.io/common/process.ExecWithCustomConfigAndLogger:77\n\tstorj.io/common/process
.ExecWithCustomConfig:72\n\tstorj.io/common/process.Exec:62\n\tmain.(*service).Execute.func1:107\n\tgolang.org/x/sync/e
rrgroup.(*Group).Go.func1:78”, “errorVerbose”: “Error migrating tables for database on storagenode: migrate: v58: datab
ase disk image is malformed\n\tstorj.io/storj/private/migrate.(*Migration).Run:212\n\tstorj.io/storj/storagenode/storag
enodedb.(*DB).MigrateToLatest:425\n\tmain.cmdRun:100\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.cleanup.func
1.4:393\n\tstorj.io/common/process.cleanup.func1:411\n\tgithub.com/spf13/cobra.(*Command).execute:983\n\tgithub.com/spf
13/cobra.(*Command).ExecuteC:1115\n\tgithub.com/spf13/cobra.(*Command).Execute:1039\n\tstorj.io/common/process.ExecWith
CustomOptions:112\n\tstorj.io/common/process.ExecWithCustomConfigAndLogger:77\n\tstorj.io/common/process.ExecWithCustom
Config:72\n\tstorj.io/common/process.Exec:62\n\tmain.(*service).Execute.func1:107\n\tgolang.org/x/sync/errgroup.(*Group
).Go.func1:78\n\tmain.cmdRun:102\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.cleanup.func1.4:393\n\tstorj.io/
common/process.cleanup.func1:411\n\tgithub.com/spf13/cobra.(*Command).execute:983\n\tgithub.com/spf13/cobra.(*Command).
ExecuteC:1115\n\tgithub.com/spf13/cobra.(*Command).Execute:1039\n\tstorj.io/common/process.ExecWithCustomOptions:112\n
tstorj.io/common/process.ExecWithCustomConfigAndLogger:77\n\tstorj.io/common/process.ExecWithCustomConfig:72\n\tstorj.i
o/common/process.Exec:62\n\tmain.(*service).Execute.func1:107\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78”}

nerdatwork · July 26, 2024, 10:43am

You need to fix it by following guide.

Alexey · July 27, 2024, 3:35am

Please use the code block next time (two new lines with three backticks on each) and place the excerpt inside the code block, like this:

```
2024-07-26T02:05:36-07:00 FATAL Unrecoverable error {“error”: “Error migrating tables for database on storagenode: migrate: v58: database disk image is malformed\n\tstorj.io/storj/private/migrate.(*Migration).Run:212\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).MigrateToLatest:425\n\tmain.cmdRun:100\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.cleanup.func1.4:393\n\tstorj.io/common/process.cleanup.func1:411\n\tgithub.com/spf13/cobra.*Command).execute:983\n[tgithub.com/spf13/cobra.(*Command).ExecuteC:1115](http://tgithub.com/spf13/cobra.(*Command).ExecuteC:1115)\n\tgithub.com/spf13/cobra.*Command).Execute:1039\n\tstorj.io/common/process.ExecWithCustomOptions:112\n\tstorj.io/common/process.ExecWithCustomConfigAndLogger:77\n\tstorj.io/common/process.ExecWithCustomConfig:72\n\tstorj.io/common/process.Exec:62\n\tmain.(*service).Execute.func1:107\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78”, “errorVerbose”: “Error migrating tables for database on storagenode: migrate: v58: database disk image is malformed\n\tstorj.io/storj/private/migrate.(*Migration).Run:212\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).MigrateToLatest:425\n\tmain.cmdRun:100\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.cleanup.func1.4:393\n\tstorj.io/common/process.cleanup.func1:411\n\tgithub.com/spf13/cobra.(*Command).execute:983\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1115\n\tgithub.com/spf13/cobra.(*Command).Execute:1039\n\tstorj.io/common/process.ExecWithCustomOptions:112\n\tstorj.io/common/process.ExecWithCustomConfigAndLogger:77\n\tstorj.io/common/process.ExecWithCustomConfig:72\n\tstorj.io/common/process.Exec:62\n\tmain.(*service).Execute.func1:107\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78\n\tmain.cmdRun:102\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.cleanup.func1.4:393\n\tstorj.io/common/process.cleanup.func1:411\n\tgithub.com/spf13/cobra.(*Command).execute:983\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1115\n\tgithub.com/spf13/cobra.(*Command).Execute:1039\n\tstorj.io/common/process.ExecWithCustomOptions:112\n[tstorj.io/common/process.ExecWithCustomConfigAndLogger:77](http://tstorj.io/common/process.ExecWithCustomConfigAndLogger:77)\n\tstorj.io/common/process.ExecWithCustomConfig:72\n\tstorj.io/common/process.Exec:62\n\tmain.(*service).Execute.func1:107\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78”}
```