ERROR failure during run

Node Operators troubleshooting
, , ,
1

Hi. One of my node suddently go down out og the blue.
Node running on Unraid. No powerloss or restart.
The parameters setup is right to where the config is.
Cant figure it out whats wrong.

Log shows this:

2024-08-24T20:10:44+02:00 ERROR failure during run {“Process”: “storagenode”, “error”: “Error opening database on storagenode: database: info opening file "config/storage/info.db" failed: unable to open database file: no such file or directory\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).openDatabaseWithStat:407\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).openDatabase:384\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).openExistingDatabase:379\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).openDatabases:354\n\tstorj.io/storj/storagenode/storagenodedb.OpenExisting:319\n\tmain.cmdRun:67\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.cleanup.func1.4:392\n\tstorj.io/common/process.cleanup.func1:410\n\tgithub.com/spf13/cobra.(*Command).execute:983\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1115\n\tgithub.com/spf13/cobra.(*Command).Execute:1039\n\tstorj.io/common/process.ExecWithCustomOptions:112\n\tmain.main:34\n\truntime.main:271”, “errorVerbose”: “Error opening database on storagenode: database: info opening file "config/storage/info.db" failed: unable to open database file: no such file or directory\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).openDatabaseWithStat:407\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).openDatabase:384\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).openExistingDatabase:379\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).openDatabases:354\n\tstorj.io/storj/storagenode/storagenodedb.OpenExisting:319\n\tmain.cmdRun:67\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.cleanup.func1.4:392\n\tstorj.io/common/process.cleanup.func1:410\n\tgithub.com/spf13/cobra.(*Command).execute:983\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1115\n\tgithub.com/spf13/cobra.(*Command).Execute:1039\n\tstorj.io/common/process.ExecWithCustomOptions:112\n\tmain.main:34\n\truntime.main:271\n\tmain.cmdRun:69\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.cleanup.func1.4:392\n\tstorj.io/common/process.cleanup.func1:410\n\tgithub.com/spf13/cobra.(*Command).execute:983\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1115\n\tgithub.com/spf13/cobra.(*Command).Execute:1039\n\tstorj.io/common/process.ExecWithCustomOptions:112\n\tmain.main:34\n\truntime.main:271”}
Error: Error opening database on storagenode: database: info opening file “config/storage/info.db” failed: unable to open database file: no such file or directory

2

looks like DB is corrupted or lost access to it. does your DB on DATA location or SSD?

3

It’s on data location. Docker have premission to the files

4

Hello @SRS,
Welcome back!

Could you please show the list with ls -l for the storage subfolder of your data location?

Please also run fsck or likely xfs_repair for the disk(s) of your data location.

I would also recommend to check all databases after the FS is fixed:

5

Skjermbilde 2024-08-25 150803

6

Ok. So you can safely remove the --user option in your docker run command.
However, I do not see any databases here. Did you also moved them?

7

Skjermbilde 2024-08-25 175936

Databases is under storage. Have not moved them. Always been like this. also on my other nodes.

Here is my docker run parameter:

–mount type=bind,source=“/mnt/user/Storj/id/storagenode/”,destination=/app/identity --mount type=bind,source=“/mnt/user/Storj/Storage/”,destination=/app/config --restart unless-stopped

8

Here is full logg:

2024-08-25T19:05:43+02:00       INFO    Configuration loaded    {"Process": "storagenode-updater", "Location": "/app/config/config.yaml"}
2024-08-25T19:05:43+02:00       INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "contact.external-address"}
2024-08-25T19:05:43+02:00       INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "server.private-address"}
2024-08-25T19:05:43+02:00       INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "storage.allocated-bandwidth"}
2024-08-25T19:05:43+02:00       INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "storage.allocated-disk-space"}
2024-08-25T19:05:43+02:00       INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "server.address"}
2024-08-25T19:05:43+02:00       INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "operator.email"}
2024-08-25T19:05:43+02:00       INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "server.debug-log-traffic"}
2024-08-25T19:05:43+02:00       INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "operator.wallet"}
2024-08-25T19:05:43+02:00       INFO    Anonymized tracing enabled      {"Process": "storagenode-updater"}
2024-08-25T19:05:43+02:00       INFO    Running on version      {"Process": "storagenode-updater", "Service": "storagenode-updater", "Version": "v1.111.4"}
2024-08-25T19:05:43+02:00       INFO    Downloading versions.   {"Process": "storagenode-updater", "Server Address": "https://version.storj.io"}
2024-08-25T19:05:43+02:00       INFO    Configuration loaded    {"Process": "storagenode", "Location": "/app/config/config.yaml"}
2024-08-25T19:05:43+02:00       INFO    Anonymized tracing enabled      {"Process": "storagenode"}
2024-08-25T19:05:43+02:00       INFO    Operator email  {"Process": "storagenode", "Address": "XXXX"}
2024-08-25T19:05:43+02:00       INFO    Operator wallet {"Process": "storagenode", "Address": "XXXX"}
2024-08-25T19:05:43+02:00       ERROR   failure during run      {"Process": "storagenode", "error": "Error opening database on storagenode: database: info opening file \"config/storage/info.db\" failed: unable to open database file: no such file or directory\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).openDatabaseWithStat:407\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).openDatabase:384\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).openExistingDatabase:379\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).openDatabases:354\n\tstorj.io/storj/storagenode/storagenodedb.OpenExisting:319\n\tmain.cmdRun:67\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.cleanup.func1.4:392\n\tstorj.io/common/process.cleanup.func1:410\n\tgithub.com/spf13/cobra.(*Command).execute:983\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1115\n\tgithub.com/spf13/cobra.(*Command).Execute:1039\n\tstorj.io/common/process.ExecWithCustomOptions:112\n\tmain.main:34\n\truntime.main:271", "errorVerbose": "Error opening database on storagenode: database: info opening file \"config/storage/info.db\" failed: unable to open database file: no such file or directory\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).openDatabaseWithStat:407\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).openDatabase:384\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).openExistingDatabase:379\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).openDatabases:354\n\tstorj.io/storj/storagenode/storagenodedb.OpenExisting:319\n\tmain.cmdRun:67\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.cleanup.func1.4:392\n\tstorj.io/common/process.cleanup.func1:410\n\tgithub.com/spf13/cobra.(*Command).execute:983\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1115\n\tgithub.com/spf13/cobra.(*Command).Execute:1039\n\tstorj.io/common/process.ExecWithCustomOptions:112\n\tmain.main:34\n\truntime.main:271\n\tmain.cmdRun:69\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.cleanup.func1.4:392\n\tstorj.io/common/process.cleanup.func1:410\n\tgithub.com/spf13/cobra.(*Command).execute:983\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1115\n\tgithub.com/spf13/cobra.(*Command).Execute:1039\n\tstorj.io/common/process.ExecWithCustomOptions:112\n\tmain.main:34\n\truntime.main:271"}
Error: Error opening database on storagenode: database: info opening file "config/storage/info.db" failed: unable to open database file: no such file or directory
        storj.io/storj/storagenode/storagenodedb.(*DB).openDatabaseWithStat:407
        storj.io/storj/storagenode/storagenodedb.(*DB).openDatabase:384
        storj.io/storj/storagenode/storagenodedb.(*DB).openExistingDatabase:379
        storj.io/storj/storagenode/storagenodedb.(*DB).openDatabases:354
        storj.io/storj/storagenode/storagenodedb.OpenExisting:319
        main.cmdRun:67
        main.newRunCmd.func1:33
        storj.io/common/process.cleanup.func1.4:392
        storj.io/common/process.cleanup.func1:410
        github.com/spf13/cobra.(*Command).execute:983
        github.com/spf13/cobra.(*Command).ExecuteC:1115
        github.com/spf13/cobra.(*Command).Execute:1039
        storj.io/common/process.ExecWithCustomOptions:112
        main.main:34
        runtime.main:271
2024-08-25 19:05:43,907 INFO exited: storagenode (exit status 1; not expected)

.
.
.
.
.
.
When i’m trying to take the integrity check i getting this message:
.
.
.

root@Tower:/mnt/user/Storj/Storage/storage# find /mnt/user/Storj/Storage/storage -maxdepth 1 -iname "*.db" -print0 -exec sqlite3 '{}' 'PRAGMA integrity_check;' ';'
/mnt/user/Storj/Storage/storage/secret.dbError: in prepare, unable to open database file (14)
/mnt/user/Storj/Storage/storage/reputation.dbError: in prepare, unable to open database file (14)
/mnt/user/Storj/Storage/storage/satellites.dbError: in prepare, unable to open database file (14)
/mnt/user/Storj/Storage/storage/storage_usage.dbError: in prepare, unable to open database file (14)
/mnt/user/Storj/Storage/storage/used_serial.dbError: in prepare, unable to open database file (14)
/mnt/user/Storj/Storage/storage/used_space_per_prefix.dbError: in prepare, unable to open database file (14)
/mnt/user/Storj/Storage/storage/bandwidth.dbError: in prepare, unable to open database file (14)
/mnt/user/Storj/Storage/storage/garbage_collection_filewalker_progress.dbError: in prepare, unable to open database file (14)
/mnt/user/Storj/Storage/storage/heldamount.dbError: in prepare, unable to open database file (14)
/mnt/user/Storj/Storage/storage/info.dbError: in prepare, unable to open database file (14)
/mnt/user/Storj/Storage/storage/notifications.dbError: in prepare, unable to open database file (14)
/mnt/user/Storj/Storage/storage/orders.dbError: in prepare, unable to open database file (14)
/mnt/user/Storj/Storage/storage/piece_expiration.dbError: in prepare, unable to open database file (14)
/mnt/user/Storj/Storage/storage/piece_spaced_used.dbError: in prepare, unable to open database file (14)
/mnt/user/Storj/Storage/storage/pieceinfo.dbError: in prepare, unable to open database file (14)
/mnt/user/Storj/Storage/storage/pricing.dbError: in prepare, unable to open database file (14)
9

Then I would recommend to stop the node and run the xfs_repair command for this disk, seems the filesystem is corrupted.

1 Like
10

The xfs_repair did the trick and node is up and running again. Thanks

2 Likes