Structure needs cleaning

Hi, today there was a power outage and when I start the Node I get this error and it doesn’t start.
It is a 3-year-old node and 12TB of information, I want to do everything possible to recover it.
txh

Error: Error starting master database on storagenode: database: piece_spaced_used opening file “config/storage/piece_spaced_used.db” failed: unable to open database file: structure needs cleaning

storj.io/storj/storagenode/storagenodedb.(*DB).openDatabase:347

storj.io/storj/storagenode/storagenodedb.(*DB).openExistingDatabase:324

storj.io/storj/storagenode/storagenodedb.(*DB).openDatabases:299

storj.io/storj/storagenode/storagenodedb.OpenExisting:266

main.cmdRun:62

main.newRunCmd.func1:32

storj.io/private/process.cleanup.func1.4:399

storj.io/private/process.cleanup.func1:417

github.com/spf13/cobra.(*Command).execute:852

github.com/spf13/cobra.(*Command).ExecuteC:960

github.com/spf13/cobra.(*Command).Execute:897

storj.io/private/process.ExecWithCustomOptions:113

main.main:30

runtime.main:250

2023-06-14 22:43:17,219 INFO exited: storagenode (exit status 1; not expected)

2023-06-14 22:43:18,221 INFO gave up: storagenode entered FATAL state, too many start retries too quickly

2023-06-14 22:43:18,747 WARN received SIGTERM indicating exit request

2023-06-14 22:43:18,747 INFO waiting for processes-exit-eventlistener, storagenode-updater to die

2023-06-14T22:43:18.749Z INFO Got a signal from the OS: “terminated” {“Process”: “storagenode-updater”}

2023-06-14 22:43:18,750 WARN received SIGQUIT indicating exit request

2023-06-14 22:43:18,753 INFO stopped: storagenode-updater (exit status 0)

2023-06-14 22:43:19,756 INFO stopped: processes-exit-eventlistener (terminated by SIGTERM)

Please use a UPS.

2 Likes

meaning that your filesystem is corrupted. So, before trying to fix a database you need to fix your filesystem first with fsck, the node should be stopped and the disk is unmounted.
In case of xfs you need to use a different procedure, see

After you fix the filesystem, you may check databases and fix them as well.

1 Like

Ohoh, not another database failing… :frowning:
But seems FS-related indeed.

Hi, can you give me the commands to run fsck ?

umount -A /dev/DISK
e2fsck /dev/DISK # same as fsck.ext4 /dev/DISK

sudo umount /dev/DISK
sudo fsck /dev/DISK