Piece_expiration.db damaged

Node Operators troubleshooting
1

Well it was fun while it lasted.
A reboot of my machine, and my node took suicide.
The file is there, and the drive works as normal. Did it kill it self?

2022-05-08T09:49:52.357+0200 FATAL Unrecoverable error {“error”: “Error starting master database on storagenode: database: piece_expiration couldn’t be read (“E:\\Storjdata\\piece_expiration.db”): CreateFile E:\Storjdata\piece_expiration.db: Filen eller mappen er skadet og kan ikke leses.\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).openExistingDatabase:303\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).openDatabases:281\n\tstorj.io/storj/storagenode/storagenodedb.OpenExisting:248\n\tmain.cmdRun:193\n\tstorj.io/private/process.cleanup.func1.4:372\n\tstorj.io/private/process.cleanup.func1:390\n\tgithub.com/spf13/cobra.(*Command).execute:852\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:960\n\tgithub.com/spf13/cobra.(*Command).Execute:897\n\tstorj.io/private/process.ExecWithCustomConfigAndLogger:93\n\tstorj.io/private/process.ExecWithCustomConfig:75\n\tstorj.io/private/process.Exec:65\n\tmain.(*service).Execute.func1:61\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”, “errorVerbose”: “Error starting master database on storagenode: database: piece_expiration couldn’t be read (“E:\\Storjdata\\piece_expiration.db”): CreateFile E:\Storjdata\piece_expiration.db: Filen eller mappen er skadet og kan ikke leses.\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).openExistingDatabase:303\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).openDatabases:281\n\tstorj.io/storj/storagenode/storagenodedb.OpenExisting:248\n\tmain.cmdRun:193\n\tstorj.io/private/process.cleanup.func1.4:372\n\tstorj.io/private/process.cleanup.func1:390\n\tgithub.com/spf13/cobra.(*Command).execute:852\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:960\n\tgithub.com/spf13/cobra.(*Command).Execute:897\n\tstorj.io/private/process.ExecWithCustomConfigAndLogger:93\n\tstorj.io/private/process.ExecWithCustomConfig:75\n\tstorj.io/private/process.Exec:65\n\tmain.(*service).Execute.func1:61\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57\n\tmain.cmdRun:195\n\tstorj.io/private/process.cleanup.func1.4:372\n\tstorj.io/private/process.cleanup.func1:390\n\tgithub.com/spf13/cobra.(*Command).execute:852\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:960\n\tgithub.com/spf13/cobra.(*Command).Execute:897\n\tstorj.io/private/process.ExecWithCustomConfigAndLogger:93\n\tstorj.io/private/process.ExecWithCustomConfig:75\n\tstorj.io/private/process.Exec:65\n\tmain.(*service).Execute.func1:61\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}

2

Please stop your storagenode service either from the Services applet or from the elevated (run as Admin) PowerShell:

Stop-Service storagenode

Then check your disk for errors (run as admin):

chkdsk E:\ /f

You maybe forced to run it several times until it would not throw any errors.

3

Done it twice now, it threw some errors, but not many. Quite surprised actually as the drive is quite new.

Is my node dead now? It gives me the same errors after chkdsk

4

Actually my node gave me a different error.

2022-05-08T15:26:29.498+0200 FATAL Unrecoverable error {“error”: “Error creating tables for master database on storagenode: migrate: no such table: piece_expirations\n\tstorj.io/storj/private/migrate.SQL.Run:296\n\tstorj.io/storj/private/migrate.(*Migration).Run.func1:197\n\tstorj.io/private/dbutil/txutil.withTxOnce:75\n\tstorj.io/private/dbutil/txutil.WithTx:36\n\tstorj.io/storj/private/migrate.(*Migration).Run:196\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).MigrateToLatest:347\n\tmain.cmdRun:226\n\tstorj.io/private/process.cleanup.func1.4:372\n\tstorj.io/private/process.cleanup.func1:390\n\tgithub.com/spf13/cobra.(*Command).execute:852\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:960\n\tgithub.com/spf13/cobra.(*Command).Execute:897\n\tstorj.io/private/process.ExecWithCustomConfigAndLogger:93\n\tstorj.io/private/process.ExecWithCustomConfig:75\n\tstorj.io/private/process.Exec:65\n\tmain.(*service).Execute.func1:61\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”, “errorVerbose”: “Error creating tables for master database on storagenode: migrate: no such table: piece_expirations\n\tstorj.io/storj/private/migrate.SQL.Run:296\n\tstorj.io/storj/private/migrate.(*Migration).Run.func1:197\n\tstorj.io/private/dbutil/txutil.withTxOnce:75\n\tstorj.io/private/dbutil/txutil.WithTx:36\n\tstorj.io/storj/private/migrate.(*Migration).Run:196\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).MigrateToLatest:347\n\tmain.cmdRun:226\n\tstorj.io/private/process.cleanup.func1.4:372\n\tstorj.io/private/process.cleanup.func1:390\n\tgithub.com/spf13/cobra.(*Command).execute:852\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:960\n\tgithub.com/spf13/cobra.(*Command).Execute:897\n\tstorj.io/private/process.ExecWithCustomConfigAndLogger:93\n\tstorj.io/private/process.ExecWithCustomConfig:75\n\tstorj.io/private/process.Exec:65\n\tmain.(*service).Execute.func1:61\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57\n\tmain.cmdRun:228\n\tstorj.io/private/process.cleanup.func1.4:372\n\tstorj.io/private/process.cleanup.func1:390\n\tgithub.com/spf13/cobra.(*Command).execute:852\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:960\n\tgithub.com/spf13/cobra.(*Command).Execute:897\n\tstorj.io/private/process.ExecWithCustomConfigAndLogger:93\n\tstorj.io/private/process.ExecWithCustomConfig:75\n\tstorj.io/private/process.Exec:65\n\tmain.(*service).Execute.func1:61\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}

5

So seems the chkdsk fixed some errors, but database file is actually damaged.
You can re-create it following this guide: https://support.storj.io/hc/en-us/articles/4403032417044-How-to-fix-database-file-is-not-a-database-error

The database could be corrupted, if there was an unexpected restart of your PC, maybe because of temporary power loss.
I would expect that you could have more errors or maybe even data corruption (the future will show).
How is this disk connected to your PC? If it’s USB, make sure that your drive have an external power supply, the USB is not enough. And also usually USB controllers are not reliable enough, they could overheat and disconnect your drive during the high load. Consider to connect it as internal drive instead.

6

That made the trick. It is back online.

It is a USB drive as you mention, but it has external power.
No powerloss registered, as the drives are powered on UPS and the “server” is a laptop with healthy battery.

Well, i will have to monitor this one, maybe i have a drive going bad. Quite dissapointing regarding its age of under 6 months.

Fun fact: This issue made me aware of the size of the logfile… Is it really known that these get really big? Mine was 10GB more or less.

7

Since it’s an USB drive, it can disconnect because of its USB controller. Most of external USB disks are not designed to work 24/7.
And the external power supply could be a low quality, especially USB hubs.
The much rare case that drive is going bad, but possible too.

To cut the log you can stop the storagenode service and rename the current log file, then start the node back - it will create a new log file much less size. The old log you can archive for future references (may be not needed).

1 Like