Fatal unrecoverable

Node Operators troubleshooting
1

hai guys…need help here…what the meaning line here…??

Unrecoverable error {“error”: “Error during preflight check for storagenode databases: preflight: database "heldamount": failed drop if test_table: database disk image is malformed\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).preflight:422\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).Preflight:360\n\tmain.cmdRun:241\n\tstorj.io/private/process.cleanup.func1.4:377\n\tstorj.io/private/process.cleanup.func1:395\n\tgithub.com/spf13/cobra.(*Command).execute:852\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:960\n\tgithub.com/spf13/cobra.(*Command).Execute:897\n\tstorj.io/private/process.ExecWithCustomConfigAndLogger:92\n\tstorj.io/private/process.ExecWithCustomConfig:74\n\tstorj.io/private/process.Exec:64\n\tmain.(*service).Execute.func1:61\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75”, “errorVerbose”: “Error during preflight check for storagenode databases: preflight: database "heldamount": failed drop if test_table: database disk image is malformed\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).preflight:422\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).Preflight:360\n\tmain.cmdRun:241\n\tstorj.io/private/process.cleanup.func1.4:377\n\tstorj.io/private/process.cleanup.func1:395\n\tgithub.com/spf13/cobra.(*Command).execute:852\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:960\n\tgithub.com/spf13/cobra.(*Command).Execute:897\n\tstorj.io/private/process.ExecWithCustomConfigAndLogger:92\n\tstorj.io/private/process.ExecWithCustomConfig:74\n\tstorj.io/private/process.Exec:64\n\tmain.(*service).Execute.func1:61\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75\n\tmain.cmdRun:243\n\tstorj.io/private/process.cleanup.func1.4:377\n\tstorj.io/private/process.cleanup.func1:395\n\tgithub.com/spf13/cobra.(*Command).execute:852\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:960\n\tgithub.com/spf13/cobra.(*Command).Execute:897\n\tstorj.io/private/process.ExecWithCustomConfigAndLogger:92\n\tstorj.io/private/process.ExecWithCustomConfig:74\n\tstorj.io/private/process.Exec:64\n\tmain.(*service).Execute.func1:61\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75”}

My node have offline for 4 days…please help me…

2

You need to fix your databases…

2 Likes
3

im using windows OS …i dont no what to do…help me

4

i have fix “heldamount.db”…when i start my node…i get this error then…

Unrecoverable error {“error”: “Error creating tables for master database on storagenode: migrate: v32: table paystubs already exists\n\tstorj.io/storj/private/migrate.(*Migration).Run:209\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).MigrateToLatest:354\n\tmain.cmdRun:226\n\tstorj.io/private/process.cleanup.func1.4:377\n\tstorj.io/private/process.cleanup.func1:395\n\tgithub.com/spf13/cobra.(*Command).execute:852\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:960\n\tgithub.com/spf13/cobra.(*Command).Execute:897\n\tstorj.io/private/process.ExecWithCustomConfigAndLogger:92\n\tstorj.io/private/process.ExecWithCustomConfig:74\n\tstorj.io/private/process.Exec:64\n\tmain.(*service).Execute.func1:61\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75”, “errorVerbose”: “Error creating tables for master database on storagenode: migrate: v32: table paystubs already exists\n\tstorj.io/storj/private/migrate.(*Migration).Run:209\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).MigrateToLatest:354\n\tmain.cmdRun:226\n\tstorj.io/private/process.cleanup.func1.4:377\n\tstorj.io/private/process.cleanup.func1:395\n\tgithub.com/spf13/cobra.(*Command).execute:852\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:960\n\tgithub.com/spf13/cobra.(*Command).Execute:897\n\tstorj.io/private/process.ExecWithCustomConfigAndLogger:92\n\tstorj.io/private/process.ExecWithCustomConfig:74\n\tstorj.io/private/process.Exec:64\n\tmain.(*service).Execute.func1:61\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75\n\tmain.cmdRun:228\n\tstorj.io/private/process.cleanup.func1.4:377\n\tstorj.io/private/process.cleanup.func1:395\n\tgithub.com/spf13/cobra.(*Command).execute:852\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:960\n\tgithub.com/spf13/cobra.(*Command).Execute:897\n\tstorj.io/private/process.ExecWithCustomConfigAndLogger:92\n\tstorj.io/private/process.ExecWithCustomConfig:74\n\tstorj.io/private/process.Exec:64\n\tmain.(*service).Execute.func1:61\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75”}

5

Check the other database files.

6

orait…on it…tq very much

7

i have check all database like this…

PS D:\storj> Get-ChildItem D:\storj*.db -File | %{$.Name + " " + $(sqlite3.exe $.FullName “PRAGMA integrity_check;”)}

bandwidth.db ok
heldamount.db ok
info.db ok
notifications.db ok
orders.db ok
pieceinfo.db ok
piece_expiration.db ok
piece_spaced_used.db ok
pricing.db ok
reputation.db ok
satellites.db ok
secret.db ok
storage_usage.db ok
used_serial.db ok

but i still got an error like this…

FATAL Unrecoverable error {“error”: “Error creating tables for master database on storagenode: migrate: v26: duplicate column name: trash\n\tstorj.io/storj/private/migrate.(*Migration).Run:209\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).MigrateToLatest:354\n\tmain.cmdRun:226\n\tstorj.io/private/process.cleanup.func1.4:377\n\tstorj.io/private/process.cleanup.func1:395\n\tgithub.com/spf13/cobra.(*Command).execute:852\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:960\n\tgithub.com/spf13/cobra.(*Command).Execute:897\n\tstorj.io/private/process.ExecWithCustomConfigAndLogger:92\n\tstorj.io/private/process.ExecWithCustomConfig:74\n\tstorj.io/private/process.Exec:64\n\tmain.(*service).Execute.func1:61\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75”, “errorVerbose”: “Error creating tables for master database on storagenode: migrate: v26: duplicate column name: trash\n\tstorj.io/storj/private/migrate.(*Migration).Run:209\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).MigrateToLatest:354\n\tmain.cmdRun:226\n\tstorj.io/private/process.cleanup.func1.4:377\n\tstorj.io/private/process.cleanup.func1:395\n\tgithub.com/spf13/cobra.(*Command).execute:852\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:960\n\tgithub.com/spf13/cobra.(*Command).Execute:897\n\tstorj.io/private/process.ExecWithCustomConfigAndLogger:92\n\tstorj.io/private/process.ExecWithCustomConfig:74\n\tstorj.io/private/process.Exec:64\n\tmain.(*service).Execute.func1:61\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75\n\tmain.cmdRun:228\n\tstorj.io/private/process.cleanup.func1.4:377\n\tstorj.io/private/process.cleanup.func1:395\n\tgithub.com/spf13/cobra.(*Command).execute:852\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:960\n\tgithub.com/spf13/cobra.(*Command).Execute:897\n\tstorj.io/private/process.ExecWithCustomConfigAndLogger:92\n\tstorj.io/private/process.ExecWithCustomConfig:74\n\tstorj.io/private/process.Exec:64\n\tmain.(*service).Execute.func1:61\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75”}

wht i have to do?

8

If all the databases are showing ok that means there’s no corruption but some formatting isn’t correct. At this point I’d suggest moving all the databases to a different folder location and then starting the node. All databases will be recreated from scratch.

If you have time you can then stop the node and try swapping out different databases instead of the new ‘blank’ ones, then start the node and see if it runs. However running with the blank databases is fine, you will only lose all statistics about the node; as long as you still have all the data the node will rebuild and reindex the files.

2 Likes
9

TQ @Stob i run new blank database on node has online but total used storage seem not right…any solution?

10

Only wait when the filewalker will update an empty database.

11

owh…just wait only…ok tq @Alexey @Stob im really appreaciate for your assistance…

1 Like
12

after 7 hours node is run up…i got this error…

ERROR piecestore:cache error getting current used space: {“error”: “CreateFile D:\storj\blobs\qstuylguhrn2ozjv4h2c6xpxykd622gtgurhql2k7k75wqaaaaaa\je\vicgkqlirkzjshbgmpjawfgzbv2c5u2zj5yz72kpjtf5kbap7q.sj1: The file or directory is corrupted and unreadable.; CreateFile D:\storj\blobs\ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa\23\cudwpqycid4htiete2idmcqjhry7nomc3v4ptt5qxcblucw2xq.sj1: The file or directory is corrupted and unreadable.; CreateFile D:\storj\blobs\v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa\34\7vdr5jqfthepagzuyk7fyqnteeh6krfksepvek4x3yyoxnfvua.sj1: The file or directory is corrupted and unreadable.”, “errorVerbose”: “group:\n— CreateFile D:\storj\blobs\qstuylguhrn2ozjv4h2c6xpxykd622gtgurhql2k7k75wqaaaaaa\je\vicgkqlirkzjshbgmpjawfgzbv2c5u2zj5yz72kpjtf5kbap7q.sj1: The file or directory is corrupted and unreadable.\n— CreateFile D:\storj\blobs\ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa\23\cudwpqycid4htiete2idmcqjhry7nomc3v4ptt5qxcblucw2xq.sj1: The file or directory is corrupted and unreadable.\n— CreateFile D:\storj\blobs\v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa\34\7vdr5jqfthepagzuyk7fyqnteeh6krfksepvek4x3yyoxnfvua.sj1: The file or directory is corrupted and unreadable.”}

13

You should run chkdsk with fix on the D: volume.

1 Like
14

use command line like this right?
chkdsk /f
am i right?

15

Yeah, in cmd

chkdsk /f d:

1 Like
16

after run a check disk, i still got same error like this…

ERROR piecestore:cache error getting current used space: {“error”: “CreateFile D:\storj\blobs\qstuylguhrn2ozjv4h2c6xpxykd622gtgurhql2k7k75wqaaaaaa\je\vicgkqlirkzjshbgmpjawfgzbv2c5u2zj5yz72kpjtf5kbap7q.sj1: The file or directory is corrupted and unreadable.; CreateFile D:\storj\blobs\ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa\23\cudwpqycid4htiete2idmcqjhry7nomc3v4ptt5qxcblucw2xq.sj1: The file or directory is corrupted and unreadable.; CreateFile D:\storj\blobs\v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa\34\7vdr5jqfthepagzuyk7fyqnteeh6krfksepvek4x3yyoxnfvua.sj1: The file or directory is corrupted and unreadable.”, “errorVerbose”: “group:\n— CreateFile D:\storj\blobs\qstuylguhrn2ozjv4h2c6xpxykd622gtgurhql2k7k75wqaaaaaa\je\vicgkqlirkzjshbgmpjawfgzbv2c5u2zj5yz72kpjtf5kbap7q.sj1: The file or directory is corrupted and unreadable.\n— CreateFile D:\storj\blobs\ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa\23\cudwpqycid4htiete2idmcqjhry7nomc3v4ptt5qxcblucw2xq.sj1: The file or directory is corrupted and unreadable.\n— CreateFile D:\storj\blobs\v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa\34\7vdr5jqfthepagzuyk7fyqnteeh6krfksepvek4x3yyoxnfvua.sj1: The file or directory is corrupted and unreadable.”}

17

Run chkdsk again until no more errors.

1 Like