After reboot, the node does not start. Version 1.146.8 PLEASE HELP!

Node Operators troubleshooting
,
1

The node is running on Windows. The log shows an error:

ERROR failure during run {“error”: “Failed to create storage node peer: hashstore: missing hashtbl when log files exist\n\tstorj.io/storj/storagenode/hashstore.NewStore-range3:221\n\tstorj.io/storj/storagenode/hashstore.NewStore.NewStore.parseFiles[…].func13.func14:41\n\tpath/filepath.walkDir:310\n\tpath/filepath.walkDir:332\n\tpath/filepath.walkDir:332\n\tpath/filepath.WalkDir:400\n\tstorj.io/storj/storagenode/hashstore.NewStore.parseFiles[…].func13:27\n\tstorj.io/storj/storagenode/hashstore.NewStore:213\n\tstorj.io/storj/storagenode/hashstore.New:110\n\tstorj.io/storj/storagenode/piecestore.(*HashStoreBackend).getDB:271\n\tstorj.io/storj/storagenode/piecestore.NewHashStoreBackend:122\n\tstorj.io/storj/storagenode.New:607\n\tmain.cmdRun:84\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.InitBeforeExecute.func1.2:389\n\tstorj.io/common/process.InitBeforeExecute.func1:407\n\tgithub.com/spf13/cobra.(*Command).execute:985\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1117\n\tgithub.com/spf13/cobra.(*Command).Execute:1041\n\tstorj.io/common/process.ExecWithCustomOptions:115\n\tstorj.io/common/process.ExecWithCustomConfigAndLogger:80\n\tstorj.io/common/process.ExecWithCustomConfig:75\n\tstorj.io/common/process.Exec:65\n\tmain.(*service).Execute.func1:107\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:93”, “errorVerbose”: “Failed to create storage node peer: hashstore: missing hashtbl when log files exist\n\tstorj.io/storj/storagenode/hashstore.NewStore-range3:221\n\tstorj.io/storj/storagenode/hashstore.NewStore.NewStore.parseFiles[…].func13.func14:41\n\tpath/filepath.walkDir:310\n\tpath/filepath.walkDir:332\n\tpath/filepath.walkDir:332\n\tpath/filepath.WalkDir:400\n\tstorj.io/storj/storagenode/hashstore.NewStore.parseFiles[…].func13:27\n\tstorj.io/storj/storagenode/hashstore.NewStore:213\n\tstorj.io/storj/storagenode/hashstore.New:110\n\tstorj.io/storj/storagenode/piecestore.(*HashStoreBackend).getDB:271\n\tstorj.io/storj/storagenode/piecestore.NewHashStoreBackend:122\n\tstorj.io/storj/storagenode.New:607\n\tmain.cmdRun:84\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.InitBeforeExecute.func1.2:389\n\tstorj.io/common/process.InitBeforeExecute.func1:407\n\tgithub.com/spf13/cobra.(*Command).execute:985\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1117\n\tgithub.com/spf13/cobra.(*Command).Execute:1041\n\tstorj.io/common/process.ExecWithCustomOptions:115\n\tstorj.io/common/process.ExecWithCustomConfigAndLogger:80\n\tstorj.io/common/process.ExecWithCustomConfig:75\n\tstorj.io/common/process.Exec:65\n\tmain.(*service).Execute.func1:107\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:93\n\tmain.cmdRun:86\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.InitBeforeExecute.func1.2:389\n\tstorj.io/common/process.InitBeforeExecute.func1:407\n\tgithub.com/spf13/cobra.(*Command).execute:985\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1117\n\tgithub.com/spf13/cobra.(*Command).Execute:1041\n\tstorj.io/common/process.ExecWithCustomOptions:115\n\tstorj.io/common/process.ExecWithCustomConfigAndLogger:80\n\tstorj.io/common/process.ExecWithCustomConfig:75\n\tstorj.io/common/process.Exec:65\n\tmain.(*service).Execute.func1:107\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:93”}

FATAL Unrecoverable error {“error”: “Failed to create storage node peer: hashstore: missing hashtbl when log files exist\n\tstorj.io/storj/storagenode/hashstore.NewStore-range3:221\n\tstorj.io/storj/storagenode/hashstore.NewStore.NewStore.parseFiles[…].func13.func14:41\n\tpath/filepath.walkDir:310\n\tpath/filepath.walkDir:332\n\tpath/filepath.walkDir:332\n\tpath/filepath.WalkDir:400\n\tstorj.io/storj/storagenode/hashstore.NewStore.parseFiles[…].func13:27\n\tstorj.io/storj/storagenode/hashstore.NewStore:213\n\tstorj.io/storj/storagenode/hashstore.New:110\n\tstorj.io/storj/storagenode/piecestore.(*HashStoreBackend).getDB:271\n\tstorj.io/storj/storagenode/piecestore.NewHashStoreBackend:122\n\tstorj.io/storj/storagenode.New:607\n\tmain.cmdRun:84\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.InitBeforeExecute.func1.2:389\n\tstorj.io/common/process.InitBeforeExecute.func1:407\n\tgithub.com/spf13/cobra.(*Command).execute:985\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1117\n\tgithub.com/spf13/cobra.(*Command).Execute:1041\n\tstorj.io/common/process.ExecWithCustomOptions:115\n\tstorj.io/common/process.ExecWithCustomConfigAndLogger:80\n\tstorj.io/common/process.ExecWithCustomConfig:75\n\tstorj.io/common/process.Exec:65\n\tmain.(*service).Execute.func1:107\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:93”, “errorVerbose”: “Failed to create storage node peer: hashstore: missing hashtbl when log files exist\n\tstorj.io/storj/storagenode/hashstore.NewStore-range3:221\n\tstorj.io/storj/storagenode/hashstore.NewStore.NewStore.parseFiles[…].func13.func14:41\n\tpath/filepath.walkDir:310\n\tpath/filepath.walkDir:332\n\tpath/filepath.walkDir:332\n\tpath/filepath.WalkDir:400\n\tstorj.io/storj/storagenode/hashstore.NewStore.parseFiles[…].func13:27\n\tstorj.io/storj/storagenode/hashstore.NewStore:213\n\tstorj.io/storj/storagenode/hashstore.New:110\n\tstorj.io/storj/storagenode/piecestore.(*HashStoreBackend).getDB:271\n\tstorj.io/storj/storagenode/piecestore.NewHashStoreBackend:122\n\tstorj.io/storj/storagenode.New:607\n\tmain.cmdRun:84\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.InitBeforeExecute.func1.2:389\n\tstorj.io/common/process.InitBeforeExecute.func1:407\n\tgithub.com/spf13/cobra.(*Command).execute:985\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1117\n\tgithub.com/spf13/cobra.(*Command).Execute:1041\n\tstorj.io/common/process.ExecWithCustomOptions:115\n\tstorj.io/common/process.ExecWithCustomConfigAndLogger:80\n\tstorj.io/common/process.ExecWithCustomConfig:75\n\tstorj.io/common/process.Exec:65\n\tmain.(*service).Execute.func1:107\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:93\n\tmain.cmdRun:86\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.InitBeforeExecute.func1.2:389\n\tstorj.io/common/process.InitBeforeExecute.func1:407\n\tgithub.com/spf13/cobra.(*Command).execute:985\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1117\n\tgithub.com/spf13/cobra.(*Command).Execute:1041\n\tstorj.io/common/process.ExecWithCustomOptions:115\n\tstorj.io/common/process.ExecWithCustomConfigAndLogger:80\n\tstorj.io/common/process.ExecWithCustomConfig:75\n\tstorj.io/common/process.Exec:65\n\tmain.(*service).Execute.func1:107\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:93”}

Sorry, I don’t know how to hide the error text.

2

Acording to error, you got hashtbl missing or broken during restart you need find what sattelite is broken and s0 or s1, i think you can check them all, in each satellite there is s0 and s1 folders there you can find meta folder, there shold be hashtbl files with some aditional numbers. if no then you need to rebild it, serch on forum there is topic about it.

1 Like
3

Vadim, could you please help me in more detail in Russian? PLEASE! I think you’ve encountered this problem before.
I didn’t find the table:
12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S
12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs
121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6

4

There was some kind of tool posted here for restoring tables, but I don’t know how to use it on Windows.
https://github.com/elek/storj-write-hashtbl/releases/tag/v1.0.0

5

use this instruction from Alexey
Unrecoverable error - Node Operators / troubleshooting - Storj Community Forum (official)

1 Like
6

I’ve started recreating the tables. But it’s going so slowly that I’m afraid it will take a week for three satellites (s1 / s0). The disk read speed is 500 KB/sec.
Is it possible to speed up the table creation process?
I use utility @https://github.com/elek/storj-write-hashtbl/releases/tag/v1.0.0@

7

No, it’s limited by disk seek latency. But there is no hurry. If it takes a week — it takes a week.

Also you need to find out what caused data loss. Assuming you have UPS, likely reason is an abrupt reboot — investigate why.

8

The node was rebooted via “shutdown.” That’s the strange part of the situation.
@elek

9

It’s better to use the latest version from main:

10

I run it and get an error.

write-hashtbl -f -s 24 x:!STORJ920\STORAGE-920\hashstore\12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S\s0
Using logSlots=24
The file exists.
main.(*cmdRoot).Execute:97
github.com/zeebo/clingy.(*Environment).dispatchDesc:129
github.com/zeebo/clingy.Environment.Run:41
main.main:30
runtime.main:290

But there are no old tables at the specified path.

11

It generates a new hashtbl in the current directory, so you need to remove a previous one before starting to generate a new one.
You also should move the generated table to the directory for it and remove old hashtables from there, if they are exist. However, if they are already exist there, you probably shouldn’t regenerate them.
Or switch the current directory to the right location with cd /d x:\STORJ920\STORAGE-920\hashstore.....meta....\s0 command before run the tool. Just make sure that you removed old hashtbl* from there.

13

Understood

Is it necessary to use the “-f -s **” option to run?

What is this parameter for?

14

You can see a short help with
write-hashtbl -h
See also

15

@Alexey
Errors occurred while restoring the table:

record at offset=135195648 in x:!STORJ920\STORAGE-920\hashstore\121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6\s0\c4\log-00000000000000c4-00000000 has invalid piece hash=76b143f91da35e6970d7b588517b7dc972551955f9eef2f240ae9e5eeeb669d0

" so the --fast flag causes it to skip piece integrity checks. it can make the table faster, but it might include pieces that are bad due to corruption or whatever."

What if I use the “–fast” parameter, but still get these errors? The table file not build, or won’t work when I start the node?

16

At the moment the tool doesn’t have a recover functionality. This log file is corrupted. Even if it has some healthy pieces, there is no way currently to save what’s healthy.

Perhaps Windows is a worst OS for hashstore at the moment. It corrupts files.

17

So will the node start after restoring the tables if there were similar errors during the restore? (I am currently NOT using the --fast parameter)

18

If the hashtbl file will be created and be placed to a right directory - it should.

19

Ok.
And if I use the --fast parameter to speed up the process, will this prevent the created table from starting? Or does using this parameter have no effect (from a table creation perspective)?

рус

А если использовать параметр --fast для ускорения процесса - это может повлиять на то что созданная таблица не запустится, учитывая подобные ошибки? Или применение параметра ни на что не влияет (с точки зрения создания таблицы)?

20

It should skip unrecoverable pieces and process them as is. The compaction may throw errors on them later.

на русском

Этот параметр пропустит ошибочные кусочки и занесёт в таблицу в том числе не корректные.
При сжатии, скорее всего, будут возникать ошибки.

21

Is it possible to downgrade a node from 1.146.8 to 1.144.3?
The thing is, I have many nodes running 1.144.3 that are working fine and haven’t lost any tables. There was only one node running 1.146.8, and the tables were deleted on it.
I read the forum and found that Vadim also lost tables on 1.146. So, I don’t want to restart the node on 1.146 after restoring the tables.

1 Like