Recovering hashstore on Windows

What if you run with -f -s 21. Skips some checks, and does not do count pass.

It could be a file hashtbl in the directory, where you run the command

now node that gone live, gone back offline with this eeror
what this error mean for me?
2026-01-30T13:07:47+02:00 FATAL Unrecoverable error {“error”: “Failed to create storage node peer: hashstore: The file exists.\n\tstorj.io/storj/storagenode/hashstore.newAtomicFile:218\n\tstorj.io/storj/storagenode/hashstore.(*Store).reconcileLog:1575\n\tstorj.io/storj/storagenode/hashstore.NewStore.func6:324\n\tstorj.io/storj/storagenode/hashstore.NewStore-range4:344\n\tstorj.io/storj/storagenode/hashstore.NewStore.NewStore.parseFiles[…].func15.func16:41\n\tpath/filepath.walkDir:310\n\tpath/filepath.walkDir:332\n\tpath/filepath.walkDir:332\n\tpath/filepath.WalkDir:400\n\tstorj.io/storj/storagenode/hashstore.NewStore.parseFiles[…].func15:27\n\tstorj.io/storj/storagenode/hashstore.NewStore:271\n\tstorj.io/storj/storagenode/hashstore.New:122\n\tstorj.io/storj/storagenode/piecestore.(*HashStoreBackend).getDB:271\n\tstorj.io/storj/storagenode/piecestore.NewHashStoreBackend:122\n\tstorj.io/storj/storagenode.New:607\n\tmain.cmdRun:84\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.InitBeforeExecute.func1.2:389\n\tstorj.io/common/process.InitBeforeExecute.func1:407\n\tgithub.com/spf13/cobra.(*Command).execute:985\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1117\n\tgithub.com/spf13/cobra.(*Command).Execute:1041\n\tstorj.io/common/process.ExecWithCustomOptions:115\n\tstorj.io/common/process.ExecWithCustomConfigAndLogger:80\n\tstorj.io/common/process.ExecWithCustomConfig:75\n\tstorj.io/common/process.Exec:65\n\tmain.(*service).Execute.func1:107\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:93”, “errorVerbose”: “Failed to create storage node peer: hashstore: The file exists.\n\tstorj.io/storj/storagenode/hashstore.newAtomicFile:218\n\tstorj.io/storj/storagenode/hashstore.(*Store).reconcileLog:1575\n\tstorj.io/storj/storagenode/hashstore.NewStore.func6:324\n\tstorj.io/storj/storagenode/hashstore.NewStore-range4:344\n\tstorj.io/storj/storagenode/hashstore.NewStore.NewStore.parseFiles[…].func15.func16:41\n\tpath/filepath.walkDir:310\n\tpath/filepath.walkDir:332\n\tpath/filepath.walkDir:332\n\tpath/filepath.WalkDir:400\n\tstorj.io/storj/storagenode/hashstore.NewStore.parseFiles[…].func15:27\n\tstorj.io/storj/storagenode/hashstore.NewStore:271\n\tstorj.io/storj/storagenode/hashstore.New:122\n\tstorj.io/storj/storagenode/piecestore.(*HashStoreBackend).getDB:271\n\tstorj.io/storj/storagenode/piecestore.NewHashStoreBackend:122\n\tstorj.io/storj/storagenode.New:607\n\tmain.cmdRun:84\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.InitBeforeExecute.func1.2:389\n\tstorj.io/common/process.InitBeforeExecute.func1:407\n\tgithub.com/spf13/cobra.(*Command).execute:985\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1117\n\tgithub.com/spf13/cobra.(*Command).Execute:1041\n\tstorj.io/common/process.ExecWithCustomOptions:115\n\tstorj.io/common/process.ExecWithCustomConfigAndLogger:80\n\tstorj.io/common/process.ExecWithCustomConfig:75\n\tstorj.io/common/process.Exec:65\n\tmain.(*service).Execute.func1:107\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:93\n\tmain.cmdRun:86\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.InitBeforeExecute.func1.2:389\n\tstorj.io/common/process.InitBeforeExecute.func1:407\n\tgithub.com/spf13/cobra.(*Command).execute:985\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1117\n\tgithub.com/spf13/cobra.(*Command).Execute:1041\n\tstorj.io/common/process.ExecWithCustomOptions:115\n\tstorj.io/common/process.ExecWithCustomConfigAndLogger:80\n\tstorj.io/common/process.ExecWithCustomConfig:75\n\tstorj.io/common/process.Exec:65\n\tmain.(*service).Execute.func1:107\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:93”}

Update>
with 1.146.7 I started to get also, on one additional node.
It is nightmare to debug error like this, because we have 4 satelites S0 and S1 in each with own hashtable which of them need to rebild?

Update2> I checked visually all tables and almost all of them are gone, i do not understand how it can be before update node worked. now i know what to do, rebilding them :slight_smile:

2026-01-30T20:43:20+02:00 FATAL Unrecoverable error {“error”: “Failed to create storage node peer: hashstore: missing hashtbl when log files exist\n\tstorj.io/storj/storagenode/hashstore.NewStore-range3:221\n\tstorj.io/storj/storagenode/hashstore.NewStore.NewStore.parseFiles[…].func13.func14:41\n\tpath/filepath.walkDir:310\n\tpath/filepath.walkDir:332\n\tpath/filepath.walkDir:332\n\tpath/filepath.WalkDir:400\n\tstorj.io/storj/storagenode/hashstore.NewStore.parseFiles[…].func13:27\n\tstorj.io/storj/storagenode/hashstore.NewStore:213\n\tstorj.io/storj/storagenode/hashstore.New:110\n\tstorj.io/storj/storagenode/piecestore.(*HashStoreBackend).getDB:271\n\tstorj.io/storj/storagenode/piecestore.NewHashStoreBackend:122\n\tstorj.io/storj/storagenode.New:607\n\tmain.cmdRun:84\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.InitBeforeExecute.func1.2:389\n\tstorj.io/common/process.InitBeforeExecute.func1:407\n\tgithub.com/spf13/cobra.(*Command).execute:985\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1117\n\tgithub.com/spf13/cobra.(*Command).Execute:1041\n\tstorj.io/common/process.ExecWithCustomOptions:115\n\tstorj.io/common/process.ExecWithCustomConfigAndLogger:80\n\tstorj.io/common/process.ExecWithCustomConfig:75\n\tstorj.io/common/process.Exec:65\n\tmain.(*service).Execute.func1:107\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:93”, “errorVerbose”: “Failed to create storage node peer: hashstore: missing hashtbl when log files exist\n\tstorj.io/storj/storagenode/hashstore.NewStore-range3:221\n\tstorj.io/storj/storagenode/hashstore.NewStore.NewStore.parseFiles[…].func13.func14:41\n\tpath/filepath.walkDir:310\n\tpath/filepath.walkDir:332\n\tpath/filepath.walkDir:332\n\tpath/filepath.WalkDir:400\n\tstorj.io/storj/storagenode/hashstore.NewStore.parseFiles[…].func13:27\n\tstorj.io/storj/storagenode/hashstore.NewStore:213\n\tstorj.io/storj/storagenode/hashstore.New:110\n\tstorj.io/storj/storagenode/piecestore.(*HashStoreBackend).getDB:271\n\tstorj.io/storj/storagenode/piecestore.NewHashStoreBackend:122\n\tstorj.io/storj/storagenode.New:607\n\tmain.cmdRun:84\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.InitBeforeExecute.func1.2:389\n\tstorj.io/common/process.InitBeforeExecute.func1:407\n\tgithub.com/spf13/cobra.(*Command).execute:985\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1117\n\tgithub.com/spf13/cobra.(*Command).Execute:1041\n\tstorj.io/common/process.ExecWithCustomOptions:115\n\tstorj.io/common/process.ExecWithCustomConfigAndLogger:80\n\tstorj.io/common/process.ExecWithCustomConfig:75\n\tstorj.io/common/process.Exec:65\n\tmain.(*service).Execute.func1:107\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:93\n\tmain.cmdRun:86\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.InitBeforeExecute.func1.2:389\n\tstorj.io/common/process.InitBeforeExecute.func1:407\n\tgithub.com/spf13/cobra.(*Command).execute:985\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1117\n\tgithub.com/spf13/cobra.(*Command).Execute:1041\n\tstorj.io/common/process.ExecWithCustomOptions:115\n\tstorj.io/common/process.ExecWithCustomConfigAndLogger:80\n\tstorj.io/common/process.ExecWithCustomConfig:75\n\tstorj.io/common/process.Exec:65\n\tmain.(*service).Execute.func1:107\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:93”}

elek, It looks like I need this tool to help me recover tables. Are there any instructions on how to use “write-hashtbl_1.0.0_windows_amd64.exe”? What commands/keys for run should I use and where should I run it?

Unrecoverable error - Node Operators / troubleshooting - Storj Community Forum (official)

1 Like