Recovering hashstore on Windows

still getting error on 2 windows nodes (12 fixed with updates):slight_smile: 2025-12-

Summary

02T10:14:35+03:00 ERROR failure during run {“error”: “Failed to create storage node peer: hashstore: unable to create hashtbl: hashstore: The file exists.\n\tstorj.io/storj/storagenode/hashstore.NewStore.func3:243\n\tstorj.io/storj/storagenode/hashstore.NewStore:264\n\tstorj.io/storj/storagenode/hashstore.New:96\n\tstorj.io/storj/storagenode/piecestore.(*HashStoreBackend).getDB:252\n\tstorj.io/storj/storagenode/piecestore.NewHashStoreBackend:117\n\tstorj.io/storj/storagenode.New:604\n\tmain.cmdRun:84\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.InitBeforeExecute.func1.2:389\n\tstorj.io/common/process.InitBeforeExecute.func1:407\n\tgithub.com/spf13/cobra.(*Command).execute:985\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1117\n\tgithub.com/spf13/cobra.(*Command).Execute:1041\n\tstorj.io/common/process.ExecWithCustomOptions:115\n\tstorj.io/common/process.ExecWithCustomConfigAndLogger:80\n\tstorj.io/common/process.ExecWithCustomConfig:75\n\tstorj.io/common/process.Exec:65\n\tmain.(*service).Execute.func1:107\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:93”, “errorVerbose”: “Failed to create storage node peer: hashstore: unable to create hashtbl: hashstore: The file exists.\n\tstorj.io/storj/storagenode/hashstore.NewStore.func3:243\n\tstorj.io/storj/storagenode/hashstore.NewStore:264\n\tstorj.io/storj/storagenode/hashstore.New:96\n\tstorj.io/storj/storagenode/piecestore.(*HashStoreBackend).getDB:252\n\tstorj.io/storj/storagenode/piecestore.NewHashStoreBackend:117\n\tstorj.io/storj/storagenode.New:604\n\tmain.cmdRun:84\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.InitBeforeExecute.func1.2:389\n\tstorj.io/common/process.InitBeforeExecute.func1:407\n\tgithub.com/spf13/cobra.(*Command).execute:985\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1117\n\tgithub.com/spf13/cobra.(*Command).Execute:1041\n\tstorj.io/common/process.ExecWithCustomOptions:115\n\tstorj.io/common/process.ExecWithCustomConfigAndLogger:80\n\tstorj.io/common/process.ExecWithCustomConfig:75\n\tstorj.io/common/process.Exec:65\n\tmain.(*service).Execute.func1:107\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:93\n\tmain.cmdRun:86\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.InitBeforeExecute.func1.2:389\n\tstorj.io/common/process.InitBeforeExecute.func1:407\n\tgithub.com/spf13/cobra.(*Command).execute:985\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1117\n\tgithub.com/spf13/cobra.(*Command).Execute:1041\n\tstorj.io/common/process.ExecWithCustomOptions:115\n\tstorj.io/common/process.ExecWithCustomConfigAndLogger:80\n\tstorj.io/common/process.ExecWithCustomConfig:75\n\tstorj.io/common/process.Exec:65\n\tmain.(*service).Execute.func1:107\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:93”}
2025-12-02T10:14:35+03:00 FATAL Unrecoverable error {“error”: “Failed to create storage node peer: hashstore: unable to create hashtbl: hashstore: The file exists.\n\tstorj.io/storj/storagenode/hashstore.NewStore.func3:243\n\tstorj.io/storj/storagenode/hashstore.NewStore:264\n\tstorj.io/storj/storagenode/hashstore.New:96\n\tstorj.io/storj/storagenode/piecestore.(*HashStoreBackend).getDB:252\n\tstorj.io/storj/storagenode/piecestore.NewHashStoreBackend:117\n\tstorj.io/storj/storagenode.New:604\n\tmain.cmdRun:84\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.InitBeforeExecute.func1.2:389\n\tstorj.io/common/process.InitBeforeExecute.func1:407\n\tgithub.com/spf13/cobra.(*Command).execute:985\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1117\n\tgithub.com/spf13/cobra.(*Command).Execute:1041\n\tstorj.io/common/process.ExecWithCustomOptions:115\n\tstorj.io/common/process.ExecWithCustomConfigAndLogger:80\n\tstorj.io/common/process.ExecWithCustomConfig:75\n\tstorj.io/common/process.Exec:65\n\tmain.(*service).Execute.func1:107\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:93”, “errorVerbose”: “Failed to create storage node peer: hashstore: unable to create hashtbl: hashstore: The file exists.\n\tstorj.io/storj/storagenode/hashstore.NewStore.func3:243\n\tstorj.io/storj/storagenode/hashstore.NewStore:264\n\tstorj.io/storj/storagenode/hashstore.New:96\n\tstorj.io/storj/storagenode/piecestore.(*HashStoreBackend).getDB:252\n\tstorj.io/storj/storagenode/piecestore.NewHashStoreBackend:117\n\tstorj.io/storj/storagenode.New:604\n\tmain.cmdRun:84\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.InitBeforeExecute.func1.2:389\n\tstorj.io/common/process.InitBeforeExecute.func1:407\n\tgithub.com/spf13/cobra.(*Command).execute:985\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1117\n\tgithub.com/spf13/cobra.(*Command).Execute:1041\n\tstorj.io/common/process.ExecWithCustomOptions:115\n\tstorj.io/common/process.ExecWithCustomConfigAndLogger:80\n\tstorj.io/common/process.ExecWithCustomConfig:75\n\tstorj.io/common/process.Exec:65\n\tmain.(*service).Execute.func1:107\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:93\n\tmain.cmdRun:86\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.InitBeforeExecute.func1.2:389\n\tstorj.io/common/process.InitBeforeExecute.func1:407\n\tgithub.com/spf13/cobra.(*Command).execute:985\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1117\n\tgithub.com/spf13/cobra.(*Command).Execute:1041\n\tstorj.io/common/process.ExecWithCustomOptions:115\n\tstorj.io/common/process.ExecWithCustomConfigAndLogger:80\n\tstorj.io/common/process.ExecWithCustomConfig:75\n\tstorj.io/common/process.Exec:65\n\tmain.(*service).Execute.func1:107\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:93”}

What version?
Did you install this one:


its 142.5

1 Like

Shared with the team. Strange that other didn’t mention this issue, so, perhaps not related to a new version.

is it possible to check smth to start nodes? both of them are currently offline

those stack traces line up with v1.141.7, not v1.142.5+. i would try double checking how you updated/check the versions on those nodes.

1 Like

I’ve installed 1.142.4, but why can’t I open the web GUI? Has anyone else experienced this?

You need to install

I’ve tried with 142.5, still can’t open web gui and I rolled back to 141.7 (web gui opens), im on windows

I am on latest version 1.142.7 windows gui, all works OK.
Any log errors?
Are you on windows GUI or you use docker in windows?

I haven’t tried version 1.142.7 yet, but it sounds great.
I use Windows GUI (not docker).

updated to 142.7 and 143rc and 143.2- no luck
any ideas pls pls pls

hmmm, found smth interesting -both nodes have had the same folder in hashstore
hashstore\12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S\s0\meta
with following content:

after renaming folder to
hashstore\12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S-1\s0\meta

after that both nodes started (it wasn’t an issue with acl’s or corrupted disk\folder

I am currently on version 1.141.7, running on windows (not docker), using hashstore and it runs smoothly, then I tried 1.142.5 and 1.142.7 and the result is that the web gui still cannot be opened with the following error,

Summary

2025-12-04T09:52:08+07:00 ERROR failure during run {“error”: “Failed to create storage node peer: hashstore: missing hashtbl when log files exist\n\tstorj.io/storj/storagenode/hashstore.NewStore:194\n\tstorj.io/storj/storagenode/hashstore.New:96\n\tstorj.io/storj/storagenode/piecestore.(*HashStoreBackend).getDB:252\n\tstorj.io/storj/storagenode/piecestore.NewHashStoreBackend:117\n\tstorj.io/storj/storagenode.New:604\n\tmain.cmdRun:84\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.InitBeforeExecute.func1.2:389\n\tstorj.io/common/process.InitBeforeExecute.func1:407\n\tgithub.com/spf13/cobra.(*Command).execute:985\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1117\n\tgithub.com/spf13/cobra.(*Command).Execute:1041\n\tstorj.io/common/process.ExecWithCustomOptions:115\n\tstorj.io/common/process.ExecWithCustomConfigAndLogger:80\n\tstorj.io/common/process.ExecWithCustomConfig:75\n\tstorj.io/common/process.Exec:65\n\tmain.(*service).Execute.func1:107\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:93”, “errorVerbose”: “Failed to create storage node peer: hashstore: missing hashtbl when log files exist\n\tstorj.io/storj/storagenode/hashstore.NewStore:194\n\tstorj.io/storj/storagenode/hashstore.New:96\n\tstorj.io/storj/storagenode/piecestore.(*HashStoreBackend).getDB:252\n\tstorj.io/storj/storagenode/piecestore.NewHashStoreBackend:117\n\tstorj.io/storj/storagenode.New:604\n\tmain.cmdRun:84\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.InitBeforeExecute.func1.2:389\n\tstorj.io/common/process.InitBeforeExecute.func1:407\n\tgithub.com/spf13/cobra.(*Command).execute:985\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1117\n\tgithub.com/spf13/cobra.(*Command).Execute:1041\n\tstorj.io/common/process.ExecWithCustomOptions:115\n\tstorj.io/common/process.ExecWithCustomConfigAndLogger:80\n\tstorj.io/common/process.ExecWithCustomConfig:75\n\tstorj.io/common/process.Exec:65\n\tmain.(*service).Execute.func1:107\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:93\n\tmain.cmdRun:86\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.InitBeforeExecute.func1.2:389\n\tstorj.io/common/process.InitBeforeExecute.func1:407\n\tgithub.com/spf13/cobra.(*Command).execute:985\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1117\n\tgithub.com/spf13/cobra.(*Command).Execute:1041\n\tstorj.io/common/process.ExecWithCustomOptions:115\n\tstorj.io/common/process.ExecWithCustomConfigAndLogger:80\n\tstorj.io/common/process.ExecWithCustomConfig:75\n\tstorj.io/common/process.Exec:65\n\tmain.(*service).Execute.func1:107\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:93”}

2025-12-04T09:52:08+07:00 FATAL Unrecoverable error {“error”: “Failed to create storage node peer: hashstore: missing hashtbl when log files exist\n\tstorj.io/storj/storagenode/hashstore.NewStore:194\n\tstorj.io/storj/storagenode/hashstore.New:96\n\tstorj.io/storj/storagenode/piecestore.(*HashStoreBackend).getDB:252\n\tstorj.io/storj/storagenode/piecestore.NewHashStoreBackend:117\n\tstorj.io/storj/storagenode.New:604\n\tmain.cmdRun:84\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.InitBeforeExecute.func1.2:389\n\tstorj.io/common/process.InitBeforeExecute.func1:407\n\tgithub.com/spf13/cobra.(*Command).execute:985\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1117\n\tgithub.com/spf13/cobra.(*Command).Execute:1041\n\tstorj.io/common/process.ExecWithCustomOptions:115\n\tstorj.io/common/process.ExecWithCustomConfigAndLogger:80\n\tstorj.io/common/process.ExecWithCustomConfig:75\n\tstorj.io/common/process.Exec:65\n\tmain.(*service).Execute.func1:107\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:93”, “errorVerbose”: “Failed to create storage node peer: hashstore: missing hashtbl when log files exist\n\tstorj.io/storj/storagenode/hashstore.NewStore:194\n\tstorj.io/storj/storagenode/hashstore.New:96\n\tstorj.io/storj/storagenode/piecestore.(*HashStoreBackend).getDB:252\n\tstorj.io/storj/storagenode/piecestore.NewHashStoreBackend:117\n\tstorj.io/storj/storagenode.New:604\n\tmain.cmdRun:84\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.InitBeforeExecute.func1.2:389\n\tstorj.io/common/process.InitBeforeExecute.func1:407\n\tgithub.com/spf13/cobra.(*Command).execute:985\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1117\n\tgithub.com/spf13/cobra.(*Command).Execute:1041\n\tstorj.io/common/process.ExecWithCustomOptions:115\n\tstorj.io/common/process.ExecWithCustomConfigAndLogger:80\n\tstorj.io/common/process.ExecWithCustomConfig:75\n\tstorj.io/common/process.Exec:65\n\tmain.(*service).Execute.func1:107\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:93\n\tmain.cmdRun:86\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.InitBeforeExecute.func1.2:389\n\tstorj.io/common/process.InitBeforeExecute.func1:407\n\tgithub.com/spf13/cobra.(*Command).execute:985\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1117\n\tgithub.com/spf13/cobra.(*Command).Execute:1041\n\tstorj.io/common/process.ExecWithCustomOptions:115\n\tstorj.io/common/process.ExecWithCustomConfigAndLogger:80\n\tstorj.io/common/process.ExecWithCustomConfig:75\n\tstorj.io/common/process.Exec:65\n\tmain.(*service).Execute.func1:107\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:93”}

@Alexey help me

I would recommend to stop it immediately, you are risking with DQ on US1. You likely need to regenerate hashtable for s0 hash storage, otherwise the node will not be aware about pieces there and will respond with “file not found” on audit requests.

You need to regenerate it

P.S. Used Search results for 'hashstore: missing hashtbl when log files exist' - Storj Community Forum (official)

@Alexey can u kindly help with powershell command for this case?

@Alexey @elek @arrogantrabbit Guys, my node is running on Windows 10 (not docker), can you give me an easy to understand tutorial, because I’m not as expert as you in this matter, thanks

I have used the “write-tbl” tool on the 1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE\S0 satellite, the result is hashtbl 0 KB.
What should I do to recover the 1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE\S0 satellite? There is approximately 5 GB of data there.

fyi, There are no logs with a size of 0 bytes. The smallest log size is 892 KB.

Summary

|2025-12-05T11:42:59+07:00|ERROR|failure during run|{error: Failed to create storage node peer: hashstore: unable to read header: EOF\n\tstorj.io/storj/storagenode/hashstore.ReadTblHeader:169\n\tstorj.io/storj/storagenode/hashstore.OpenTable:111\n\tstorj.io/storj/storagenode/hashstore.NewStore:235\n\tstorj.io/storj/storagenode/hashstore.New:96\n\tstorj.io/storj/storagenode/piecestore.(*HashStoreBackend).getDB:252\n\tstorj.io/storj/storagenode/piecestore.NewHashStoreBackend:117\n\tstorj.io/storj/storagenode.New:604\n\tmain.cmdRun:84\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.InitBeforeExecute.func1.2:389\n\tstorj.io/common/process.InitBeforeExecute.func1:407\n\tgithub.com/spf13/cobra.(*Command).execute:985\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1117\n\tgithub.com/spf13/cobra.(*Command).Execute:1041\n\tstorj.io/common/process.ExecWithCustomOptions:115\n\tstorj.io/common/process.ExecWithCustomConfigAndLogger:80\n\tstorj.io/common/process.ExecWithCustomConfig:75\n\tstorj.io/common/process.Exec:65\n\tmain.(*service).Execute.func1:107\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:93, errorVerbose: Failed to create storage node peer: hashstore: unable to read header: EOF\n\tstorj.io/storj/storagenode/hashstore.ReadTblHeader:169\n\tstorj.io/storj/storagenode/hashstore.OpenTable:111\n\tstorj.io/storj/storagenode/hashstore.NewStore:235\n\tstorj.io/storj/storagenode/hashstore.New:96\n\tstorj.io/storj/storagenode/piecestore.(*HashStoreBackend).getDB:252\n\tstorj.io/storj/storagenode/piecestore.NewHashStoreBackend:117\n\tstorj.io/storj/storagenode.New:604\n\tmain.cmdRun:84\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.InitBeforeExecute.func1.2:389\n\tstorj.io/common/process.InitBeforeExecute.func1:407\n\tgithub.com/spf13/cobra.(*Command).execute:985\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1117\n\tgithub.com/spf13/cobra.(*Command).Execute:1041\n\tstorj.io/common/process.ExecWithCustomOptions:115\n\tstorj.io/common/process.ExecWithCustomConfigAndLogger:80\n\tstorj.io/common/process.ExecWithCustomConfig:75\n\tstorj.io/common/process.Exec:65\n\tmain.(*service).Execute.func1:107\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:93\n\tmain.cmdRun:86\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.InitBeforeExecute.func1.2:389\n\tstorj.io/common/process.InitBeforeExecute.func1:407\n\tgithub.com/spf13/cobra.(*Command).execute:985\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1117\n\tgithub.com/spf13/cobra.(*Command).Execute:1041\n\tstorj.io/common/process.ExecWithCustomOptions:115\n\tstorj.io/common/process.ExecWithCustomConfigAndLogger:80\n\tstorj.io/common/process.ExecWithCustomConfig:75\n\tstorj.io/common/process.Exec:65\n\tmain.(*service).Execute.func1:107\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:93}

|2025-12-05T11:42:59+07:00|FATAL|Unrecoverable error|{error: Failed to create storage node peer: hashstore: unable to read header: EOF\n\tstorj.io/storj/storagenode/hashstore.ReadTblHeader:169\n\tstorj.io/storj/storagenode/hashstore.OpenTable:111\n\tstorj.io/storj/storagenode/hashstore.NewStore:235\n\tstorj.io/storj/storagenode/hashstore.New:96\n\tstorj.io/storj/storagenode/piecestore.(*HashStoreBackend).getDB:252\n\tstorj.io/storj/storagenode/piecestore.NewHashStoreBackend:117\n\tstorj.io/storj/storagenode.New:604\n\tmain.cmdRun:84\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.InitBeforeExecute.func1.2:389\n\tstorj.io/common/process.InitBeforeExecute.func1:407\n\tgithub.com/spf13/cobra.(*Command).execute:985\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1117\n\tgithub.com/spf13/cobra.(*Command).Execute:1041\n\tstorj.io/common/process.ExecWithCustomOptions:115\n\tstorj.io/common/process.ExecWithCustomConfigAndLogger:80\n\tstorj.io/common/process.ExecWithCustomConfig:75\n\tstorj.io/common/process.Exec:65\n\tmain.(*service).Execute.func1:107\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:93, errorVerbose: Failed to create storage node peer: hashstore: unable to read header: EOF\n\tstorj.io/storj/storagenode/hashstore.ReadTblHeader:169\n\tstorj.io/storj/storagenode/hashstore.OpenTable:111\n\tstorj.io/storj/storagenode/hashstore.NewStore:235\n\tstorj.io/storj/storagenode/hashstore.New:96\n\tstorj.io/storj/storagenode/piecestore.(*HashStoreBackend).getDB:252\n\tstorj.io/storj/storagenode/piecestore.NewHashStoreBackend:117\n\tstorj.io/storj/storagenode.New:604\n\tmain.cmdRun:84\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.InitBeforeExecute.func1.2:389\n\tstorj.io/common/process.InitBeforeExecute.func1:407\n\tgithub.com/spf13/cobra.(*Command).execute:985\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1117\n\tgithub.com/spf13/cobra.(*Command).Execute:1041\n\tstorj.io/common/process.ExecWithCustomOptions:115\n\tstorj.io/common/process.ExecWithCustomConfigAndLogger:80\n\tstorj.io/common/process.ExecWithCustomConfig:75\n\tstorj.io/common/process.Exec:65\n\tmain.(*service).Execute.func1:107\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:93\n\tmain.cmdRun:86\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.InitBeforeExecute.func1.2:389\n\tstorj.io/common/process.InitBeforeExecute.func1:407\n\tgithub.com/spf13/cobra.(*Command).execute:985\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1117\n\tgithub.com/spf13/cobra.(*Command).Execute:1041\n\tstorj.io/common/process.ExecWithCustomOptions:115\n\tstorj.io/common/process.ExecWithCustomConfigAndLogger:80\n\tstorj.io/common/process.ExecWithCustomConfig:75\n\tstorj.io/common/process.Exec:65\n\tmain.(*service).Execute.func1:107\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:93}|

@vladro and @minerone
You need to install a go compiler and developers tools before compile the tool locally:

Then you need to do the same, as for Linux CLI:

For me the docker method is simpler and you do not need to install all dev stuff to get a working command. It independent of how you run the node - as a service or as a docker container. Here docker is not used to run the node, it’s only to build this command-line tool in a simple way.

1 Like

You need to remove the zero sized hashtable and start over. The tool is trying to reuse the existing one to continue filling it, but since it’s zero, you need to remove it before start.