Failed to create storage node peer: hashstore: invalid header:

Node Operators troubleshooting
,
1

Here we go

i think i have some corruption problem here with 2 nodes after a power loss.

is a solution possible for this?

2025-04-26T21:23:55Z FATAL Unrecoverable error {“Process”: “storagenode”, “error”: “Failed to create storage node peer: hashstore: invalid header: "\a,))"\n\tstorj.io/storj/storagenode/hashstore.ReadTblHeader:170\n\tstorj.io/storj/storagenode/hashstore.OpenTable:115\n\tstorj.io/storj/storagenode/hashstore.NewStore:258\n\tstorj.io/storj/storagenode/hashstore.New:93\n\tstorj.io/storj/storagenode/piecestore.(*HashStoreBackend).getDB:248\n\tstorj.io/storj/storagenode/piecestore.NewHashStoreBackend:114\n\tstorj.io/storj/storagenode.New:598\n\tmain.cmdRun:84\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.cleanup.func1.4:392\n\tstorj.io/common/process.cleanup.func1:410\n\tgithub.com/spf13/cobra.(*Command).execute:985\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1117\n\tgithub.com/spf13/cobra.(*Command).Execute:1041\n\tstorj.io/common/process.ExecWithCustomOptions:112\n\tmain.main:34\n\truntime.main:272”, “errorVerbose”: “Failed to create storage node peer: hashstore: invalid header: "\a,))"\n\tstorj.io/storj/storagenode/hashstore.ReadTblHeader:170\n\tstorj.io/storj/storagenode/hashstore.OpenTable:115\n\tstorj.io/storj/storagenode/hashstore.NewStore:258\n\tstorj.io/storj/storagenode/hashstore.New:93\n\tstorj.io/storj/storagenode/piecestore.(*HashStoreBackend).getDB:248\n\tstorj.io/storj/storagenode/piecestore.NewHashStoreBackend:114\n\tstorj.io/storj/storagenode.New:598\n\tmain.cmdRun:84\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.cleanup.func1.4:392\n\tstorj.io/common/process.cleanup.func1:410\n\tgithub.com/spf13/cobra.(*Command).execute:985\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1117\n\tgithub.com/spf13/cobra.(*Command).Execute:1041\n\tstorj.io/common/process.ExecWithCustomOptions:112\n\tmain.main:34\n\truntime.main:272\n\tmain.cmdRun:86\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.cleanup.func1.4:392\n\tstorj.io/common/process.cleanup.func1:410\n\tgithub.com/spf13/cobra.(*Command).execute:985\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1117\n\tgithub.com/spf13/cobra.(*Command).Execute:1041\n\tstorj.io/common/process.ExecWithCustomOptions:112\n\tmain.main:34\n\truntime.main:272”}

2

Looks like the hashtable has been corrupted, and there is no tool has been implemented yet to regenerate it from logs as far as I know.
Perhaps switch to memtable may help, but I’m not sure.
Asked the team for help.

1 Like
3

(That’s the kind of stuff I have in mind m in my rants against hashstore vs a regular, let alone CoW, filesystem).

2 Likes
4

Yes, and I didn’t migrate my nodes too, they works fine with the current piecestore, despite that they works under Windows and even worse - two of them under the Rancher Desktop. I even didn’t move DB to SSD, it’s not needed for my simple setup. Since it’s designed for Hyper-V, there is enough RAM (32GB) and CPU cores (Intel(R) Core™ i7-4770 CPU @ 3.40GHz).

1 Like
5

Couple of questions:

  1. what os/filesystem are you running? and what version of the storagenode software is running?
  2. if linux, can you give the output of find <path/to/hashstore> -type f -name 'hashtbl-*' | xargs -I {} sh -c 'xxd {} | head -n 2'? mine ends up looking like
00000000: 4854 424c 0000 4eec 0101 0000 0000 0000  HTBL..N.........
00000010: 0013 0000 0000 0000 0000 0000 0000 0000  ................
00000000: 4854 424c 0000 4eee 0101 0000 0000 0000  HTBL..N.........
00000010: 0012 0000 0000 0000 0000 0000 0000 0000  ................
00000000: 4854 424c 0000 4ede 0101 0000 0000 0000  HTBL..N.........
00000010: 000e 0000 0000 0000 0000 0000 0000 0000  ................
00000000: 4854 424c 0000 4eee 0101 0000 0000 0000  HTBL..N.........
00000010: 0011 0000 0000 0000 0000 0000 0000 0000  ................
00000000: 4854 424c 0000 4eee 0101 0000 0000 0000  HTBL..N.........
00000010: 0017 0000 0000 0000 0000 0000 0000 0000  ................
00000000: 4854 424c 0000 4eed 0101 0000 0000 0000  HTBL..N.........
00000010: 0017 0000 0000 0000 0000 0000 0000 0000  ................
00000000: 4854 424c 0000 4eee 0101 0000 0000 0000  HTBL..N.........
00000010: 0012 0000 0000 0000 0000 0000 0000 0000  ................
00000000: 4854 424c 0000 4eee 0101 0000 0000 0000  HTBL..N.........
00000010: 0012 0000 0000 0000 0000 0000 0000 0000  ................
  1. if you aren’t on linux, can you find all the files named hashtbl-<some hex digits> in your data folder and zip them up and get them to me somehow?

in the meantime, i’ll try to work on a quick program that can reconstruct hashtbls from the log files.

thanks!

4 Likes