Failed to create storage node peer: hashstore: invalid header:

javierxam · April 26, 2025, 9:27pm

Here we go

i think i have some corruption problem here with 2 nodes after a power loss.

is a solution possible for this?

2025-04-26T21:23:55Z FATAL Unrecoverable error {“Process”: “storagenode”, “error”: “Failed to create storage node peer: hashstore: invalid header: "\a,))"\n\tstorj.io/storj/storagenode/hashstore.ReadTblHeader:170\n\tstorj.io/storj/storagenode/hashstore.OpenTable:115\n\tstorj.io/storj/storagenode/hashstore.NewStore:258\n\tstorj.io/storj/storagenode/hashstore.New:93\n\tstorj.io/storj/storagenode/piecestore.(*HashStoreBackend).getDB:248\n\tstorj.io/storj/storagenode/piecestore.NewHashStoreBackend:114\n\tstorj.io/storj/storagenode.New:598\n\tmain.cmdRun:84\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.cleanup.func1.4:392\n\tstorj.io/common/process.cleanup.func1:410\n\tgithub.com/spf13/cobra.(*Command).execute:985\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1117\n\tgithub.com/spf13/cobra.(*Command).Execute:1041\n\tstorj.io/common/process.ExecWithCustomOptions:112\n\tmain.main:34\n\truntime.main:272”, “errorVerbose”: “Failed to create storage node peer: hashstore: invalid header: "\a,))"\n\tstorj.io/storj/storagenode/hashstore.ReadTblHeader:170\n\tstorj.io/storj/storagenode/hashstore.OpenTable:115\n\tstorj.io/storj/storagenode/hashstore.NewStore:258\n\tstorj.io/storj/storagenode/hashstore.New:93\n\tstorj.io/storj/storagenode/piecestore.(*HashStoreBackend).getDB:248\n\tstorj.io/storj/storagenode/piecestore.NewHashStoreBackend:114\n\tstorj.io/storj/storagenode.New:598\n\tmain.cmdRun:84\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.cleanup.func1.4:392\n\tstorj.io/common/process.cleanup.func1:410\n\tgithub.com/spf13/cobra.(*Command).execute:985\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1117\n\tgithub.com/spf13/cobra.(*Command).Execute:1041\n\tstorj.io/common/process.ExecWithCustomOptions:112\n\tmain.main:34\n\truntime.main:272\n\tmain.cmdRun:86\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.cleanup.func1.4:392\n\tstorj.io/common/process.cleanup.func1:410\n\tgithub.com/spf13/cobra.(*Command).execute:985\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1117\n\tgithub.com/spf13/cobra.(*Command).Execute:1041\n\tstorj.io/common/process.ExecWithCustomOptions:112\n\tmain.main:34\n\truntime.main:272”}

Alexey · April 27, 2025, 2:28am

Looks like the hashtable has been corrupted, and there is no tool has been implemented yet to regenerate it from logs as far as I know.
Perhaps switch to memtable may help, but I’m not sure.
Asked the team for help.

arrogantrabbit · April 27, 2025, 2:37am

(That’s the kind of stuff I have in mind m in my rants against hashstore vs a regular, let alone CoW, filesystem).

Alexey · April 27, 2025, 3:02am

Yes, and I didn’t migrate my nodes too, they works fine with the current piecestore, despite that they works under Windows and even worse - two of them under the Rancher Desktop. I even didn’t move DB to SSD, it’s not needed for my simple setup. Since it’s designed for Hyper-V, there is enough RAM (32GB) and CPU cores (Intel(R) Core™ i7-4770 CPU @ 3.40GHz).

zeebo · April 28, 2025, 7:16pm

Couple of questions:

what os/filesystem are you running? and what version of the storagenode software is running?
if linux, can you give the output of find <path/to/hashstore> -type f -name 'hashtbl-*' | xargs -I {} sh -c 'xxd {} | head -n 2'? mine ends up looking like

00000000: 4854 424c 0000 4eec 0101 0000 0000 0000  HTBL..N.........
00000010: 0013 0000 0000 0000 0000 0000 0000 0000  ................
00000000: 4854 424c 0000 4eee 0101 0000 0000 0000  HTBL..N.........
00000010: 0012 0000 0000 0000 0000 0000 0000 0000  ................
00000000: 4854 424c 0000 4ede 0101 0000 0000 0000  HTBL..N.........
00000010: 000e 0000 0000 0000 0000 0000 0000 0000  ................
00000000: 4854 424c 0000 4eee 0101 0000 0000 0000  HTBL..N.........
00000010: 0011 0000 0000 0000 0000 0000 0000 0000  ................
00000000: 4854 424c 0000 4eee 0101 0000 0000 0000  HTBL..N.........
00000010: 0017 0000 0000 0000 0000 0000 0000 0000  ................
00000000: 4854 424c 0000 4eed 0101 0000 0000 0000  HTBL..N.........
00000010: 0017 0000 0000 0000 0000 0000 0000 0000  ................
00000000: 4854 424c 0000 4eee 0101 0000 0000 0000  HTBL..N.........
00000010: 0012 0000 0000 0000 0000 0000 0000 0000  ................
00000000: 4854 424c 0000 4eee 0101 0000 0000 0000  HTBL..N.........
00000010: 0012 0000 0000 0000 0000 0000 0000 0000  ................

if you aren’t on linux, can you find all the files named hashtbl-<some hex digits> in your data folder and zip them up and get them to me somehow?

in the meantime, i’ll try to work on a quick program that can reconstruct hashtbls from the log files.

thanks!

syncamide · November 8, 2025, 10:50am

Hello zeebo
I have the same issue
how can I fix that?

root@host42:/mnt/poolx/subvol-536-disk-0/storage/hashstore# find /mnt/poolx/subvol-536-disk-0/storage/hashstore/ -type f -name 'hashtbl-*' | xargs -I {} sh -c 'xxd {} | head -n 2'
00000000: 4854 424c 0000 4fae 0100 0000 0000 0000  HTBL..O.........
00000010: 000f 0000 0000 0000 0000 0000 0000 0000  ................
00000000: 4854 424c 0000 4fad 0100 0000 0000 0000  HTBL..O.........
00000010: 000e 0000 0000 0000 0000 0000 0000 0000  ................
00000000: 4854 424c 0000 4fae 0100 0000 0000 0000  HTBL..O.........
00000010: 000e 0000 0000 0000 0000 0000 0000 0000  ................
00000000: 4854 424c 0000 4fae 0100 0000 0000 0000  HTBL..O.........
00000010: 000e 0000 0000 0000 0000 0000 0000 0000  ................
00000000: 4854 424c 0000 4fae 0100 0000 0000 0000  HTBL..O.........
00000010: 0011 0000 0000 0000 0000 0000 0000 0000  ................
00000000: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00000010: 0000 0000 0000 0000 0000 0000 0000 0000  ................

zeebo · November 8, 2025, 5:14pm

it looks like the last hashtbl file is invalid (it consists of all 0s). side question: what filesystem are you using?

there does exist a write-hashtbl command that can be used to reconstruct the table from the log files. it’s at storj/cmd/write-hashtbl at main · storj/storj · GitHub. here’s some steps to follow to use it:

have the Go toolchain installed (https://go.dev)
run go install storj.io/storj/cmd/write-hashtbl@latest to install it (it will probably end up in $HOME/go/bin)
run it like write-hashtbl /path/to/hashstore/satellite_id/sX. it will make a file named hashtbl in the current working directory
move it into the appropriate meta directory. (mv hashtbl /path/to/hashstore/satellite_id/sX/meta/hashtbl)

to identify which directory, it should be the last file printed by find /mnt/poolx/subvol-536-disk-0/storage/hashstore/ -type f -name 'hashtbl-*'. you can double check that it’s the right one by running xxd /path/to/hashstore/satellite_id/sX/meta/hashtbl-whatever | head -n 2 and it should be all zeros.

be careful when doing any destructive actions (like moving a file over top of a different one). it would be wise to move files out of the way (like, mv foo foo.bak to rename it out of the way) instead.

a more user friendly change is coming in the form of https://review.dev.storj.tools/c/storj/storj/+/19363 but it’s not yet reviewed/merged/released.

good luck and let me know if you have any questions.

syncamide · November 8, 2025, 9:08pm

Hi, zeebo.
Thank you very much for helping me
I have a zfs filesystem
not sure what I do wrong but it looks like new hashtable is empty

root@host42:~# xxd /mnt/poolx/subvol-536-disk-0/storage/hashstore/12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs/s0/meta/hashtbl-0000000000000034 | head -n 2
00000000: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00000010: 0000 0000 0000 0000 0000 0000 0000 0000  ................
root@host42:~# write-hashtbl /mnt/poolx/subvol-536-disk-0/storage/hashstore/12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs/s0
Counting /mnt/poolx/subvol-536-disk-0/storage/hashstore/12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs/s0/07/log-0000000000000007-00004f88...
platform: invalid argument
        storj.io/storj/storagenode/hashstore/platform.mmap:30
        storj.io/storj/storagenode/hashstore/platform.Mmap:16
        main.openFile:37
        main.(*cmdRoot).iterateRecords:148
        main.(*cmdRoot).countRecords:212
        main.(*cmdRoot).Execute:79
        github.com/zeebo/clingy.(*Environment).dispatchDesc:129
        github.com/zeebo/clingy.Environment.Run:41
        main.main:29
        runtime.main:285
root@host42:~# ls -la hashtbl
-rw------- 1 root root 0 Nov  8 23:54 hashtbl

I try to give it another sX and it find some records but hastbl is zero anyway:

root@host42:~# write-hashtbl /mnt/poolx/subvol-536-disk-0/storage/hashstore/1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE/s0
Counting /mnt/poolx/subvol-536-disk-0/storage/hashstore/1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE/s0/03/log-0000000000000003-00000000...
Counting /mnt/poolx/subvol-536-disk-0/storage/hashstore/1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE/s0/0a/log-000000000000000a-00000000...
Counting /mnt/poolx/subvol-536-disk-0/storage/hashstore/1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE/s0/0f/log-000000000000000f-00000000...
Counting /mnt/poolx/subvol-536-disk-0/storage/hashstore/1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE/s0/11/log-0000000000000011-00000000...
Counting /mnt/poolx/subvol-536-disk-0/storage/hashstore/1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE/s0/13/log-0000000000000013-00000000...
Counting /mnt/poolx/subvol-536-disk-0/storage/hashstore/1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE/s0/14/log-0000000000000014-00004fb6...
Counting /mnt/poolx/subvol-536-disk-0/storage/hashstore/1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE/s0/16/log-0000000000000016-00000000...
Counting /mnt/poolx/subvol-536-disk-0/storage/hashstore/1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE/s0/17/log-0000000000000017-00004fbb...
Counting /mnt/poolx/subvol-536-disk-0/storage/hashstore/1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE/s0/18/log-0000000000000018-00004fbd...
Record count=2610
Using logSlots=13
hashstore: logSlots too small: logSlots=13
        storj.io/storj/storagenode/hashstore.CreateHashTbl:73
        storj.io/storj/storagenode/hashstore.CreateTable:129
        main.(*cmdRoot).Execute:101
        github.com/zeebo/clingy.(*Environment).dispatchDesc:129
        github.com/zeebo/clingy.Environment.Run:41
        main.main:29
        runtime.main:285
root@host42:~# ls -la hashtbl
-rw------- 1 root root 0 Nov  9 00:03 hashtbl

alpharabbit · November 8, 2025, 9:45pm

Check logfile folder for zero byte files and delete them. I remember write-hashtbl crashing on such files.

Alexey · November 9, 2025, 5:01am

zeebo · November 9, 2025, 3:30pm

this error hashstore: logSlots too small: logSlots=13 is a bug. the counting estimation code doesn’t take into account that a hashtbl needs to be at least 14. oops.

you can override the counting/estimation by supplying the -s flag, so it would be write-hashtbl <dir> -s 14.

syncamide · November 9, 2025, 3:34pm

guys thank you all
deleting zero-sized files and -s parameter helps me to recover node