Looks like the hashstore node has a problem. It refuses to start:
Failed to create storage node peer: hashstore: invalid header: "\xd1\xf1\xb0o"
According to
Failed to create storage node peer: hashstore: invalid header: - #7 by zeebo
there should be a tool available to reconstruct the table from the logs:
So what am I supposed to do now to get the node back up?
Version is 1.152.6 and I do not see such a tool or config option.
Is it still required to build and run it manually?
What’s up with the full table reconstruction that should be in v1.146 and later versions?
It’s disabled by default and hidden to do not be added to the autogenerated config.
Based on the code, the configuration option name in config.yaml for ReconstructTable is:
storage.hashstore.store.reconstruct-table
The struct field is ReconstructTable (line 74 of config.go), which gets converted to kebab-case (reconstruct-table) when used in YAML configuration files following the standard Storj configuration naming convention.
Key details:
ReconstructTablestorage.hashstore.store.reconstruct-tablefalsetrue (won’t appear in auto-generated config docs)So in your config.yaml, you would add:
storage.hashstore.store.reconstruct-table: true
or
storage:
hashstore:
store:
reconstruct-table: true
But it will not help you to rebuild the hashtbl, if it will exist. So you need to at least delete the existing table.
So it should work as docker command --storage.hashstore.store.reconstruct-table true?
And I do not pass anything to it? Like names of corrupted files, satellite or something?
And it will definitely overwrite the broken hashtable or will it backup it first? Shall I manually backup the broken hashtbl file first maybe?
Yes, this option should overwrite the broken hashtable accordingly Copilot:
I’ll help you find information about the StorageNode hashtable reconstruction behavior. Let me search the Storj repository for details about this configuration option.Based on the code, yes, when storage.hashstore.store.reconstruct-table is enabled, the storagenode will automatically reconstruct the hash table from log files if the hashtable is missing or broken during startup.
Here’s how it works:
When the option is enabled (ReconstructTable = true):
On Startup: If the hash table file is missing but log files exist, instead of failing with an error, the store will attempt to reconstruct the hash table from the log files.
Automatic Recovery: The store reads through all log files and rebuilds the hash table by inserting records from the logs. This happens automatically without requiring manual intervention.
Data Consistency: If any records in the table are found to be invalid during reconciliation, the table is rewritten to remove them.
From the config definition:
ReconstructTable bool `help:"if set, reconstructs the table from the log files on startup if table is missing and logs are present" default:"false" hidden:"true"`
The reconstruction happens when:
ReconstructTable option is enabledWithout this option enabled, if the hashtable is missing but log files exist, the store will fail to start with an error about potential misconfiguration.
So enabling this option provides automatic recovery from hashtable corruption/loss by replaying the data from transaction logs—a safety mechanism to prevent data loss.
Error: unknown flag: --storage.hashstore.store.reconstruct-table
Seems the Copilot was wrong, I think that storage is excess config path, and the option should be --hashstore.store.reconstruct-table true
Amazing. It is doing something… 
I hope it doesn’t ruin your node by chewing it like a kebab 
Same.
Let’s see what happened:
I do not receive the same error message, but no real success message as well:
There are many messages like WARN hashstore mismatched log tail {"process": "storagenode", "satellite": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs
and then it quits:
2026-05-31 08:25:03,177 WARN received SIGTERM indicating exit request
2026-05-31 08:25:03,178 INFO waiting for storagenode, processes-exit-eventlistener, storagenode-updater to die
2026-05-31T08:25:03Z INFO Got a signal from the OS: "terminated" {"process": "storagenode-updater"}
2026-05-31 08:25:03,182 INFO stopped: storagenode-updater (exit status 0)
2026-05-31T08:25:03Z INFO Got a signal from the OS: "terminated" {"process": "storagenode"}
2026-05-31 08:25:06,217 INFO waiting for storagenode, processes-exit-eventlistener to die
2026-05-31 08:25:09,219 INFO waiting for storagenode, processes-exit-eventlistener to die
2026-05-31 08:25:12,222 INFO waiting for storagenode, processes-exit-eventlistener to die
2026-05-31 08:25:13,222 WARN killing 'storagenode' (72) with SIGKILL
2026-05-31 08:25:13,322 INFO stopped: storagenode (terminated by SIGKILL)
2026-05-31 08:25:13,322 INFO stopped: processes-exit-eventlistener (terminated by SIGTERM)
in the folder I can still see the backup file, but no new hashtbl file with the same name. But there is another one with a new name and the file data suggests it could be new.
I don’t know if it is supposed to re-create the hashtbl file with the same name?
Question is, shall I start the node or shall I run the reconstruction again?
My bad. It seems that after that message it restarts and processes another few and gets killed again. This seems weird but I see the folder number progressing.
However I wonder if this is how it is supposed to operate? It does not seem to be very effective if it gets killed and restarted over and over again.
@Alexey
Finally the node is running again.
It seems those periodic restarts are intendent. Still seems weird.
Let’s see if the node survives…
Perhaps you shouldn’t keep this flag in your docker run command.
Killing could be from the supervisor, because the node didn’t respond within its default timeout.
Thanks. Yes, I have removed it after its final completion. So far the node is still running…