Unrecoverable error

[Vadim]

Hello

does someone know what this mean and how to fix it?
2026-01-29T18:04:38+02:00 FATAL Unrecoverable error {“error”: “Failed to create storage node peer: hashstore: put:{key:abf462e56920cae88bf35f0583292161f73c4219bda3843b5fe0bdb54757857a offset:807168 log:385 length:23552 created:20467 (2026-01-14) expires:20484 (2026-01-31) trash:false} != exist:{key:abf462e56920cae88bf35f0583292161f73c4219bda3843b5fe0bdb54757857a offset:32775104 log:398 length:23552 created:20467 (2026-01-14) expires:20481 (2026-01-28) trash:true}: hashstore: collision detected\n\tstorj.io/storj/storagenode/hashstore.(*MemTbl).insertLocked:531\n\tstorj.io/storj/storagenode/hashstore.(*MemTbl).Insert:468\n\tstorj.io/storj/storagenode/hashstore.(*Store).reconcileLog:1495\n\tstorj.io/storj/storagenode/hashstore.NewStore.func6:299\n\tstorj.io/storj/storagenode/hashstore.NewStore-range4:318\n\tstorj.io/storj/storagenode/hashstore.NewStore.NewStore.parseFiles[…].func15.func16:41\n\tpath/filepath.walkDir:310\n\tpath/filepath.walkDir:332\n\tpath/filepath.walkDir:332\n\tpath/filepath.WalkDir:400\n\tstorj.io/storj/storagenode/hashstore.NewStore.parseFiles[…].func15:27\n\tstorj.io/storj/storagenode/hashstore.NewStore:257\n\tstorj.io/storj/storagenode/hashstore.New:110\n\tstorj.io/storj/storagenode/piecestore.(*HashStoreBackend).getDB:271\n\tstorj.io/storj/storagenode/piecestore.NewHashStoreBackend:122\n\tstorj.io/storj/storagenode.New:607\n\tmain.cmdRun:84\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.InitBeforeExecute.func1.2:389\n\tstorj.io/common/process.InitBeforeExecute.func1:407\n\tgithub.com/spf13/cobra.(*Command).execute:985\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1117\n\tgithub.com/spf13/cobra.(*Command).Execute:1041\n\tstorj.io/common/process.ExecWithCustomOptions:115\n\tstorj.io/common/process.ExecWithCustomConfigAndLogger:80\n\tstorj.io/common/process.ExecWithCustomConfig:75\n\tstorj.io/common/process.Exec:65\n\tmain.(*service).Execute.func1:107\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:93”, “errorVerbose”: “Failed to create storage node peer: hashstore: put:{key:abf462e56920cae88bf35f0583292161f73c4219bda3843b5fe0bdb54757857a offset:807168 log:385 length:23552 created:20467 (2026-01-14) expires:20484 (2026-01-31) trash:false} != exist:{key:abf462e56920cae88bf35f0583292161f73c4219bda3843b5fe0bdb54757857a offset:32775104 log:398 length:23552 created:20467 (2026-01-14) expires:20481 (2026-01-28) trash:true}: hashstore: collision detected\n\tstorj.io/storj/storagenode/hashstore.(*MemTbl).insertLocked:531\n\tstorj.io/storj/storagenode/hashstore.(*MemTbl).Insert:468\n\tstorj.io/storj/storagenode/hashstore.(*Store).reconcileLog:1495\n\tstorj.io/storj/storagenode/hashstore.NewStore.func6:299\n\tstorj.io/storj/storagenode/hashstore.NewStore-range4:318\n\tstorj.io/storj/storagenode/hashstore.NewStore.NewStore.parseFiles[…].func15.func16:41\n\tpath/filepath.walkDir:310\n\tpath/filepath.walkDir:332\n\tpath/filepath.walkDir:332\n\tpath/filepath.WalkDir:400\n\tstorj.io/storj/storagenode/hashstore.NewStore.parseFiles[…].func15:27\n\tstorj.io/storj/storagenode/hashstore.NewStore:257\n\tstorj.io/storj/storagenode/hashstore.New:110\n\tstorj.io/storj/storagenode/piecestore.(*HashStoreBackend).getDB:271\n\tstorj.io/storj/storagenode/piecestore.NewHashStoreBackend:122\n\tstorj.io/storj/storagenode.New:607\n\tmain.cmdRun:84\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.InitBeforeExecute.func1.2:389\n\tstorj.io/common/process.InitBeforeExecute.func1:407\n\tgithub.com/spf13/cobra.(*Command).execute:985\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1117\n\tgithub.com/spf13/cobra.(*Command).Execute:1041\n\tstorj.io/common/process.ExecWithCustomOptions:115\n\tstorj.io/common/process.ExecWithCustomConfigAndLogger:80\n\tstorj.io/common/process.ExecWithCustomConfig:75\n\tstorj.io/common/process.Exec:65\n\tmain.(*service).Execute.func1:107\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:93\n\tmain.cmdRun:86\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.InitBeforeExecute.func1.2:389\n\tstorj.io/common/process.InitBeforeExecute.func1:407\n\tgithub.com/spf13/cobra.(*Command).execute:985\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1117\n\tgithub.com/spf13/cobra.(*Command).Execute:1041\n\tstorj.io/common/process.ExecWithCustomOptions:115\n\tstorj.io/common/process.ExecWithCustomConfigAndLogger:80\n\tstorj.io/common/process.ExecWithCustomConfig:75\n\tstorj.io/common/process.Exec:65\n\tmain.(*service).Execute.func1:107\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:93”}

[Alexey]

Seems the same:

Write-hashtbl hashstore: collision detected

This is not a dangerous error. Old piecestore silently ignored double uploads, hashstore doesn’t. Should be very rare (for example: piece is deleted, and repair tries to re-upload with the same ID). never delete log files because this. That will delete healthy pieces and will cause disqualification.

However there is no workaround at the moment. Here are proposed changes:
https://review.dev.storj.tools/c/storj/storj/+/20196 storagenode/hashstore: skip log file check bool
https://review.dev.storj.tools/c/storj/storj/+/20197 storagenode/hashstore: log hint info
https://review.dev.storj.tools/c/storj/storj/+/20198 storagenode/hashstore: ignore collisions during fsck

[Vadim]

one of the nodes has hashtable problem, i started to rebild but write-hashtbl gives me this
do i need to do something with it or it fixed it somehow?
looks like it going ahead

record at offset=247272768 in F:\hashstore\121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6\s0\4d\log-000000000000014d-00000000 has invalid piece hash=69d03b1809e87b2b41cf3a4e8af1771305bca1534249119f5a1a7cb2e90a1b9a
record at offset=247200512 in F:\hashstore\121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6\s0\4d\log-000000000000014d-00000000 has invalid piece hash=28f065dc17f3a2fd3b4c8c8facbe40bec153811dd1e4e77070de7d78a5e5e8b1
record at offset=244880576 in F:\hashstore\121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6\s0\4d\log-000000000000014d-00000000 has invalid piece hash=07fa8a94dd06b17cdd8a23295f9687cd861be80591e8ab912dafabf21117f264
record at offset=244874368 in F:\hashstore\121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6\s0\4d\log-000000000000014d-00000000 has invalid piece hash=ad7facb2586fc6e966c004d7d1d16b024f5805ff7cb47c7a85dabd8b48892ca7
record at offset=244869696 in F:\hashstore\121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6\s0\4d\log-000000000000014d-00000000 has invalid piece hash=41cb22109da26a6ff5464d6915db81c1c60f9e0808d8dbd63df1550b86372165
record at offset=244821504 in F:\hashstore\121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6\s0\4d\log-000000000000014d-00000000 has invalid piece hash=28f065dc17f3a2fd3b4c8c8facbe40bec153811dd1e4e77070de7d78a5e5e8b1
record at offset=242501568 in F:\hashstore\121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6\s0\4d\log-000000000000014d-00000000 has invalid piece hash=2cc2784c0ddcd0769b915ab468a65b378a10302cf058230ac75c9735750559d8
record at offset=242495104 in F:\hashstore\121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6\s0\4d\log-000000000000014d-00000000 has invalid piece hash=76f6b79828f76acb810744448ea8c0acd41f5743775c7ae6139ebbe7f0a0b53b
record at offset=242168128 in F:\hashstore\121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6\s0\4d\log-000000000000014d-00000000 has invalid piece hash=8ce8ba8e726ee8925e6560d86ac35be1097691d1cfac888e6bd20e804ea9eb15
record at offset=242164992 in F:\hashstore\121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6\s0\4d\log-000000000000014d-00000000 has invalid piece hash=56a43ef88ddfcd0f56f7dd973312c0e73d62f59655c01d7b4e59aaa3be8b3fb6
record at offset=242158016 in F:\hashstore\121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6\s0\4d\log-000000000000014d-00000000 has invalid piece hash=2cc2784c0ddcd0769b915ab468a65b378a10302cf058230ac75c9735750559d8
record at offset=242151552 in F:\hashstore\121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6\s0\4d\log-000000000000014d-00000000 has invalid piece hash=1d0ea33861b16f7f6f044ec540ecde9876ebf2d6da2c87850279db9e1495f9e3
record at offset=241969472 in F:\hashstore\121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6\s0\4d\log-000000000000014d-00000000 has invalid piece hash=6bb4877dfebc6d4f819999f0f8b65d06aa540746bc63b7c1261282636efcaa15
record at offset=241965568 in F:\hashstore\121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6\s0\4d\log-000000000000014d-00000000 has invalid piece hash=28f065dc17f3a2fd3b4c8c8facbe40bec153811dd1e4e77070de7d78a5e5e8b1
record at offset=239645632 in F:\hashstore\121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6\s0\4d\log-000000000000014d-00000000 has invalid piece hash=bfe492baf731a0dbf6e1e050f5bc3fe8c1b049383194dcdf82f023bfa409f462
record at offset=239643776 in F:\hashstore\121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6\s0\4d\log-000000000000014d-00000000 has invalid piece hash=1d0ea33861b16f7f6f044ec540ecde9876ebf2d6da2c87850279db9e1495f9e3
record at offset=239461696 in F:\hashstore\121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6\s0\4d\log-000000000000014d-00000000 has invalid piece hash=80422bc3d307b4a25bdafcc84ac7fb01cb55a09810e8b0f37bb12e0edb5c48ca
record at offset=239459584 in F:\hashstore\121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6\s0\4d\log-000000000000014d-00000000 has invalid piece hash=bfe492baf731a0dbf6e1e050f5bc3fe8c1b049383194dcdf82f023bfa409f462
record at offset=239457728 in F:\hashstore\121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6\s0\4d\log-000000000000014d-00000000 has invalid piece hash=6fc74890c52aef57c0b98ace4a7dec52972c8498fbe56e84e68d40b7c8087e4e
record at offset=239452800 in F:\hashstore\121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6\s0\4d\log-000000000000014d-00000000 has invalid piece hash=1d0ea33861b16f7f6f044ec540ecde9876ebf2d6da2c87850279db9e1495f9e3
record at offset=239270720 in F:\hashstore\121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6\s0\4d\log-000000000000014d-00000000 has invalid piece hash=9e635f518975d1cfaec0334264395043b7539faab201693bb795de7d24ce929b
record at offset=239249152 in F:\hashstore\121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6\s0\4d\log-000000000000014d-00000000 has invalid piece hash=3b7b2d6a621d51bcbcab08970d2873f967e179b5a017b61401883cb3af458349
record at offset=239226048 in F:\hashstore\121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6\s0\4d\log-000000000000014d-00000000 has invalid piece hash=07fa8a94dd06b17cdd8a23295f9687cd861be80591e8ab912dafabf21117f264
record at offset=239219840 in F:\hashstore\121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6\s0\4d\log-000000000000014d-00000000 has invalid piece hash=6fc74890c52aef57c0b98ace4a7dec52972c8498fbe56e84e68d40b7c8087e4e
record at offset=239214912 in F:\hashstore\121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6\s0\4d\log-000000000000014d-00000000 has invalid piece hash=e5a00aa9991ac8a5ee3109844d84a55583bd20572ad3ffcd42792f3c36b183ad
record at offset=239212288 in F:\hashstore\121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6\s0\4d\log-000000000000014d-00000000 has invalid piece hash=06da35fdfe464d39d3994ecdfa367195c2b3453ccb33acef1390c2d9d0d09c27
record at offset=239136960 in F:\hashstore\121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6\s0\4d\log-000000000000014d-00000000 has invalid piece hash=12a42cfe372b164bdba3fcaf9084fc39b2032c8aec11bdf0c2acb46a6b614fb7
Processing F:\hashstore\121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6\s0\4f\log-000000000000014f-00000000…

[Alexey]

Looks like a data corruption. If it’s continues, then let it finish. Yes, your node will fail audits on these pieces, however, there is a chance that it may survive.

[Vadim]

what mean if write-hashtbl gives this error?
panic: runtime error: slice bounds out of range [:18522] with capacity 512

goroutine 1 [running]:
main.(*cmdRoot).iterateRecords(0xc00007c340, {0x7ff66e45ca80, 0xc00001fe90}, {0xc00017af00, 0x64}, 0x0, 0xc000029b70)
/home/elek/jr/storj-write-hashtbl/main.go:188 +0xc45
main.(*cmdRoot).Execute(0xc00007c340, {0x7ff66e45ca80, 0xc00001fe90})
/home/elek/jr/storj-write-hashtbl/main.go:109 +0x95c
github.com/zeebo/clingy.(*Environment).dispatchDesc(0xc000149de0, {0x7ff66e45ca10, 0x7ff66e6967c0}, 0xc00007e120, {{0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, …})
/home/elek/go/pkg/mod/github.com/zeebo/clingy@v0.0.0-20250609200504-bc547d5f79c0/run.go:129 +0x6dd
github.com/zeebo/clingy.Environment.Run({{0x7ff66e45b700, 0xc00007c340}, {0x7ff66e3f4190, 0xd}, {0xc00006e410, 0x1, 0x1}, 0x0, 0x0, 0x7ff66e40ccd8, …}, …)
/home/elek/go/pkg/mod/github.com/zeebo/clingy@v0.0.0-20250609200504-bc547d5f79c0/run.go:41 +0x198
main.main()
/home/elek/jr/storj-write-hashtbl/main.go:29 +0x148

[Alexey]

Seems also one more corruption. I do not know, how to fix

[Alexey]

The engineer told me, that’s fixed on main.

go install storj.io/storj/cmd/write-hashtbl@main

[Vadim]

I work on windows, this install not work like this, can give direct link on github?

[Alexey]

No. However, it works like this for Windows too. But you need to install GoLang and Git, of course:

winget install GoLang.Go Git.Git

then in a new window:

go install storj.io/storj/cmd/write-hashtbl@main

and it will be available right from the command line.

[Vadim]

Yas it worked thank you.

[Vadim]

Does it also ignore collisions like last node version, because it also do not give to rebuild because of them.

[Alexey]

It should, at least it’s expected

[Vadim]

no colisions is error here this is not fixed.
@elek can you fix it in write-hashtbl also

hashstore: put:{key:86f0171bbe2d37903f82aeb8ebafad56374ab2fba779b1951c2d8e97359ba248 offset:1073687424 log:2339 length:62208 created:20325 (2025-08-25) expires:0 (1970-01-01) trash:false} != exist:{key:86f0171bbe2d37903f82aeb8ebafad56374ab2fba779b1951c2d8e97359ba248 offset:957022976 log:2842 length:62208 created:20325 (2025-08-25) expires:0 (1970-01-01) trash:false}: hashstore: collision detected
storj.io/storj/storagenode/hashstore.(*HashTbl).insertLocked:411
storj.io/storj/storagenode/hashstore.(*HashTblConstructor).Append:552
main.(*cmdRoot).Execute.func2:110
main.(*cmdRoot).iterateRecords:206
main.(*cmdRoot).Execute:109
github.com/zeebo/clingy.(*Environment).dispatchDesc:129
github.com/zeebo/clingy.Environment.Run:41
main.main:29
runtime.main:285

[Vadim]

@Alexey

go install storj.io/storj/cmd/write-hashtbl@main

hashstore: put:{key:86f0171bbe2d37903f82aeb8ebafad56374ab2fba779b1951c2d8e97359ba248 offset:1073687424 log:2339 length:62208 created:20325 (2025-08-25) expires:0 (1970-01-01) trash:false} != exist:{key:86f0171bbe2d37903f82aeb8ebafad56374ab2fba779b1951c2d8e97359ba248 offset:957022976 log:2842 length:62208 created:20325 (2025-08-25) expires:0 (1970-01-01) trash:false}: hashstore: collision detected
storj.io/storj/storagenode/hashstore.(*HashTbl).insertLocked:411
storj.io/storj/storagenode/hashstore.(*HashTblConstructor).Append:552
main.(*cmdRoot).Execute.func2:110
main.(*cmdRoot).iterateRecords:206
main.(*cmdRoot).Execute:109
github.com/zeebo/clingy.(*Environment).dispatchDesc:129
github.com/zeebo/clingy.Environment.Run:41
main.main:29
runtime.main:285

colissions not fixed

[Alexey]

Seems so, it has been reported for the team.

[Alexey]

The fix is merged to main, please try to build a new version.