ok, i’ve removed old hashtable file. now version from ‘main’ gives same result as ‘latest’:
Processing /storj/STORAGE/config/storage/hashstore/12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs/s1/ff/log-00000000000004ff-00000000...
hashstore: put:{key:368697015cb3eb710f70ad81855d6b6f5f948daad263bdba0a23ee333538b22e offset:972229120 log:1544 length:26368 created:20097 (2025-01-09) expires:0 (1970-01-01) trash:false} != exist:{key:368697015cb3eb710f70ad81855d6b6f5f948daad263bdba0a23ee333538b22e offset:629982208 log:1281 length:26368 created:20097 (2025-01-09) expires:0 (1970-01-01) trash:false}: hashstore: collision detected
storj.io/storj/storagenode/hashstore.(*HashTbl).insertLocked:411
storj.io/storj/storagenode/hashstore.(*HashTblConstructor).Append:563
main.(*cmdRoot).Execute.func2:111
main.(*cmdRoot).iterateRecords:211
main.(*cmdRoot).Execute:110
github.com/zeebo/clingy.(*Environment).dispatchDesc:129
github.com/zeebo/clingy.Environment.Run:41
main.main:30
runtime.main:290
write-hashtbl appears to have a bug in how it handles hashstore.ErrCollision.
The intent in cmd/write-hashtbl/main.go is clearly to ignore collisions during reconstruction:
ok, err := tcons.Append(ctx, rec)
if errors.Is(err, hashstore.ErrCollision) {
// ignore collisions ...
}
However, HashTblConstructor.Append() makes errors sticky internally:
So the actual behavior is:
- the first collision is observed in
Append();
main.go tries to ignore it;- but
Append() has already stored that collision in c.err;
- all subsequent appends are effectively poisoned;
Done() fails at the end with the same collision.
This means collisions are not really ignored in practice, despite the comment in write-hashtbl.
One more detail: the final printed stack trace points to the first collision site, not the final Done() call, because errs preserves the stack from when the error was originally created. That makes the failure look like an immediate append failure even when the process only aborts at the end.
In short: cmd/write-hashtbl currently cannot successfully complete when duplicate records produce ErrCollision, even though the code comments explicitly say those collisions should be ignored.
Solutions
Option A: narrow fix, only for write-hashtbl
Change only cmd/write-hashtbl:
- keep
hashstore library untouched;
- create the table with
CreateTable(...);
- immediately call
Done() to obtain a Tbl;
- insert reconstructed records with
Tbl.Insert(...) instead of TblConstructor.Append(...).
Why this works:
TblConstructor.Append() has sticky error state;Tbl.Insert() does not;- so
write-hashtbl can continue to ignore ErrCollision locally, without changing global hashstore behavior.
Pros:
- smallest blast radius;
- no behavior change for other
hashstore users;
- safest choice if the bug is considered specific to this utility.
Cons:
- changes the control flow of the utility more noticeably;
- works around the constructor behavior instead of fixing it at the source.
Option B: broader fix in hashstore constructors
Change HashTblConstructor.Append() and MemTblConstructor.Append() so that ErrCollision is returned but not stored as sticky c.err.
Why this works:
- callers like
write-hashtbl can catch ErrCollision and continue;
Done() no longer fails later because of a previously ignored collision.
Pros:
- smaller conceptual fix;
- matches the expectation of callers that want to handle collisions explicitly;
- preserves the existing
CreateTable -> Append -> Done flow.
Cons:
- affects all users of
TblConstructor.Append(), not just write-hashtbl;
- changes general
hashstore semantics;
- could weaken collision handling in other code paths that currently rely on sticky failure.
Recommendation
If the goal is “fix only write-hashtbl with minimal risk”, I would recommend Option A.
If the maintainers think “a caller that explicitly handles ErrCollision should always be able to continue”, then Option B is cleaner, but it is a library-wide behavior change and should be reviewed as such.
