Is it an Unrecoverable error?
Also looks like it’s not a collision error, something new. And it looks like a minor corruption.
Please check and fix the errors on the disk, then probably you need to use the write-hashtbl tool on it or maybe the next compaction could pass through it.
I don’t think so, there is an error “EOF”, which is usually mean that the file is truncated before the expected length. In this case likely the source.
I use 1.147.5 version
Found one node that get all the time same compaction error.
2026-02-01T06:01:00+02:00 ERROR hashstore compaction failed {“satellite”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “error”: “hashstore: writing into compacted log (rec={key:2118e13f15d6e6a8001f49c0c5b31b4ae0233e751c9162988432bdd914f306d5 offset:1072252032 log:108 length:2174976 created:20324 (2025-08-24) expires:0 (1970-01-01) trash:false}) (from=R:\hashstore\12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S\s0\6c\log-000000000000006c-00000000) (size=1074158400): EOF”, “errorVerbose”: “hashstore: writing into compacted log (rec={key:2118e13f15d6e6a8001f49c0c5b31b4ae0233e751c9162988432bdd914f306d5 offset:1072252032 log:108 length:2174976 created:20324 (2025-08-24) expires:0 (1970-01-01) trash:false}) (from=R:\hashstore\12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S\s0\6c\log-000000000000006c-00000000) (size=1074158400): EOF\n\tstorj.io/storj/storagenode/hashstore.(*Store).rewriteRecord:1383\n\tstorj.io/storj/storagenode/hashstore.(*Store).compactOnce.func8:1130\n\tstorj.io/storj/storagenode/hashstore.(*Store).compactOnce:1140\n\tstorj.io/storj/storagenode/hashstore.(*Store).Compact:843\n\tstorj.io/storj/storagenode/hashstore.(*DB).performPassiveCompaction:566”}
I found in logs about 30 rekords about this rec key, looks like it is problem with same file.
2118e13f15d6e6a8001f49c0c5b31b4ae0233e751c9162988432bdd914f306d5
why it try to compact to same file or rec?
Developers said that they will try to implement a recovery, but this error is only confirming that your disk have issues and corrupting data. Maybe the problem in cables or the disk controller if the disk is healthy.
Is that the type of damage the new amnesty system handles? If a data log is damaged the system will still try to rewrite-what-it-can: and report any holes to the satellite?
If node will get error on log, it cold skipp it, if not possible to get something.
But looks like instead of, it not compacting at all. Some nodes with that error has about 1 tb reclaimable data.
Sounds like this node has hardware issues to have 1TB of corrupted files.
You may try to regenerate hashtbl without using of the --fast flag, the developer said that this will allow it to do not add a corrupted piece to the hash table, so with the next compaction this garbage should be removed.
it just has some corrupted file in several s0 or s1 division, looks like as it hit this file it end compaction, and it hit same files every day. I see same error every day with same files.
Of course it will. Thus I suggest to try to regenerate hashtbl without using --fast flag, it should not add links to corrupted pieces to the table. And then the next compaction should get rid of skipped pieces as from the garbage. Of course, it could hit your audit score, if the lost piece will be audited. But it will anyway, since it’s already corrupted.
What I want to know, will it throw the error and continue or will stop on the first corrupted piece?
Interesting thing, that as I tried to delete this log file, instantly got error that someone cant download from it. I restored, error not show any more, after 5 min i tried again, and instantly got same error
2026-02-20T23:06:47+02:00 ERROR piecestore download failed {“piece_id”: “QOHNGQALMGVQTYAB7Y2D35BONFZGHZEY6LVURJDVGHS7FGMBW4KA”, “satellite_id”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “action”: “GET”, “offset”: 0, “size”: 3584, “remote_address”: “212.227.75.241:31817”, “error”: “hashstore: unable to open log file="L:\\hashstore\\12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S\\s0\\0e\\log-000000000000000e-00000000": The system cannot find the file specified.”, “errorVerbose”: “hashstore: unable to open log file="L:\\hashstore\\12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S\\s0\\0e\\log-000000000000000e-00000000": The system cannot find the file specified.\n\tstorj.io/storj/storagenode/hashstore.(*Store).readerForRecord:673\n\tstorj.io/storj/storagenode/hashstore.(*Store).Read:659\n\tstorj.io/storj/storagenode/hashstore.(*DB).Read:383\n\tstorj.io/storj/storagenode/piecestore.(*HashStoreBackend).Reader:333\n\tstorj.io/storj/storagenode/piecestore.(*MigratingBackend).Reader:197\n\tstorj.io/storj/storagenode/piecestore.(*TestingBackend).Reader:105\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:728\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:302\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:62\n\tstorj.io/common/experiment.(*Handler).HandleRPC:43\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:166\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:108\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:156\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35”}
is it possible that, someone all the time downloading from this file very often, thats why it geting error about compaction, so may be it has some shared read restrictions, but as i seen it shold be fixe in 1.147.5 that i am using.
No, you can skip corrupted pieces in the file but not the whole log file. So the difference will be between losing a 1MB of corrupted pieces and 1TB of log files.
You need to restart the node to flush the cache.
Unlikely, you would not get an error with a EOF type. So I would like to suggest to use write-hashtbl without a --fast flag. It should throw the error and continue, then compaction should not stop next time.