Failing audits for pieces that don't seem to have been put on the node

Node Operators troubleshooting
1

I’ve launched a new node about 71 hours ago, which has already been disqualified on the US satellite, for failing audits. These errors:


2025-10-08T18:31:01Z    ERROR   piecestore      download failed {"Process": "storagenode", "Piece ID": "CJPOJHIP5G6QPQX562R4Y26AKDPDB4ZH5XGAZ2PA4EKAKUDUL6IQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET_REPAIR", "Offset": 0, "Size": 768, "Remote Address": "46.62.212.1:18004", "error": "hashstore: file does not exist", "errorVerbose": "hashstore: file does not exist\n\tstorj.io/storj/storagenode/hashstore.(*DB).Read:376\n\tstorj.io/storj/storagenode/piecestore.(*HashStoreBackend).Reader:303\n\tstorj.io/storj/storagenode/piecestore.(*MigratingBackend).Reader:197\n\tstorj.io/storj/storagenode/piecestore.(*TestingBackend).Reader:105\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:690\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:302\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:62\n\tstorj.io/common/experiment.(*Handler).HandleRPC:43\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:166\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:108\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:156\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35"}

consantly happen. I’ve searched the logs (rotated every 72 hours, so all of them should still be available) and I haven’t found the request to actually put that piece (and the other ones which have had the download error).

What should I do?

2

If I remember right Storj suspended PUT log, to reduce logs amount. there is only 1 line, successful or failed.

3

They are still there in debug log level, if I understand correctly what you mean
— things like upload started(debug)/uploaded(info).

4

I still see PUT requests, and for those pieces it doesn’t seem to be any problems with audits

5

Do you have a hashstore backend enabled?
I do not expect that it’s a default currently, but maybe you have enabled it?

Actually the more realistic if you re-used the old identity with empty storage.

You need to generate a new identity, not copy a previous one.

6

Looks like I accidentaly put the same identity on two nodes. Thank you for helping!

1 Like
7

Then you will get DQ soon.

1 Like