Failed Audit from us1

Node Operators troubleshooting
,
1

Hi, One of my nodes, over the last 4 weeks, it has failed 8x Audit requests. Node is running on default backend - piecestore. Am I missing data somehow?

Each of the failed Audit looks like:

2025-05-12T18:06:07Z ERROR piecestore download failed {“Process”: “storagenode”, “Piece ID”: “BOOPW75ZPRAFWXIMI5ABY4TSBTMHTHZLRTTHJLIRPT7VSUCOMNJA”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “GET_AUDIT”, “Offset”: 1762048, “Size”: 256, “Remote Address”: “35.188.235.2:15964”, “error”: “hashstore: file does not exist”, “errorVerbose”: “hashstore: file does not exist\n\tstorj.io/storj/storagenode/hashstore.(*DB).Read:359\n\tstorj.io/storj/storagenode/piecestore.(*HashStoreBackend).Reader:298\n\tstorj.io/storj/storagenode/piecestore.(*MigratingBackend).Reader:180\n\tstorj.io/storj/storagenode/piecestore.(*TestingBackend).Reader:105\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:676\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:302\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:62\n\tstorj.io/common/experiment.(*Handler).HandleRPC:43\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:166\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:108\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:156\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35”}

Regards

2

Are you seeing the audit % drop in the UI… or is this only something you see in your logs? You could chkdsk/fsck/scrub to see if there are filesystem errors… but if I didn’t see my audit number dropping I’d probably just ignore it.

Maybe audits are coming in: and both the piecestore and hashstore backends are being triggered… and since you’re not using hashstore it’s just failing but not affecting anything?

3

It drops the audit % by 0.01% (huge!!!). Back to 100% in a couple of days. So, no dramatic effect.

Routine scrub ran last night, and detected no errors.’

4

I have observed this when node would delay/drop repair requests.

5

Specifically GET_REPAIR. In short, failing GET_AUDIT and GET_REPAIR will affect audit scores.

6

Could you please search your logs for this piece to see a history with it?

7

My logs go back 6 weeks, no other reference to that Piece ID.

8

Interesting. Perhaps it was a result of moving this piece to the trash and the sent recover command didn’t restore it (because of too late).
I do not remember, is PieceID logged, when it’s moved to the trash on info level?

Or there was a real piece lost, hashstore checked after piecestore.

I have a one more idea - could you please search one of the piece on your disk in the piecestore?
See how to convert a PieceID to the file:

9

Been through all five failed audit errors, no other reference to the respective peice id in the logs or file system.

10

Then these pieces are actually lost. However, a little bit concerning the fact that you have no evidence, that these pieces were ever uploaded to your node.

11

Concerning that I have missing data. Node is 15 months old, and I only keep six weeks of logs, not surprising I don’t have any other record of missing files.
. What surprised me, was that it’s looking in Hashstore for data, but that’s not enabled.

12

It’s checked as a second source of truth, so no wondering.

13

Today I got a similar GET_AUDIT error from us1. Searching the log by “Piece ID” only finds this error message.

Dec 20 07:00:54 storj storagenode[19332]: 2025-12-20T07:00:54Z        ERROR        piecestore        download failed        {"Process": "storagenode", "Piece ID": "6X544FSRVZMXXD6X45PQD3VKXVR2QSD2P2YBAXN7E3VVGF4JI2XA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET_AUDIT", "Offset": 1908480, "Size": 256, "Remote Address": "35.212.66.219:39391", "error": "hashstore: file does not exist", "errorVerbose": "hashstore: file does not exist\n\tstorj.io/storj/storagenode/hashstore.(*DB).Read:382\n\tstorj.io/storj/storagenode/piecestore.(*HashStoreBackend).Reader:304\n\tstorj.io/storj/storagenode/piecestore.(*MigratingBackend).Reader:197\n\tstorj.io/storj/storagenode/piecestore.(*TestingBackend).Reader:105\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:690\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:302\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:62\n\tstorj.io/common/experiment.(*Handler).HandleRPC:43\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:166\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:108\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:156\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35"}
14

Hello @NodeOperator,
Welcome to the forum!

Then your node lost this piece. If you have older logs, then you may find when it was uploaded.

15

Hello. Yes, you’re right. I checked, and my old logs were deleted. I forgot to restart journald after increasing the log space :person_facepalming:
Unfortunately, now we can only guess what happened to this file. Similar issues haven’t occurred since, the node is new, only a month old. SMART is ok, filesystem ext4 (Linux host)

16

It maybe lost if you have had a power loss.

17

Everything is fine with this it’s a server in a data center. The only thing I did was increase the available disk space in the configuration file and restart the storagenode process via systemctl

18

The satellite may try to request it again, this would happen until it will not be deleted by the customer or if this segment would fail below the repair threshold and the pointer for this piece on your node could be removed.

19

Thanks for the information. There have been no more requests for this segment at this time, and the audit is again approaching 100%. (From 99.91 to 99.95)

1 Like