File does not exist and Audit Score decreasing

[Tommy]

Hello everyone,

I found my audit score was decreased few weeks ago (15-March-2023).
I checked my log and found it is due to “download failed” while “GET_AUDIT” , the file does not exist on my node.

I checked my harddisk and no error found.

And now (1-April-2023) the audit score decrease again. I checked the log it is due to “download failed” while “GET_AUDIT” , the file does not exist on my node … again.

And the missing file is same as the one on 15-Mar.

My node seems never try to recover the missing file but it just download fail again and again then decrease the audit score.

What should I do to fix the problem? Thank you.

Please find my log attached below. (by searching the Piece ID “HLZ3KL2FXCCGSX2KBSB73PTVV5ART7EVUQHIZZYLTGNYORCXKLKA” )

Line 1083386: 2023-03-15T19:09:33.415Z	INFO	piecestore	download started	{Process: storagenode, Piece ID: HLZ3KL2FXCCGSX2KBSB73PTVV5ART7EVUQHIZZYLTGNYORCXKLKA, Satellite ID: 12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S, Action: GET_AUDIT, Offset: 1702912, Size: 256, Remote Address: 172.17.0.1:45224}
Line 1083387: 2023-03-15T19:09:33.451Z	ERROR	piecestore	download failed	{Process: storagenode, Piece ID: HLZ3KL2FXCCGSX2KBSB73PTVV5ART7EVUQHIZZYLTGNYORCXKLKA, Satellite ID: 12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S, Action: GET_AUDIT, Offset: 1702912, Size: 0, Remote Address: 172.17.0.1:45224, error: file does not exist, errorVerbose: file does not exist\n\tstorj.io/common/rpc/rpcstatus.Wrap:75\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:650\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:251\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:61\n\tstorj.io/common/experiment.(*Handler).HandleRPC:42\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:124\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:114\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35}
Line 7935791: 2023-03-31T22:50:06.422Z	INFO	piecestore	download started	{Process: storagenode, Piece ID: HLZ3KL2FXCCGSX2KBSB73PTVV5ART7EVUQHIZZYLTGNYORCXKLKA, Satellite ID: 12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S, Action: GET_AUDIT, Offset: 2074112, Size: 256, Remote Address: 172.17.0.1:55388}
Line 7935794: 2023-03-31T22:50:06.767Z	ERROR	piecestore	download failed	{Process: storagenode, Piece ID: HLZ3KL2FXCCGSX2KBSB73PTVV5ART7EVUQHIZZYLTGNYORCXKLKA, Satellite ID: 12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S, Action: GET_AUDIT, Offset: 2074112, Size: 0, Remote Address: 172.17.0.1:55388, error: file does not exist, errorVerbose: file does not exist\n\tstorj.io/common/rpc/rpcstatus.Wrap:75\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:650\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:251\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:61\n\tstorj.io/common/experiment.(*Handler).HandleRPC:42\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:124\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:114\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35}

[jammerdan]

@Tommy

My node seems never try to recover the missing file

That’s a misconception on your side. Your node cannot recover a lost piece. That is not how the network is working.
Unless you have backed up that missing file somewhere it is lost and remains lost for your node.
In theory it could be a bug in the software that you get audited for a piece that has been deleted, but it is not very likely.

[Tommy]

The file (the piece ID) did not appears at any earlier log. So I believed that this piece never save to my harddisk before.

So the missing file will make my score decreasing again and again without fix.

[jammerdan]

@Tommy

So I believed that this piece never save to my harddisk before.

That is very unlikely. It would mean that the satellite databases are holding wrong information about the pieces.
More likely is that it is an older piece you don’t have logs for anymore. Depending on your setup they get deleted frequently.
A missing pieces gets audited several times until marked as finally lost. I am not sure but I think it is only 3 times or something? After that you should not receive any more audits for the same piece AFAIK.

[Tommy]

That should be a grate news if it do not audit the missing piece again and again so I don’t need to worry about the audit score becomes too low and disqualify my node.

[Alexey]

@jammerdan

I am not sure but I think it is only 3 times or something?

Unfortunately not. It will be audited until deleted or recovered to another node.
Just with increased used storage it could be audited less frequently.

[jammerdan]

@Alexey

Unfortunately not. It will be audited until deleted or recovered to another node.

Oh. Really?
How does this makes sense?

[Alexey]

It has no memory of a previous audits, it just update an audit score and go on.
Keep in mind, there are many audit workers.

[jammerdan]

Just saw this on Github:

GET_AUDIT - File does not exist