Audit Score decreasing slowly

Node Operators troubleshooting
1

Hi,

I have noticed that my audit score has slowly been decreasing over the last week or so.

I took a look at my logs and i’m getting an error:

2023-05-13T13:55:21.702Z ERROR piecestore download failed {“Process”: “storagenode”, “Piece ID”: “TAV6S2VYLV4Y7OYXME5WBSFIL5576N4DTNJCCGNT5D3H2TZR4AVA”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “GET_AUDIT”, “Offset”: 1147136, “Size”: 0, “Remote Address”: “34.138.159.102:38408”, “error”: “file does not exist”, “errorVerbose”: “file does not exist\n\tstorj.io/common/rpc/rpcstatus.Wrap:75\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:650\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:251\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:61\n\tstorj.io/common/experiment.(*Handler).HandleRPC:42\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:124\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:114\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35”}

There are many others like it, as well as errors for GET_REPAIR

How shall i go about fixing this problem?

2

Apparantly you lost some files or you can’t read the files. On what OS are you operating? Linux or Windows? Are the filesystems accessible? No errors in system log?

1 Like
3

Seems your node managed to lost a piece.
Can you search for this piece in your logs, when it was uploaded and perhaps when it was lost?

4

i’m on Linux using Docker for the node. i can access all the files through SFTP and it seems to be fine.

5

So, i was running the node for a few days in april but the entire VM corrupted, i then reisntalled but obviously couldn’t get the data back which it had already downloaded. Could that be the reason it’s saying that files are lost? I think i lost about 20GB :frowning:

6

I would consider to start all over with the node. Especially if you don’t know how much you have lost and with this young node. But I’m really curious how your setup is. Since failing this soon, is kinda worrisome.

With 20GB you have to compensate with about 500GB, in order not to be disqualified within a month or so.

No messages in dmesg?

7

I’ll start over then. no messages in dmesg either.

thanks for the help :slight_smile:

8

If it was corrupted, just curious - why do you wonder on failing audits?
If more than 4% data is lost - the node will be disqualified.
So, I guess this is correct decision to start from scratch.

However, it’s better to figure out why your VM was corrupted to not meet the same with a new node. Could you please describe your setup? Especially why data was corrupted if it’s supposedly on a separate disk?

Did you know that you may run a node in a docker container on the host to reduce abstraction levels and resources usage?

9

You should definitely make sure that a new VM has a seperate disk for the Storj stuff that is independent of the VM so that you can just move the disk to a new VM if something fails.
Do not use the virtual disk of the VM itself for storage.

1 Like
10

Agree, that’s for sure a recipe for terrible troubles coming over you.