Audit Fail: Order created too long ago

Node Operators troubleshooting
1

Hi,

I just got a couple of these errors in my logs from different nodes on the same server, should i be worried? Suspension score dropped to 98 - 95 and 92%

2020-10-26T21:40:19.732Z ERROR piecestore download failed {“Piece ID”: “Z6RTC7XEAGPR5WJ6NYAXMKBR3FLI6JILPSCLVQGXGWMRNHDTQXDQ”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “GET_AUDIT”, “error”: “order created too long ago: OrderCreation 2020-10-26 18:36:18.25454345 +0000 UTC < SystemClock 2020-10-26 21:40:19.731915518 +0000 UTC m=+1313.118877418”, “errorVerbose”: “order created too long ago: OrderCreation 2020-10-26 18:36:18.25454345 +0000 UTC < SystemClock 2020-10-26 21:40:19.731915518 +0000 UTC m=+1313.118877418\n\tstorj.io/common/rpc/rpcstatus.Errorf:87\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).verifyOrderLimit:45\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:461\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:1004\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:107\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:56\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:111\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:62\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:99\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51”}

2 Likes
2

Any failed audit should concern you.
in this case, please, sync clocks on your host with storagenode.
If you use a VM, make sure that you have an integration service installed. Then sync clocks between host and VM.
Stop and remove the container, then run it again (docker) or restart the storagenode service (binary, i.e. Windows GUI).

3

Im running esxi, host clock is set to be synced with vm (storj is running in photonOS vm-s)

How should i sync the clock?

4

Install the integration services for your hypervisor to the guest OS. Or install an NTP on your guest OS

5

Esxis ntp server didn sync, and 1 hour behind, but the vms where syncing to an NTP server.

What should i be looking into? To see if this fixed it?

6

You should use either NTP inside a guest OS and sync it or use integration service from the hypervisor and sync clocks on the host.
The TZ of the host doesn’t matter unless there is a bug in hypervisor code or in the integration services of that hypervisor.

7

I think its ok now, suspension score is growing back to 100%
But still don’t know why this happened, all the guest OS had ntp enabled.