Your Node Is Disqualified on the us-central-1 Satellite

[readonly]

How this could happen just on one satellite? I got an email with that statement, without any previous warnings.

No interventions done recently over the storj node.

Node ID: 12f9cUsSgipDzUrkeS2iLbVkqo6DanmdoLTnX3bXRGvDCmVVFap

[Alexey]

Please, check your logs for failed GET_AUDIT and GET_REPAIR
Files related to each satellite are stored in a different folders. Perhaps customers of this satellite have stored most of the data on your node.

[readonly]

I believe in the implementation and probably there are some… but it is the other way - it is the satellite which triggered the email and this process, so I would expect this perspective from the support.
In terms of when, why etc. leaving that it is not logical on heavily used node with 3+TiB stored

[Alexey]

We have such a feature request in the backlog, but the priority is low.

Usually search for errors in logs is enough - you could clearly see a reason (“file not found” for example).

Please, search for errors first and we can proceed further.

[readonly]

you cannot attach a file… anyway, filtered/random extracted ERRORs of the docker node log.

2021-06-11T01:11:09.202Z ERROR piecestore download failed {“Piece ID”: “ETNJQEM2RSOCL6543UX3NDG3QL7DRP5O2YPAADFWSOVGIBINFOPQ”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “GET_REPAIR”, “error”: “file does not exist”, “errorVerbose”: “file does not exist\n\tstorj.io/common/rpc/rpcstatus.Wrap:74\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:508\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:217\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:111\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:62\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:99\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51”}
2021-06-11T01:13:45.020Z ERROR piecestore upload failed {“Piece ID”: “QZIZB2RPAG3I24BYU2HDFKC4RM26TVPJIFEYE7MM57T3S3DXTBXQ”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “error”: “tls: use of closed connection”, “errorVerbose”: “tls: use of closed connection\n\tstorj.io/drpc/drpcstream.(*Stream).RawFlush:276\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:315\n\tstorj.io/common/pb.(*drpcPiecestore_UploadStream).SendAndClose:277\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload.func6:409\n\tstorj.io/common/rpc/rpctimeout.Run.func1:22”, “Size”: 24064}
2021-06-11T01:20:07.835Z ERROR piecestore upload failed

2021-06-12T10:55:52.723Z ERROR piecestore download failed {“Piece ID”: “TMFDVGMWDXYU7BTKU3WLBMTPBDL2I7YVQV4DQWDVKNSEKW25DMHQ”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “GET_REPAIR”, “error”: “file does not exist”, “errorVerbose”: “file does not exist\n\tstorj.io/common/rpc/rpcstatus.Wrap:74\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:508\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:217\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:111\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:62\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:99\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51”}
2021-06-12T10:56:11.035Z ERROR piecestore download failed

2021-06-16T06:53:55.766Z	ERROR	piecestore	download failed	{Piece ID: 4YRHE6N4MEWVNVAS3EFQY7IAWO5S6ILOHVOZZZLXD2NWHEMVVH3A, Satellite ID: 12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S, Action: GET_REPAIR, error: file does not exist, errorVerbose: file does not exist\n\tstorj.io/common/rpc/rpcstatus.Wrap:74\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:508\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:217\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:111\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:62\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:99\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51}
2021-06-16T06:54:46.052Z	ERROR	piecestore	download failed	{Piece ID: HWWHP74DBWKPF5U55Z43RYWAAKBG5Z235WC6BLB23LS3ZTUXN6XA, Satellite ID: 12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S, Action: GET_REPAIR, error: file does not exist, errorVerbose: file does not exist\n\tstorj.io/common/rpc/rpcstatus.Wrap:74\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:508\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:217\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:111\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:62\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:99\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51}

…and most recent entries

2021-06-17T06:00:01.642Z	ERROR	piecestore	download failed	{Piece ID: WIT4URGIBFLVYWQHNZZLAXZH4J3Q2VPJ6VSMMBFYRHN7GQ7AWCZQ, Satellite ID: 12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S, Action: GET_AUDIT, error: file does not exist, errorVerbose: file does not exist\n\tstorj.io/common/rpc/rpcstatus.Wrap:74\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:508\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:217\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:111\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:62\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:99\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51}
2021-06-17T06:17:54.712Z	ERROR	piecestore	upload failed	{Piece ID: 3PZ2CA4FO5JKKUMDBKFDLUQGQHKHCPE7O2RDIGHNKFEPAQOG2QQQ, Satellite ID: 12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs, Action: PUT, error: unexpected EOF, errorVerbose: unexpected EOF\n\tstorj.io/common/rpc/rpcstatus.Error:82\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:327\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:209\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:111\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:62\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:99\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51, Size: 0}
2021-06-17T06:24:31.188Z	ERROR	piecestore	upload failed	{Piece ID: 6WIK35V43QMMGPQ7OF3EA57GMVYSDXJQHZ5TCUVMU4LYEFRJAUWQ, Satellite ID: 12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs, Action: PUT, error: unexpected EOF, errorVerbose: unexpected EOF\n\tstorj.io/common/rpc/rpcstatus.Error:82\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:327\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:209\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:111\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:62\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:99\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51, Size: 0}
2021-06-17T07:26:36.702Z	ERROR	piecestore	upload failed	{Piece ID: Q7UIIBGNUY5NEDPCRRX4HNESNZRYNVCLT352IMN6NCYYQHR6FO6Q, Satellite ID: 12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs, Action: PUT, error: context deadline exceeded, errorVerbose: context deadline exceeded\n\tstorj.io/common/rpc/rpcstatus.Wrap:74\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:329\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:209\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:111\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:62\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:99\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51, Size: 0}

[Alexey]

PowerShell

(sls "GET_AUDIT|GET_REPAIR" "C:\Program Files\Storj\Storage Node\storagenode.log" | sls failed).Count

bash

docker logs storagenode 2>&1 | grep -E "GET_AUDIT|GET_REPAIR" | grep failed -c