Disqualified from saltlake

windex · June 24, 2020, 7:43pm

I understand that either data missing from my node or my node being down too long causes it to be disqualified. I received an email saying my node has been disqualified from the saltlake satellite. I am certain that I haven’t lost any data as I’m certain I haven’t had any disk failures. I’m a little curious as to whats the difference between the audit checks and the uptime check. Also does this mean I would have to start all over or is my node still valid on the other satellites?

deathlessdd · June 24, 2020, 9:07pm

Right now you can only get disqualified from audits failing downtime doesn’t affect it yet. So you need to find your logs and check for audits failed.

kevink · June 24, 2020, 9:10pm

With an audit check value of 66.3% you either lost data or your node was unresponsive to audit checks (e.g. if you have a SMR drive that could happen or if you had too many “database is locked” errors).
But you would need to check in the logs.

Alexey · June 24, 2020, 10:33pm

database is locked error can only suspend the node.
To be disqualified your node should fail audits because of missing or inaccessible data.

windex · June 25, 2020, 12:50am

Right before I got the email for being disqualified the logs are showing a bunch of this:
2020-06-24T19:48:32.205Z ERROR piecestore download failed {“Piece ID”: “WKYQQW7RHPAV2MSRLDJZWB6UZNPVBXMAFKFUVGC46VUBVGDW4GKA”, “Satellite ID”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”, “Action”: “GET”, “error”: “usedserialsdb error: context canceled”, “errorVerbose”: “usedserialsdb error: context canceled\n\tstorj.io/storj/storagenode/storagenodedb.(*usedSerialsDB).Add:35\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).verifyOrderLimit:76\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:459\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:1004\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:107\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:56\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:111\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:62\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:99\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51”}

2020-06-24T19:48:32.205Z	ERROR	piecestore	download failed	{"Piece ID": "ZUN5AVMMPB24ZHTW6FWLVJUEJKDLWEB736GX4B52SKCD3HMIC4JQ", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "GET_REPAIR", "error": "usedserialsdb error: context canceled", "errorVerbose": "usedserialsdb error: context canceled\n\tstorj.io/storj/storagenode/storagenodedb.(*usedSerialsDB).Add:35\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).verifyOrderLimit:76\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:459\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:1004\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:107\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:56\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:111\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:62\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:99\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51"}


2020-06-24T19:48:32.208Z	ERROR	piecestore	download failed	{"Piece ID": "6QEOY7TAL45A6BLRZRWZFYI3FP5V6CU5KSMOGJYQOSTFBCLNKFBA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "error": "usedserialsdb error: context canceled", "errorVerbose": "usedserialsdb error: context canceled\n\tstorj.io/storj/storagenode/storagenodedb.(*usedSerialsDB).Add:35\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).verifyOrderLimit:76\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:459\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:1004\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:107\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:56\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:111\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:62\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:99\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51"}


2020-06-24T19:48:32.208Z	ERROR	piecestore	download failed	{"Piece ID": "ZAN4CJ3VNOE7H4W3GTNW6UHRLSYVC7HCUOLIXVTE4ZPLXHKPOEVA", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "GET_AUDIT", "error": "usedserialsdb error: context canceled", "errorVerbose": "usedserialsdb error: context canceled\n\tstorj.io/storj/storagenode/storagenodedb.(*usedSerialsDB).Add:35\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).verifyOrderLimit:76\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:459\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:1004\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:107\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:56\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:111\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:62\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:99\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51"}


2020-06-24T19:48:32.208Z	ERROR	piecestore	download failed	{"Piece ID": "BAVRTE4LKHNGCTOS7WFG4PEFAVSSYM2GXMOG32Y4S4BEZZBHNDUA", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "GET_REPAIR", "error": "usedserialsdb error: context canceled", "errorVerbose": "usedserialsdb error: context canceled\n\tstorj.io/storj/storagenode/storagenodedb.(*usedSerialsDB).Add:35\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).verifyOrderLimit:76\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:459\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:1004\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:107\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:56\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:111\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:62\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:99\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51"}


2020-06-24T19:48:32.223Z	ERROR	piecestore	download failed	{"Piece ID": "4TNBCBXGKNGIFXXRZLG7GBT5IEEX5SARRWUO3RPGRA75HVQ6O2TQ", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "GET_REPAIR", "error": "usedserialsdb error: context canceled", "errorVerbose": "usedserialsdb error: context canceled\n\tstorj.io/storj/storagenode/storagenodedb.(*usedSerialsDB).Add:35\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).verifyOrderLimit:76\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:459\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:1004\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:107\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:56\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:111\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:62\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:99\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51"}

nerdatwork · June 25, 2020, 1:00am

How is your HDD connected ?

windex · June 25, 2020, 1:11am

Its 6 Sas drives in raid6.

Alexey · June 25, 2020, 7:03am

Can you search for GET_AUDIT and failed and not usedserialsdb in your logs?

Docker Linux

docker logs storagenode 2>&1 | grep GET_AUDIT | grep failed | grep -v usedserialsdb

Docker Windows (in Powershell)

docker logs storagenode 2>&1 | Select-String GET_AUDIT | Select-String failed | Select-String -NotMatch usedserialsdb

Windows GUI (in Powershell)

Select-String GET_AUDIT "$env:ProgramFiles\Storj\Storage Node\storagenode.log" | Select-String failed | Select-String -NotMatch usedserialsdb