Node Suspension

goldyka · April 22, 2020, 11:15pm

restarted the node with the above command, still showing more free space on dashboard then with df -h

masta · April 22, 2020, 10:15pm

I have received an email that my node is suspended.
But when I use the dashboard it says it is up and running.
So was there a mistake?

jocelyn · April 22, 2020, 10:23pm

Ive pinged the on-call engineer and asked them to hop in with reply @brandon

masta · April 22, 2020, 10:24pm

Do they need my node’s ID or any other info from me?
Thanks!

Lukasz · April 22, 2020, 10:59pm

I got similar if not the same e-mail saying that my Node is suspended referring one of the '“us-central-1” satellite. I have checked the “uptime checks” and that one has the lowest from all other satellites 83.2%. Audit check is 98.5%. What do I suppose to do with that? Thank you. Node_suspended

deathlessdd · April 22, 2020, 11:16pm

You may need to do it a few times to soft restart your node.

itnok · April 22, 2020, 10:53pm

what should I do now?

The email is not really clearing out how should I behave to get back on track: have no idea at all on how to “resolve the issue causing audit failures on your node”, because have no idea about the issue…

Can anybody please help?

Thanks

jocelyn · April 22, 2020, 11:12pm

Hi @itnok, have you seen this thread ?

Suspension mode and disqualification emails

Blueprint: Downtime Disqualification

I also asked our oncall engineer to hop in and clarify further when they are online. But those 2 posts are a great place to start

goldyka · April 22, 2020, 11:17pm

its the same for me… so much for transparency

Alexey · April 22, 2020, 11:19pm

Please, search for failed audits in your logs

goldyka · April 22, 2020, 11:19pm

posted them above, but have no idea how to fix it. Help

Alexey · April 22, 2020, 11:26pm

I do not see audits errors in your screenshots.
Regarding ordersdb is locked: Ordersdb error: database is locked
Please, post logs in text, the picture is hard to analyze.
Please, restart your node and check is your error regarding boltdb is gone?
How is your hdd connected?

goldyka · April 22, 2020, 11:33pm

2020-04-22T23:32:50.275Z ERROR piecestore failed to add bandwidth usage {“error”: “bandwidthdb error: database is locked”, “errorVerbose”: “bandwidthdb error: database is locked\n\tstorj.io/storj/storagenode/storagenodedb.(*bandwidthDB).Add:59\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).saveOrder:721\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:443\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Upload:215\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:987\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:107\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:105\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:56\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:93\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51”}
2020-04-22T23:32:50.276Z INFO piecestore uploaded {“Piece ID”: “VGJMND5WD4AGFSXUGGKUAASSOZM43BIDMDEASOH7ILIUXYKVUTGQ”, “Satellite ID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”, “Action”: “PUT”}
2020-04-22T23:32:50.290Z ERROR piecestore failed to add bandwidth usage {“error”: “bandwidthdb error: database is locked”, “errorVerbose”: “bandwidthdb error: database is locked\n\tstorj.io/storj/storagenode/storagenodedb.(*bandwidthDB).Add:59\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).saveOrder:721\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:443\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Upload:215\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:987\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:107\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:105\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:56\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:93\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51”}
2020-04-22T23:32:50.290Z INFO piecestore uploaded {“Piece ID”: “33WBH4GHDZQTDNLKJVXUWKA5AVRPMZ36I33M7SB7WUMAXMQSH5CQ”, “Satellite ID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”, “Action”: “PUT”}
2020-04-22T23:32:50.857Z INFO piecestore download started {“Piece ID”: “F7YHOQQCC3EVZYYWKFLHARRNQVK5S2CPGNLQ3VD2XXGJ62M2SRQA”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “GET”}
2020-04-22T23:32:53.458Z ERROR piecestore failed to add bandwidth usage {“error”: “bandwidthdb error: database is locked”, “errorVerbose”: “bandwidthdb error: database is locked\n\tstorj.io/storj/storagenode/storagenodedb.(*bandwidthDB).Add:59\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).saveOrder:721\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:443\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Upload:215\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:987\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:107\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:105\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:56\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:93\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51”}
2020-04-22T23:32:53.459Z INFO piecestore upload canceled {“Piece ID”: “J4R2R4NE5Y7P7YV2SLSKWYLDMQ3LA6Y6ADTUJPWJD74WGM2IN3UA”, “Satellite ID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”, “Action”: “PUT”, “error”: “context canceled”, “errorVerbose”: “context canceled\n\tstorj.io/common/pb/pbgrpc.init.0.func3:70\n\tstorj.io/common/rpc/rpcstatus.Wrap:77\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:452\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Upload:215\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:987\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:107\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:105\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:56\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:93\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51”}
2020-04-22T23:32:56.541Z ERROR piecestore failed to add bandwidth usage {“error”: “bandwidthdb error: database is locked”, “errorVerbose”: “bandwidthdb error: database is locked\n\tstorj.io/storj/storagenode/storagenodedb.(*bandwidthDB).Add:59\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).saveOrder:721\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload.func6:674\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload:695\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Download:466\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:995\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:107\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:105\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:56\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:93\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51”}
2020-04-22T23:32:56.542Z INFO piecestore download canceled {“Piece ID”: “SJXFBZSYJ434HQH5MXEO5XDZ6ZG7I7ZY64DXMLNTUVJHVH2YEBNQ”, “Satellite ID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”, “Action”: “GET”, “error”: “context canceled”, “errorVerbose”: “context canceled\n\tstorj.io/common/pb/pbgrpc.init.0.func3:70\n\tstorj.io/common/rpc/rpcstatus.Wrap:77\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload.func5:646\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}

My hdd is connected via USB 3

Alexey · April 22, 2020, 11:35pm

Please, try to restart the storagenode

goldyka · April 22, 2020, 11:36pm

restarted it multiple times, my current uptime on the node is 17 minutes
also mounted hdd in fstab

itnok · April 22, 2020, 11:37pm

How does a failed audit look like in the logs?

If I found failed audits (it’s likely because of this… NTP Blocked by Provider... storagenode does not start), what should I do, then?

Thanks

Alexey · April 22, 2020, 11:39pm

Your node can fail audit only if data is inaccessible when the node is online.
Downtime is a separate metric and your node will not be suspended until this check will be enabled back.

goldyka · April 22, 2020, 11:40pm

itnok · April 22, 2020, 11:41pm

This is the output of (your?) script for node stats…

========== AUDIT ==============
Password:
Critically failed:     0
Critical Fail Rate:    0.000%
Recoverable failed:    0
Recoverable Fail Rate: 0.000%
Successful:            15
Success Rate:          100.000%
========== DOWNLOAD ===========
Failed:                0
Fail Rate:             0.000%
Canceled:              10
Cancel Rate:           3.165%
Successful:            306
Success Rate:          96.835%
========== UPLOAD =============
Rejected:              4168
Acceptance Rate:       40.711%
---------- accepted -----------
Failed:                0
Fail Rate:             0.000%
Canceled:              1246
Cancel Rate:           43.536%
Successful:            1616
Success Rate:          56.464%
========== REPAIR DOWNLOAD ====
Failed:                0
Fail Rate:             0.000%
Canceled:              0
Cancel Rate:           0.000%
Successful:            16
Success Rate:          100.000%
========== REPAIR UPLOAD ======
Failed:                0
Fail Rate:             0.000%
Canceled:              32
Cancel Rate:           37.647%
Successful:            53
Success Rate:          62.353%
========== DELETE =============
Failed:                0
Fail Rate:             0.000%
Successful:            321
Success Rate:          100.000%

Alexey · April 22, 2020, 11:42pm

So, you do not have failed audits since the recreation of the container.