Node Suspension

[goldyka]

I just got an e-mail from STORJ that my node for us-central is suspended, because of audit fails.

Today for first time since i’m running my node, it just stopped. Restarted the OS, started up docker again and the dashboard showed that it is online, and got more and more data during the day.

After i got the e-mail, i checked my log:

…

2020-04-22T22:04:35.317Z ERROR piecestore failed to add order {“error”: “ordersdb error: database is locked”, “errorVerbose”: “ordersdb error: database is locked\n\tstorj.io/storj/storagenode/storagenodedb.(*ordersDB).Enqueue:53\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).saveOrder:714\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:443\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Upload:215\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:987\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:107\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:105\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:56\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:93\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51”}
2020-04-22T22:04:35.317Z ERROR piecestore failed to add order {“error”: “ordersdb error: database is locked”, “errorVerbose”: “ordersdb error: database is locked\n\tstorj.io/storj/storagenode/storagenodedb.(*ordersDB).Enqueue:53\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).saveOrder:714\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:443\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Upload:215\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:987\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:107\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:105\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:56\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:93\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51”}
2020-04-22T22:04:35.317Z INFO piecestore uploaded {“Piece ID”: “BJTKDDBCXGN6QYISIZ4KM5G6TUZYJGXMQP7YV5JCVGS7IXLVD7QQ”, “Satellite ID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”, “Action”: “PUT”}
2020-04-22T22:04:35.317Z INFO piecestore upload canceled {“Piece ID”: “472Y5Y7YMM5SWNRJPFXQTPBEVCNUOTHRQMBI4IUHKHHET7RSCWJA”, “Satellite ID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”, “Action”: “PUT”, “error”: “context canceled”, “errorVerbose”: “context canceled\n\tstorj.io/common/pb/pbgrpc.init.0.func3:70\n\tstorj.io/common/rpc/rpcstatus.Wrap:77\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:452\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Upload:215\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:987\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:107\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:105\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:56\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:93\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51”}
2020-04-22T22:04:35.668Z ERROR piecestore failed to add order {“error”: “ordersdb error: database is locked”, “errorVerbose”: “ordersdb error: database is locked\n\tstorj.io/storj/storagenode/storagenodedb.(*ordersDB).Enqueue:53\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).saveOrder:714\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:443\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Upload:215\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:987\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:107\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:105\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:56\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:93\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51”}
2020-04-22T22:04:35.668Z INFO piecestore uploaded {“Piece ID”: “VLXMXQZHT6JNE6XNS3NWB5YPHEM3B6ZRFQ6AXIHNYNAIUI7WKJCA”, “Satellite ID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”, “Action”: “PUT”}
2020-04-22T22:04:36.611Z ERROR piecestore failed to add order {“error”: “ordersdb error: database is locked”, “errorVerbose”: “ordersdb error: database is locked\n\tstorj.io/storj/storagenode/storagenodedb.(*ordersDB).Enqueue:53\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).saveOrder:714\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:443\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Upload:215\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:987\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:107\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:105\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:56\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:93\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51”}
2020-04-22T22:04:36.611Z INFO piecestore uploaded {“Piece ID”: “CPR3E5TO6HBVBUUOX7BTJKGISJDCMTMDFWNS4JHZAVTYXM3HSUUA”, “Satellite ID”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”, “Action”: “PUT”}
2020-04-22T22:04:38.452Z ERROR piecestore failed to add order {“error”: “ordersdb error: database is locked”, “errorVerbose”: “ordersdb error: database is locked\n\tstorj.io/storj/storagenode/storagenodedb.(*ordersDB).Enqueue:53\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).saveOrder:714\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:443\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Upload:215\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:987\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:107\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:105\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:56\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:93\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51”}
2020-04-22T22:04:38.453Z INFO piecestore uploaded {“Piece ID”: “VGZDMMU3S5PZK2MRWZEVN7BNNTNZZETSYUGDDDIQH5QZ6SOEW6HQ”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”}
2020-04-22T22:04:38.462Z ERROR piecestore failed to add order {“error”: “ordersdb error: database is locked”, “errorVerbose”: “ordersdb error: database is locked\n\tstorj.io/storj/storagenode/storagenodedb.(*ordersDB).Enqueue:53\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).saveOrder:714\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:443\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Upload:215\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:987\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:107\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:105\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:56\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:93\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51”}
2020-04-22T22:04:38.463Z INFO piecestore uploaded {“Piece ID”: “2C3ZXHQYGPSVUHEGXNRPOBFZZB333QS35U3MQBN7Q6I23D4YL4KA”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “PUT”}
2020-04-22T22:04:38.469Z ERROR piecestore failed to add order {“error”: “ordersdb error: database is locked”, “errorVerbose”: “ordersdb error: database is locked\n\tstorj.io/storj/storagenode/storagenodedb.(*ordersDB).Enqueue:53\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).saveOrder:714\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:443\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Upload:215\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:987\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:107\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:105\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:56\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:93\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51”}

…

Any ideas what i should do?

[masta]

I got a similar email. My node has been running 24/7 non-stop. And today this. There is gotta be an error with these emails. The node is up and running still.

[jocelyn]

Ive pinged the on-call engineer and asked them to hop in with reply @brandon

[goldyka]

[goldyka]

do you have any errors in your logs?

[goldyka]

Just got another e-mail from storj, also suspended on Asia.

HELP!!!

Node is Online, also did Ctrl+F5
Port is open

[deathlessdd]

Suspension mode and disqualification emails

Later today we are going to send out a bunch of automated emails to storage nodes in suspension mode and to disqualified storage nodes. You might receive multiple of these emails one from each satellite. Time to explain what you should do if you get one of these emails. First of all, give us feedback. Is the email useful for you? Is there something you would like us to change? Yes, the first email might arrive a bit late. In the future, we send out these emails a few hours after you got disqual…

[goldyka]

Thanks, but how do i fix it? The support link i got in the e-mail is not helping…

[deathlessdd]

Well first you need to figure out why you got the email in the first place im sure you wont be the last one to get it. This issues could be many things if you failed alot of audits for database lock means you could have an issue with your hard drive loading fast enough, How is your drive connected, how old is your drive, Is your hard drive SMR?

[goldyka]

used to audit check tool found on the support page:

[deathlessdd]

Is your drive full did you allow for an extra 10%?

[goldyka]

Hdd is connected to Pi4 via USB 3.0
Hard drive is 5 months old, got it brand new and only used it for STORJ.
mounted as sda1

[deathlessdd]

Is this the only node you have with this email?

[goldyka]

yes the only one, checked the node ID and it matches up with this one

[deathlessdd]

Well you could have already fixed the issue, This was meant as a warning to you to fix your node it could just be a delay. Are you still getting data on your node I saw your screen shot your node was up 28mins. Worst thing to do is panic you did something wrong and really do something wrong.

[goldyka]

Yes im still getting data on it. The only thing i found and i dont know why it’s happening is:
In my dashboard it says ive got 708GB of free space

but df -h is showing 678GB

also temp and garbage and trash folder is full of data dating back to 14th of April

[deathlessdd]

Its because your database was locked just run a docker restart -t 300 storagenode
Its cause you either restarted your node or stopped and changed a docker start command. Probably didnt recalculated the amount of space.

[jocelyn]

Hi there, have you seen this thread @lukasz and @asta ?

Suspension mode and disqualification emails

Blueprint: Downtime Disqualification

I also asked our oncall engineer to hop in and clarify further when they are online. But those 2 posts are a great place to start

[jocelyn]

oops i may have posted this in a different thread than intended!

[mortal]

I got an email saying my node is disqualified, the id from the email matches my node id in the dashboard. It’s been running 24/7 for months.

Nothing has changed about my hardware or internet for weeks, and it’s still showing ONLINE and also still showing net traffic to other storj nodes (20-40mbit down, 30-40mbit up).

I am lost on how to diagnose if there really is a problem or the email is in error. I don’t have the knowledge or experience to deal with this and I am sadface emoji.