Asia-east Suspended/Not recognized

Node Operators troubleshooting
1

I just got an e-mail about being suspended on asia-east, and I suspect that my node hasn’t gotten the information that asia-east is legit. I’m on Ubuntu 20, node version 1.3.3 via docker.
The error in the logs is as follows:

2020-05-06T16:49:06.107Z ERROR piecestore download failed {“Piece ID”: “RVPFQK7LSDAZ3YKEBPKTYZTPFPP665L2L55XOR5S7UG53TVCOUDQ”, “Satellite ID”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”, “Action”: “GET_AUDIT”, “error”: “untrusted: unable to get signee: trust: rpccompat: context deadline exceeded”, “errorVerbose”: “untrusted: unable to get signee: trust: rpccompat: context deadline exceeded\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).VerifyOrderLimitSignature:140\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).verifyOrderLimit:62\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload:523\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Download:471\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:995\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:107\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:66\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:111\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:62\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:99\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51”}
If I run the earnings.py script, it doesn’t show the asia-east satellite has having any traffic.
In addition, with the update to 1.3.3 the web dashboard broke, but the internal dashboard.sh is still showing online.
(Broke=shows offline, last contact counts from the page refresh time. The page layout loads, but all of the active content does not; it acts like it cannot read from the api )

Docker command:
#!/bin/bash

docker run -d --restart unless-stopped -p 28967:28967 -p 14002:14002
-e WALLET=“xxxx”
-e EMAIL=“xxxxx”
-e ADDRESS=“xxxxx” \

-e STORAGE=“7000GB”
–mount type=bind,source="/home/gregsachs/.local/share/storj/identity/storagenode",destination=/app/identity
–mount type=bind,source="/mnt/5222779d-0126-4c43-a34e-d95f0fc904b4",destination=/app/config
–name storagenode storjlabs/storagenode:beta

2

Ok, at least part of this was that I had GEOIP blocking on asia. I removed that, and now the dashboard will load. However, when restarting the container, now my node is not going back online…

3

Online now, I believe router config got corrupted when I changed the geoip setting.
So, how do I get unsuspended from asia-east?

4

Your node should not fail audits anymore, then it will out from suspension automatically.
Have you figured why you have a failed audits?

5

Yes, all should be good now. I had parts of Asia blocked at firewall. That was failing the asia satellite. That issue, which I believe included blocking it from identifying the asia-east satellite, was breaking the dashboard as well.

6

The blocking firewall should not fail audits. Your node just will not respond on audit requests and should not be online on that satellite and thus - no audit requests.
To fail an audit your node should be online on that satellite, response on audit request, but do not provide the requested piece for any reason in a limited time. Three failures for the same piece - and audit considered as failed.
So, please, take a look on your storage and logs - why you failing audits.

7

I am also using a GeoIp filtering, and I have half a planet in the filter list, blocking the input part. I’ve got the node up and running, and so far I’m having no problems. Can it cause any problems in the future?

8

Perhaps. Allowing only satellite will not allow customers of that satellite, because traffic is going directly between customers and node.

1 Like
9

Is it plausible that my firewall was blocking the satellite, but not the customers, thus failing the audits, but still getting traffic?

10

There is a problem: I do not know, what exactly you have blocked.
To receive data from customers, your node should be advertised by the satellite to its customers. It’s possible only if your node checked-in on that satellite and showed online status and available space.
Then customers can try to transfer their data to your node. After that satellite will come to your node and ask for audits. If node is responded on audit request, but didn’t provide the requested piece (or returned an error - this is why the suspended mode is here - to figure out those errors), it will be placed into the containment mode and will not receive any traffic but audits. Then it will be asked two more times for the same piece, if the node didn’t provide a right information, this audit considered as failed.
In case of answer with a wrong information (modified or broken pieces or fake response) the audit considered as failed immediately.

So, to be placed into suspended state your node should answer with error. This is why I suggest you in a third time to search GET_AUDIT and failed in your logs.

11

I have searched in the logs. from 5/2, I have a single database is locked error on usedserials. I have a bunch of the error mentioned above, about untrusted signee.

12

Fair enough, let’s hope it went out from the suspended state and working normally