Untrusted: unable to get signee

Node Operators troubleshooting
,
1

Suddenly one of my nodes stopped working with “error”: “untrusted: unable to get signee: trust: rpc: tcp connector failed: rpc: dial tcp: lookup us1.storj.io: operation was canceled”
I’m using NOIP address. Deleted all .db files. Deleted trust-cache.json (I’m trying everything).
Unmounted disk and “e2fsck -f” it.
Maybe identity file corruption? It make sense?

2

Check if your ISP didnt put you behind CGNAT. Check if you renewed your NOIP subscription. It needs renewal every 30 days on free tier.

3

More nodes in the same server are working as usual. Ports are open. Noip paid.

4

The error is indicating that the system couldn’t resolve the hostname us1.storj.io. You should see whether you can resolve the hostname from the command line using:

dig us1.storj.io

You should see some IPv4 addresses (A records) in the output. For example, I get

<...>
;; ANSWER SECTION:
us1.storj.io.		60	IN	A	34.172.100.72
us1.storj.io.		60	IN	A	34.150.199.48
<...>

That lookup should work on the host. If it doesn’t, something is wrong with your DNS and needs to be fixed. If it works, then the next step is to try the lookup inside the docker container.

The docker container for the storagenode is pretty minimal and doesn’t include common DNS debugging tools like dig or nslookup. But it does have openssl, so (assuming the container is running) you can try

docker exec storagenode openssl s_client us1.storj.io:443

You should get a lot of output, starting with something like

depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = R3
verify return:1
depth=0 CN = us1.storj.io
verify return:1
CONNECTED(00000003)

and then including a certificate chain.

If that does not work, something is probably wrong with the network bridge between the container and the host, or the container was otherwise not set up correctly. If it does work, then we’ll need to do some more investigation to figure out why openssl can do DNS lookups fine but storagenode can’t.

3 Likes