Offline after overnight Broadband outage

Node Operators troubleshooting
, ,
1

Ubuntu server 19.04
Node Version : 0.26.2

My internet went down for the 2nd time in as many days (BT in the UK) I noticed my node is offline

I’ve restarted the container AND the server but it’s still offline.

I’ve checked my Dynamic DNS and it’s fine with port open and accessible

Here is the error from the logs:

2019-11-28T14:21:24.175Z ERROR contact:chore ping satellite failed {“Satellite ID”: “118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW”, “attempts”: 8, “error”: “ping satellite error: check-in network error: failed to resolve IP from address: storj.hopto.me, err: address storj.hopto.me: missing port in address”, “errorVerbose”: “ping satellite error: check-in network error: failed to resolve IP from address: storj.hopto.me, err: address storj.hopto.me: missing port in address\n\tstorj.io/storj/storagenode/contact.(*Chore).pingSatellite:127\n\tstorj.io/storj/storagenode/contact.(*Chore).Run.func1:80\n\tstorj.io/storj/private/sync2.(*Cycle).Run:87\n\tstorj.io/storj/private/sync2.(*Cycle).Start.func1:68\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}

any ideas?

2

It’s not uncommon for IP addresses to change after an outage, even if they’re normally static. Double check your external and internal IP address. If the external address is changed, update your run command and recreate your container. If the internal address has changed, update your port forwards.

This is assuming you used a static IP in the run command, if you used a domain, update the DNS record. If you used DDNS, make sure the update client is running and updating your IP.

3

I have multiple times, (router is set to auto-update noip on re-connection) and I’ve manually went to the noip site to check the IP address and it’s correct and accessible.

4

Well, then it’s double checking everything. Please refer to this post and try all suggestions. Make sure you check everything even if you think it can’t have changed.

5

I have double checked everything in that FAQ a couple of times.

Everything is set up as it should be.

Still getting the “ERROR contact:chore ping satellite failed” message.

It looks like the errors started becoming MORE frequent when 0.26.2 was released. Before that it was a one or 2 times a day, then after the update it was a few times an hour till the 26th (first outage) then it was multiple blocks of errors every few hours, til last nights outage, then it was constant errors.

6

Check your dns settings, just try to ping satellite.stefan-benten.de from your host and see what happens.

7

Ping ran fine :

ping -c4 satellite.stefan-benten.de
PING satellite.stefan-benten.de (78.94.240.189) 56(84) bytes of data.
64 bytes from prod.stefan-benten.de (78.94.240.189): icmp_seq=1 ttl=50 time=66.8 ms
64 bytes from prod.stefan-benten.de (78.94.240.189): icmp_seq=2 ttl=50 time=76.9 ms
64 bytes from prod.stefan-benten.de (78.94.240.189): icmp_seq=3 ttl=50 time=83.2 ms
64 bytes from prod.stefan-benten.de (78.94.240.189): icmp_seq=4 ttl=50 time=57.1 ms

— satellite.stefan-benten.de ping statistics —
4 packets transmitted, 4 received, 0% packet loss, time 7ms
rtt min/avg/max/mdev = 57.066/70.985/83.151/9.936 ms

8

OK!
now try: docker exec -it storagenode ping satellite.stefan-benten.de the same but from docker container

9

again working fine :

docker exec -it storagenode ping -c4 satellite.stefan-benten.de
PING satellite.stefan-benten.de (78.94.240.189): 56 data bytes
64 bytes from 78.94.240.189: seq=0 ttl=49 time=93.025 ms
64 bytes from 78.94.240.189: seq=1 ttl=49 time=105.079 ms
64 bytes from 78.94.240.189: seq=2 ttl=49 time=85.705 ms
64 bytes from 78.94.240.189: seq=3 ttl=49 time=105.156 ms

— satellite.stefan-benten.de ping statistics —
4 packets transmitted, 4 packets received, 0% packet loss
round-trip min/avg/max = 85.705/97.241/105.156 ms

10

Unfortunately we can’t determine where is the issue, i think it ISP problem or DNS issue, but at this moment evryting working fine… I can reccoment setup simple free monitoring by you external ip and node port on uptime robot.

11

I’m already using Uptime robot.

Think I’ll just shutdown the server for a bit and do some maintenance.
Was going to move the node to a new server at the weekend, (end of the month) but might just so that tonight and see if that makes a difference.

1 Like
12

Please show your full docker run storagenode command with all parameters.

You may have left out specifying the port after the DDNS in the ADDRESS parameter. It should look like -e ADDRESS="storj.hopto.me:28967" \

1 Like
13

here’s my docker run command : (it has the port listed) nothing in the run command has changed in a while.

io [~/temp]$ cat storj-docker-info

docker run -d --restart unless-stopped -p 28967:28967
-p 14002:14002
-e WALLET=“0x8417aEbBBCDc034Ad5464cDB257aEDF5dA91e20b”
-e EMAIL="bill@s0l.co.uk"
-e ADDRESS=“storj.hopto.me:28967”
-e BANDWIDTH=“15TB”
-e STORAGE=“900GB”
–mount type=bind,source=“/home/bill/storj_ident/identity/storagenode”,destination=/app/identity
–mount type=bind,source=“/home/bill/temp/node”,destination=/app/config
–name storagenode storjlabs/storagenode:beta

14

It looks like you are using the wrong type of quotes in all the the environmental variable lines except -e EMAIL

Please refer to the example in https://documentation.storj.io/setup/cli/storage-node#running-the-storage-node in the tab for non-ARM and ARM based platforms to see which is the right type of quotes you should be using. Looks like the --mount lines use the correct quotes.

15

OK think i fixed it!

For some unknown reason the “kademlia.external-address:” in the config.yaml file was missing the port number.

I put the port number at the end and it’s now ONLINE.

I hope that’s fixed it!, I will keep an eye on it for the next few hours in case the errors start again.

as for the quote marks, it’s fine. It’s just the formatting of the forum that causes the different quote marks!

1 Like