Network error codes - node off line...?

Storage Node Operators troubleshooting
#1

Anyone help me trouble shoot this error… my node has been working fine for a few months and went off line last night?

g.org/x/sync/errgroup.(*Group).Go.func1:57"}

2020-12-10T08:28:18.763Z ERROR contact:service ping satellite failed {“Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “attempts”: 9, “error”: “ping satellite error: failed to dial storage node (ID: 1EgLd7jU1Z6iLc52hWSEwc2w84tGDWxg5NfzoHcxyDdp6f8nKo) at address rasberryrtk.ddns.net:28967: rpc: dial tcp 86.170.117.126:28967: i/o timeout”, “errorVerbose”: “ping satellite error: failed to dial storage node (ID: 1EgLd7jU1Z6iLc52hWSEwc2w84tGDWxg5NfzoHcxyDdp6f8nKo) at address rasberryrtk.ddns.net:28967: rpc: dial tcp 86.170.117.126:28967: i/o timeout\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:141\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:95\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:92\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}

#2

Can anyone help with this?

#3

you should start by verifying that your ddns has the correct ip address registered…
the error just seems to say that the node is offline… so maybe port forwarding in the router that was lost… and i would ofc try to turn stuff off and back on again :smiley:

most likely internet connection, firewall…
when starting to troubleshoot stuff like this i usually start by looking at the easiest to solve issues first and ask myself … what changed… it’s rare that stuff just stops working for no reason…

#4

thats my problem… nothing has changed…
was working when I went to bed… got up… Offline… doesn’t figure?

#5

stuff is always changing… could be something as simple as your storagenode gets it’s LAN IP address from a DHCP server in your router and then somebody else on the network assigned a static IP address to something and the DHCP server moved the storagenode to a new IP address to keep the network working.

check your internet, connection and port routing to the storagenode.
your node may have updated from 1.17.4 to 1.18.1 during the night… maybe watchtower isn’t configured correctly … if you are using docker…

the guide can be found here…
https://documentation.storj.io/dependencies/port-forwarding

#6

everything looks correct and all points to the right place.
Docker updated to 1.18.1 a few days ago…

#7

well if the system didn’t change nor update, and the node claims it’s offline…
then it has to be the route to the internet / internet connection.

best bet i think is the ddns… if you ping your ddns address… which seems to be

rasberryrtk.ddns.net

then you should get an ipaddress…

that ipaddress should be the same ipaddress as what you get if you go to a whats my ip website to check you ip address.

ddns is the only external factor that would make sense from what i can guess

#8

ok do the DDNS address is timing out…
thats odd.
its through My No Ip

#9

So how do I update to try a new DNS which seems to be working?

#10

as a temporary patch you can use the current ip address in the docker run command, which will get you up and running until you solve the ddns issue.

i haven’t really setup or done much with the ddns services so ain’t really much help in that regard… you should have a program or an account at a website which would control your ddns address… tho it can also be controlled by your router… the router as controlling the ddns is the recommended solution…

most likely if you access your ddns account you can most likely see whats wrong.
some ddns services needs one to renew the address and such… i duno

atleast now you know what to hit to fix it :smiley:
good luck

ddns services are plentiful… find one you like make an account, download their software or setup your router to update them with your dynamic online ip address

should be clearly explained at whatever ddns you choose… i would look at the forum for some recommendation, there may be a number of bad seeds or annoying ddns providers you might want to avoid…

ofc you might also easily be able to fix the current one… when you figure out what went wrong with it.

#11

Can i just edit a config file to update with a new DNS ? or do i have to remove the conainter from Docker and create a new one?

#12

you change it in your run command…
maybe in watchtower… i do manual updates… so not even sure how watchtower works.

https://documentation.storj.io/setup/cli/storage-node

for docker to start the storagenode a command structure like:

docker run … insanely long extensive command is run which has all your ip address, port mappings and storage folder mounts and such

its described on the link.
you should have it stored somewhere… either in watchtower or for ease of copy paste … remember to do a
docker stop storagenode
docker rm storagenode
else the docker run … command won’t work

#13

its sorted… just renamed the broken config in Docker and then created a new container with new DNS details. Back online now… many thanks for the help !!

1 Like
#14

happy to help
remember the ddns service needs to be updated with your ip address, else when your ipaddress changes or whatever it will stop working because it has an old address.

this is either done by the internet router which is a good idea, because it helps reduce latency when the ip is changed…

else a piece of software will need to run on the storagenode host which keeps a track of your online ip and updates the ddns service with the new ip address when it changes…

this must be setup correctly else this will be a very temporary fix…

#15

The watchtower do not have a configuration for the node, it uses the command from controlled containers.

1 Like
#16

why i clearly stated i haven’t used it.

but the command isn’t anywhere… normally you just write it, so it will just reuse the last used run command for active containers?

i suppose that would make good sense because then it would be easy to start, stop and upgrade even if jumping around from watchtower to manual or vice versa

#17

No, it will use a command from the running container. It doesn’t remember anything, it’s not designed for that.
FYI - try to run a

docker inspect storagenode

and you will have an idea, how is it doing so :slight_smile:

2 Likes