Node offline last night

Node Operators troubleshooting
, ,
1

macos and docker
I have restarted everything and still get this error
the port shows as open 173.22.61.45:28967 and of course it was working yesterday

2023-05-17T13:47:43.093Z ERROR contact:service ping satellite failed {“Process”: “storagenode”, “Satellite ID”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”, “attempts”: 12, “error”: “ping satellite: failed to ping storage node, your node indicated error code: 0, rpc: tcp connector failed: rpc: EOF”, “errorVerbose”: “ping satellite: failed to ping storage node, your node indicated error code: 0, rpc: tcp connector failed: rpc: EOF\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:149\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:102\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:99\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75”}

2

you definitely have problem with the node, because if you put your ip and port to brouser you shold get this
{
“Statuses”: null,
“Help”: “To access Storagenode services, please use DRPC protocol!”,
“AllHealthy”: false
}
but on your it not show it.
check may be some other service copied this port or router lagging

3

Also please check your WAN IP and compare it with IP on Open Port Check Tool - Test Port Forwarding on Your Router
If they do not match, the port forwarding will not work, because your ISP placed you behind their NAT (CG-NAT).

4

It went offline in the middle of the night so I suspected my isp. I checked the port and IP address and they are the same and the port is open. The error RPC: EOF seems to indicate that the systems are actually talking. I checked this by blocking 28967 at my machines firewall and the error changes to timeout as would be expected. I changed it back and the error is again EOF. I am baffled by this and have reloaded Docker storagenode and rebooted the router and machine. Is there something I could do on the VM running in Do
cker?

5

It could be possible that docker maintainers broke the virtual network again (this is why we recommend to use an older stable version 2.1.0.5).
What’s your current version of Docker Desktop?
Could you please try to reset it to the factory defaults (it’s in the Debug menu)?

6

I loaded a version of docker from a year ago and did a factory reset. same result.
QUIC currently doesn’t work and hasn’t for some time. Could that be an issue?

7

Not the year-old version, but exactly 2.1.0.5, if it’s a root cause and not some firewall or port forward misconfiguration.

If your node is online, then you may try to fix a QUIC issue, but if it’s offline, then you need to fix this first.

I would like to suggest to check everything from this checklist:

Especially that your ADDRESS option contains either updated public IP (you may see it on yougetsignal) or updated DDNS hostname.
if you use a DDNS hostname, make sure that it resolves to the same public IP as on yougetsignal and it’s match the WAN IP on your router:

nslookup your.ddns.hostname 8.8.8.8

should return the same IP as on yougetsignal and matches your WAN IP.

8

Back from short vacation
I installed 2.1.0.5 with the same result. Checked everything from the checklist and permissions, a docker process listening on 28967 etc. So I finally fired up wireshark. It shows a dRPC request on port 28967
and an instant RST (reset) sent back from my machine. Apperently it doesn’t want to talk for an unknown reason hence the rpc EOF error message.
Any more ideas on how to fix this.

9

This could be result of the working firewall. Make sure that no one firewall rule has a source port or IP and either all outgoing rules are disabled or you have a rule to allow any traffic to any host and port from any port of your Mac.

Does the node has other errors in the logs, except “ping satellite failed”, i.e. related to the filesystem or databases?

10

No other errors. Other things seem to work. orders sent, trash emptied
expired pieces deleted.
no outgoing rules. same result with firewall off.

11

Please try to change the external port to a different one. Do not forget to forward it as well.

And I can assume that you already tried to remove the docker desktop and install the latest one (perhaps after a reboot?).

12

I changed the port and it started to connect.
I will ask mediacom what they did to my forwarding rule as I can’t control them and they no longer will do it over the phone, at least that was what one of their people told me, however that is likely a futile waste of time.
Thanks for the help.
I still can’t figure out why the 28967 port allowed a connection that was then dropped.

13

Something is blocking this port. It could be your ISP.

so, they control forwarding rules on your (or their?) router? Then it could be possible, that the local IP of your Mac has changed, but the rule still points to the old one.
It’s better to make your local IP static to do not repeat the problem in the future when your reboot your Mac.
Or they simple made a mistake in the port forwarding rule.

14

Some ISPs can control the forward rules on clients routers, and the client can’t change anything. This can be easily avoided by asking the ISP to put their router in media bridge mode, and use your own router, bought by you, as the main router. You connect the port 1 from the bridge (ISP’s) router to the WAN port on your router, set the parameters needed for connection and you have full control over settings.
A friend told me that the ISP reseted the forward rules over night on his router and he lost the connection to whatever he was using, without telling him. So take as much control as you can from ISP into your hands. Don’t trust them with settings you can do aswell.
Also many ISPs offer different kings of traffic filters, by default (av, firewalls, etc), for free or with some fees. Just ask them to disable all filters and protections. You can use your own protections, with rules set by you.

1 Like