"ping satellite: failed to ping storage node" despite port is open

Node Operators troubleshooting
1

Hello,
I’ve an issue with my node since yesterday. I received an email saying that my node went offline, checking the dashboard it is indeed.

I check if the port in tcp/udp is open and it seems to be the case, I tryed to access the node through my browser using the external ip address and I get this response :

{
“Statuses”: null,
“Help”: “To access Storagenode services, please use DRPC protocol!”,
“AllHealthy”: true
}

In the logs of of the node I get the following error messages :

2023-11-20T06:52:00Z	ERROR	contact:service	ping satellite failed 	{"process": "storagenode", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "attempts": 11, "error": "ping satellite: failed to ping storage node, your node indicated error code: 0, rpc: tcp connector failed: rpc: read tcp 10.55.4.65:48002->151.80.67.151:28301: read: connection reset by peer", "errorVerbose": "ping satellite: failed to ping storage node, your node indicated error code: 0, rpc: tcp connector failed: rpc: read tcp 10.55.4.65:48002->151.80.67.151:28301: read: connection reset by peer\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:209\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:157\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:99\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75"}

I use a vps as gateway to expose my node, any idea of what is wrong and how to troubleshoot this?

2

I can access your node:

% curl 151.80.67.151:28301
{
  "Statuses": null,
  "Help": "To access Storagenode services, please use DRPC protocol!",
  "AllHealthy": true
}%

So, a possibility is some firewall/antiddos/software on the ovhcloud, or (depending on how you configured VPN) your router or a host node is running on is messing with connections. Turn off all anti-ddos and exclude traffic to and from node from various traffic analysis tools you may have running.

3

Not working from EU:

curl -v 151.80.67.151:28301
*   Trying 151.80.67.151:28301...
4

End of trace route:

image
image2555×306 23.6 KB

Looks to be some networking issue…

5

Thanks guys, indeed, I think I found something : the vps lost connection to internet impossible to ping google from the VPS (but I can access to the vps), that’s why my node does not manage to reach the satellites despite it is accessible.

I’ll try to fix this first and see if this solve my issue.

6

sometimes its just one satelite having “dns problems”,

i don’t remember if they go away themselve or after a router reboot.

no vps, just dyndns.

7

To part of the internet; I can still reach your node. So it’s either some routing issues with your provider or software firewalls there blocking access.

It’s the other way around — node can reach satellite, but satellite cannot talk to node.

The ip seems to be resolved correctly, connection is established and then immediately dropped.

1 Like
8

The always dumb question here: did you restart the whole operating system op the node? And if possible also the VPS, which you are using as a gateway (using VPN or something?).

Sometimes this also happens, when your filesystem is choking. Such like using a SMR-drive, taking so much CPU and RAM that it’s impoosible to incoming traffic in a timely fashion. Might this be the issue? Might also explain the fact sometimes that you sometimes seem to be reachable and other times you don’t seem to.

9

Thanks for the advice, yes I restarted the node host and the VPS without success.
Doing a traceroute on the VPS seems to shows that the request get lost somewhere in OVH infra without reaching google

traceroute 8.8.8.8
traceroute to 8.8.8.8 (8.8.8.8), 30 hops max, 60 byte packets
 1  _gateway (141.94.220.1)  0.170 ms  0.113 ms  0.082 ms
 2  192.168.143.254 (192.168.143.254)  0.102 ms  0.069 ms  0.070 ms
 3  10.225.31.188 (10.225.31.188)  0.162 ms 10.225.31.189 (10.225.31.189)  0.151 ms 10.225.31.188 (10.225.31.188)  0.126 ms
 4  10.225.29.94 (10.225.29.94)  0.395 ms  0.475 ms 10.225.29.104 (10.225.29.104)  0.436 ms
 5  10.225.22.252 (10.225.22.252)  0.187 ms 10.225.22.240 (10.225.22.240)  0.202 ms  0.175 ms
 6  10.17.155.56 (10.17.155.56)  0.569 ms 10.17.146.2 (10.17.146.2)  0.474 ms 10.17.151.118 (10.17.151.118)  0.315 ms
 7  10.73.1.32 (10.73.1.32)  0.147 ms 10.73.0.226 (10.73.0.226)  0.210 ms 10.73.0.76 (10.73.0.76)  0.167 ms
 8  * * *
 9  * * *
10  * * *
11  * * *
12  * * *
13  * * *
14  * * *
15  * * *
16  * * *
17  * * *
18  * * *
19  * * *
20  * * *
21  * * *
22  * * *
23  * * *
24  * * *
25  * * *
26  * * *
27  * * *
28  * * *
29  * * *
30  * * *

I’ve opened a ticket on OVH support this morning but still no reply from them, for now it’s ok I’ve few days left before being suspended…