I have checked with tcpdump that both udp and tcp packets can reach the node.
there are a total of 7 nodes, running on their own disk,
works for all but one of them.
the docker config contains the command for both tcp and udp being forwarded to the container
-p xxxxxx:28967/tcp
-p xxxxxx:28967/udp \
Any idea? They have been running for about 3 years or so.
TLDR: ignore it.
I get these errors, the node then crash and restarts and works for a few minutes.
2023-10-05T20:42:07Z ERROR contact:service ping satellite failed {“process”: “storagenode”, “Satellite ID”: “12tRQrMTWUWwzwGh18i7Fqs67kmdhH9t6aToeiwbo5mfS2rUmo”, “attempts”: 2, “error”: “ping satellite: check-in ratelimit: node rate limited by id”, “errorVerbose”: “ping satellite: check-in ratelimit: node rate limited by id\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:203\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:157\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:99\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75”}
2023-10-05T20:42:07Z ERROR contact:service ping satellite failed {“process”: “storagenode”, “Satellite ID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”, “attempts”: 2, “error”: “ping satellite: check-in ratelimit: node rate limited by id”, “errorVerbose”: “ping satellite: check-in ratelimit: node rate limited by id\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:203\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:157\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:99\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75”}
2023-10-05T20:42:09Z ERROR contact:service ping satellite failed {“process”: “storagenode”, “Satellite ID”: “12tRQrMTWUWwzwGh18i7Fqs67kmdhH9t6aToeiwbo5mfS2rUmo”, “attempts”: 3, “error”: “ping satellite: check-in ratelimit: node rate limited by id”, “errorVerbose”: “ping satellite: check-in ratelimit: node rate limited by id\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:203\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:157\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:99\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75”}
2023-10-05T20:42:10Z ERROR contact:service ping satellite failed {“process”: “storagenode”, “Satellite ID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”, “attempts”: 3, “error”: “ping satellite: check-in ratelimit: node rate limited by id”, “errorVerbose”: “ping satellite: check-in ratelimit: node rate limited by id\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:203\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:157\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:99\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75”}
This is different issue. Your node repeatedly asks satellite to connect to it, but it can’t, so it rate-limits your attempts. You need to check that you node is reachable externally by the address and port that is specified as the external address in the config file. Literally, take the phone, turn off WiFi and try to access your node at storj port by HTTP. You shall see this:
{
"Statuses": null,
"Help": "To access Storagenode services, please use DRPC protocol!",
"AllHealthy": true
}
If you don’t see this – fix connectivity. Check your ports, forwarding, firewall, LAN IPs, etc, etc.
I mentioned that in the very first sentence in my first post about tcpdump
Maybe I need to be more clear,
with tcpdump I can see that externally I can see traffic from all 7 ports, both udp and tcp.
For 6 of the nodes, there are no complaints about Quic and the run well.
for the 7th node, it complains about Quic and then this message happens after a while.
No, you have been perfectly clear. So were I.
This is not what I suggested to check. You can have plenty of traffic and unaccessible node if that traffic is misrouted.
If you just fixed connectivity – you need to stop the node so it stops bombarding the satellite with connection requests, and then turn it back on when backoff interval expires. I don’t know what is current backoff interval, try waiting for one hour.
This is a consequence of a previous connection failure errors, like @arrogantrabbit said
And also that’s mean that your node is OFFLINE, not only misconfigured (which is much smaller issue in case of being offline).
You should check the connectivity either as suggested:
or using Open Port Check Tool - Test Port Forwarding on Your Router for your address and port from the configuration.
I guess you uses DDNS and it’s not updated, maybe subscription is expired or you did not configure its update in your router in the DDNS section, or your ISP placed you behind CGNAT, or local IP of your host has been changed, but the port forwarding rule still uses an old one, etc.
when trying to reach it, I get as arrogantrabbit said:
{
“Statuses”: null,
“Help”: “To access Storagenode services, please use DRPC protocol!”,
“AllHealthy”: true
}
There are 7 nodes in total, all on different disks. works for the other 6.
If portforwarding is not working
The DNS is working, and on top of that the IP hasn’t changed in years.
The Open Port Check Tool also says the port is open.
doing netcat -u dnsname port from a host outside my network,
and listening on tcpdump -i interface udp and port XXXXXX I can see the requests.
ss -tulpn shows this for the port
udp UNCONN 0 0 YYYYY:XXXX 0.0.0.0:* users:((“docker-proxy”,pid=3790208,fd=4))
tcp LISTEN 0 4096 YYYYY:XXXX 0.0.0.0:* users:((“docker-proxy”,pid=3790194,fd=4))
Perhaps sometimes your router did not forward requests. Or it has something like DDoS protection.
it is a ubnt dream machine pro, there is no security features enabled that would block anything.
and would be strange if it blocks only for 1 of the 7 nodes.
are there any other way to determine why it is giving up?
You need to search for “ping satellite failed” exclude “rate”, it should show the root cause.
docker logs storagenode 2>&1 | grep "ping satellite failed" | grep -v "rate" | tail
it appears to be working now, no idea what changed to be honest
Maybe a temporary problem with your router or ISP?