Online status decreasing after domain change

Node Operators troubleshooting
,
1

Hello everybody,
I am running a storagenode since more than 15 months now without problem, but the last days I am more and more worried, as the onlinestatus for saltlake.targigrade.io is decreasing day by day.

I am running the storagenode with docker setup on a Ubuntu server at my homeplace with a not static ip. So I am using a domain with a dyndns service. This worked without problems since 15 months. Now I had to change the domain. Therefor I changed the address in my docker run startscript, stopped the storagenode, recreated it and started it again.

The dashboard shows me, it is online. I am also using UptimeRobot to track, if the storagenode is reachable from extern. UptimeRobot shows me, 100.000% (last 24 hours) , 99.992% (last 7 days), 99.992% (last 30 days).

So far, I don’t find the problem, why the online status for saltlake is dropping.

Thanks in advance for any help.

2022-06-11 08.42.13 192.168.0.5 f50b36a17483
2022-06-11 08.42.13 192.168.0.5 f50b36a17483859Ă—324 26.4 KB

2

This satellite is unable to contact your node for some reason.
Please search for “ping satellite failed” for that satellite and copy the last 3-5 such errors from your logs.

docker logs storagenode 2>&1 | grep "ping satellite failed" | grep 1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE | tail
3

Here is the output:

2022-06-11T13:24:50.806Z	ERROR	contact:service	ping satellite failed 	{"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "attempts": 7, "error": "ping satellite: check-in network: failed to resolve IP from address: obont.de:28967, err: lookup obont.de on 10.102.0.10:53: server misbehaving", "errorVerbose": "ping satellite: check-in network: failed to resolve IP from address: obont.de:28967, err: lookup obont.de on 10.102.0.10:53: server misbehaving\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:136\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:98\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:152\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
2022-06-11T13:26:00.357Z	ERROR	contact:service	ping satellite failed 	{"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "attempts": 8, "error": "ping satellite: check-in ratelimit: node rate limited by id", "errorVerbose": "ping satellite: check-in ratelimit: node rate limited by id\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:136\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:98\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:152\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
2022-06-11T13:28:13.897Z	ERROR	contact:service	ping satellite failed 	{"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "attempts": 9, "error": "ping satellite: check-in ratelimit: node rate limited by id", "errorVerbose": "ping satellite: check-in ratelimit: node rate limited by id\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:136\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:98\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:152\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
2022-06-11T13:32:35.446Z	ERROR	contact:service	ping satellite failed 	{"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "attempts": 10, "error": "ping satellite: check-in ratelimit: node rate limited by id", "errorVerbose": "ping satellite: check-in ratelimit: node rate limited by id\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:136\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:98\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:152\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
2022-06-11T13:41:22.985Z	ERROR	contact:service	ping satellite failed 	{"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "attempts": 11, "error": "ping satellite: check-in network: failed to resolve IP from address: obont.de:28967, err: lookup obont.de on 10.102.0.10:53: read udp 10.100.2.26:37095->10.102.0.10:53: i/o timeout", "errorVerbose": "ping satellite: check-in network: failed to resolve IP from address: obont.de:28967, err: lookup obont.de on 10.102.0.10:53: read udp 10.100.2.26:37095->10.102.0.10:53: i/o timeout\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:136\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:98\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:152\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
4

Any advice for my problem, @Alexey ?

5

Dein Container hat ein Container hat ein DNS problem. Wenn ich obont.de auflöse kommt 79.214.207.102 raus.

6

Ja, aber das ist doch auch korrekt die IP.

7

dein Container löst die IP gar nicht auf.
→ ```
read udp 10.100.2.26:37095->10.102.0.10:53: i/o timeout"

8

Mmh, aber das hat doch die letzten 15 Monate geklappt. Wieso jetzt nicht mehr?

Vor allem, wie lässt sich das beheben?

9

Update from my side:

I changed back to the old domain temporarily, the problem is solved and I don’t see any errors in the logs.
Beside that, I also registered for a ddnss account and also tried this hostname. Also here everything running smoothly and no problems in the logs.

So I assume that there is no misconfiguration in my docker container or my local network.

What could be the problem with the new domain?

10

This domain sometimes can’t be resolved. Maybe your DNS provider have some throttling for number of requests for this domain.
I asked the team, maybe they could check what could be a problem with this domain.

11

Hello @Alexey , I transfered the new domain now to another provider.

Now everything works fine, even with the new domain.

Thanks for help!

1 Like