QUIC Ok then change to Misconfigured after a few minutes

Node Operators troubleshooting
,
1

Hi,

Got an issue with my new node I cannot figure it out :

Running v1.139.6 on docker /linux

I’m using a fortigate as a firewall with two rules : one for TCP, another for UDP/QUIC

As soon as I start or restart my node, everything is fine : Online OK, QUIC OK

I can see some traffic for both TCP and UDP rules

then after a few minutes my nodes is changing to : Online OK, QUIC MISCONFIGURED and no traffic anymore through the UDP rule.

In the log got :

WARN contact:service Your node is still considered to be online but encountered an error. {“Process”: “storagenode”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Error”: “contact: failed to ping storage node using QUIC, your node indicated error code: 0, rpc: quic: timeout: no recent network activity”}

Only traffic through the UDP/QUICK rules are from :

35.207.107.183

34.2.138.64

35.215.106.201

35.212.20.216

There is no drop from the foritgate log

Thx for your help

2

You can try to use this:

QUIC is checked on each check-in on the satellites, it’s every hour by default.

3

Thx a lot for the tip

I changed server.address to the container ip and restarted. So far it is running for 20 minutes, the node is up, QUIC is green and no error into the logs .. cross fingers

4

ah crap, it last 1h30 and back to misconfigured

5

Then you may ignore it, it’s rarely used.

6

Just increase the buffers to 7.5MB, the official QUIC recommendation.

7

I also have two nodes on Windows with Docker Desktop that have switched to Quick Misconfigured. As far as I understand, you can’t increase the buffer on Windows with Docker Desktop.

8

thx for the tip ! I just released my sysctl.conf has been renamed sysctl.conf.distUpgrade recently and so lost the config….

1 Like
9

ok the pb is back again , I have checked :

sysctl net.core.rmem_max → net.core.rmem_max = 16777216 > 7500000

sysctl net.ipv4.tcp_fastopen → net.ipv4.tcp_fastopen = 3

from config.yaml :

server.address: 172.17.0.2:28967

docker inspect storagenode | grep -i “ipaddress”
“SecondaryIPAddresses”: null,
“IPAddress”: “172.17.0.2”,
“IPAddress”: “172.17.0.2”,

On the firewall side I can see :

TCP rule : 8Gb so far today

UDP rule : only 600k from the satellites just after the restart. All accepted state. Then no connection attempt anymore and :

contact:service Your node is still considered to be online but encountered an error. {“Process”: “storagenode”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Error”: “contact: failed to ping storage node using QUIC, your node indicated error code: 0, rpc: quic: timeout: no recent network activity”}

I can telnet to port 777 to the satellite from the node too

I can telnet from outside of the LAN using the node dns name to port 28967

I used : Pingdom

got :

- started
-
- from: St.Petersburg, Russia
- TCP: dialed node in 521ms
- TCP: pinged node in 244ms
- TCP: total: 765ms
- QUIC: dialed node in 524ms
- QUIC: pinged node in 244ms
- QUIC: total: 768ms
-
- from: OVH, France
- TCP: dialed node in 572ms
- TCP: pinged node in 250ms
- TCP: total: 822ms
- QUIC: dialed node in 584ms
- QUIC: pinged node in 258ms
- QUIC: total: 841ms
-
- done.
- started
-
- from: St.Petersburg, Russia
- TCP: dialed node in 469ms
- QUIC: dialed node in 475ms
-
- from: OVH, France
- QUIC: dialed node in 545ms
- TCP: dialed node in 553ms
-
- done.M

Everything looks fine but still it is displayed as MISCONFIGURED for QUIC

no idea anymore

10

Ok suddenly after 3h running it turns GREEN

1 Like
11

It re-checks it on every check-in (every hour by default).

12

You may try, but you need to login to the docker desktop VM. You can do so using a Terminal, it shows the docker desktop VM (wsl2) as well.

13

Ok it failed again. very unstable,

14

It depends on a network configuration. Thus the usage of it so low.

15

is there any setting to force a QUIC refresh much frequently than 1 hour ? thx

16

I increased the buffer to 7500000 in Windows Docker Desktop

PS C:\WINDOWS\system32> wsl -d docker-desktop
docker-desktop:/tmp/docker-desktop-root/mnt/host/c/WINDOWS/system32# cat /proc/sys/net/core/rmem_max
t /proc/sys/net/core/wmem_max
7500000
docker-desktop:/tmp/docker-desktop-root/mnt/host/c/WINDOWS/system32# cat /proc/sys/net/core/wmem_max
7500000

  • started
  • from: St.Petersburg, Russia
  • TCP: dialed node in 152ms
  • TCP: pinged node in 63ms
  • TCP: total: 215ms
  • QUIC: dialed node in 162ms
  • QUIC: pinged node in 70ms
  • QUIC: total: 231ms
  • from: OVH, France
  • QUIC: dialed node in 113ms
  • QUIC: pinged node in 36ms
  • QUIC: total: 149ms
  • TCP: dialed node in 119ms
  • TCP: pinged node in 37ms
  • TCP: total: 157ms
  • done.

but the quick stay misconfigured

17

Ok so I debug further :

  • I moved the docker container to host network to remove one layer of complexity
  • I changed the contact.interval to: 0h10m0s : I can see the satellites contacting the node through the UDP firewall rule every 10 minutes. No issue on that side. However the QUIC status is constantly changing btw OK to Misconfigured.
  • Still : error code: 0, rpc: quic: timeout: no recent network activity happening like 50% of time
18

The satellite may start to rate limit your node, please consider to remove this option and make it default, since it didn’t help anyway.

Then it’s losing somewhere else, like on your router or ISP. UDP is designed to be not reliable, unlike TCP.

19

Hi,

I saw the throttling into the logs and changed it back to 1hour. I see no error or warning on my Fortigate router and no error from other services hosted on the same address.

Does the QUIC issue is impacting the node score ?

20

No, the reputation (audit, suspension and online scores) is not affected. Or what do you mean by score?
Success rate? Unlikely, see