QUIC Misconfigured after upgrade to v1.90.2

[LordWigo]

Hello community,

I have multiple nodes on the one SYNOLOGY NAS.

Few days ago one node started flashing as “Misconfigured”:

I have tried restart docker container - fixed for few hours but then received email that my node went offline.

Tried to delete and recreate - the same. Fixed but after few hours Misconfigured again.

In Logs I can see only these errors:

I did not change anything on network/config level. Other nodes runs in different containers from the same NAS and network (only different port of course).

Any ideas what could be wrong?

Thank you

Wigo

[daki82]

Whats the setup of the synology? drives?

[LordWigo]

Hi Daki,

There will be no issue with Synology itself as all the drives/nodes are the same but only this one is Misconfigured. When configuring it I am use this pattern:

sudo docker run -d --restart unless-stopped --stop-timeout 300 -p 28975:28967/tcp -p 28975:28967/udp -p 14009:14002 -e WALLET=“0xbxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx0” -e EMAIL="tomas.gajda@gmail.com" -e ADDRESS=“178.xxx.xxx.xxx:28975” -e STORAGE=“15TB” --mount type=bind,source=“/volume9/STORJ_08/Identity/storagenode”,destination=/app/identity --mount type=bind,source=“/volume9/STORJ_08”,destination=/app/config --name STORJ_08 storjlabs/storagenode:latest

[daki82]

maybe your router/ firewall is flooded because of the many nodes?
are they separated by vpn?

[LordWigo]

Hmm this wasn´t problem till now and other nodes are behind the same router.
I have tried now recreate this node and set other port which I have opened on router of course.
Now all green but let see within few hours.

[Alexey]

If you have several network interfaces (local IPs) for your setup, you may bind this container to the IP, which is provided in the port forwarding rule, i.e.

-p 192.168.10.24:28975:28967/tcp \
-p 192.168.10.24:28975:28967/udp \

Regarding untrusted satellites warning it’s because we removed US2 and EUN1 satellites from the trusted list:

Announcement: Storj to shut down europe-north-1 and us2

[1000009521] Farewell, europe-north-1 and us2. It’s been fun!

You may remove them:

How To Forget Untrusted Satellites

Following the announcement to decommission Europe-north-1 and US2 satellites, manually deleting the remaining data of these satellites from the node can be a tedious task as it requires finding the blobs folder name for each satellite, so we wanted to improve the storagenode CLI to semi-automate cleaning up a satellite data after it is marked as untrusted. The storagenode CLI (versions from 1.88.0) provides a forget-satellite subcommand to remove satellites from the trust cache and clean up the…

[LordWigo]

Hi Alexey,

Thanks for the reply.

Yes I have this on place for each node:

For now I have change it to “-p 28979:28967/tcp -p 28979:28967/udp -p 14009:14002”

Change it on router NAT Port forwarding and works 11 hours since change which is good sign.

Will monitor it.

Thank you

[LordWigo]

Now the node is working without QUIC issue. Not sure why just one port is issue as it was forwarded on Router the same way as other ports/nodes.
I will try latter return port to original one and will see.

Thanks for you support

Wigo

[Alexey]

Perhaps the port change restarted something on your router and QUIC started to work for the new port.

[snorkel]

Because you are on Syno, check my replys in “Please enable TCP-fastopen” thread. You should use network host mode and use unique ports for each node. Check also “My docker run commands for Synology”. I will link both threads later.

[snorkel]

https://forum.storj.io/t/my-docker-run-commands-for-multinodes-on-synology-nas/22034/7?u=snorkel

[LordWigo]

Hi I have multiple nodes. All other works properly. Each docker container - Node is configured on different port:

Tried more different ports for the next one (28975 right now) but still misconfigured.

My router has for each node specific port set as you can see here:

All these are works properly:

I have had issue in the past but all nodes on this SYNOLOGY was affected back then. It was some router setting issue. This looks like different issue.

I have set now:

Let see when reboot.

Thank you

Wigo

[snorkel]

In the docker run command, use the same ports for internal and external, not the default for all internals. Or use the network host mode with different ports; just mimique my commands that I reffered to in the other post, and all your problems will go away. I used the same default for all myself. I tried every combo. Dosen’t work on syno using the same internal port.

[LordWigo]

Hi Snorkel,

Not sure if I am understand it correctly.

This is how I am create nodes and was never issue till last weeks:

sudo docker run -d --restart unless-stopped --stop-timeout 300 -p 28975:28967/tcp -p 28975:28967/udp -p 14010:14002 -e WALLET=“0xbxxxxxxxxxxxxxxxxxxxxxxxxxxxxx00” -e EMAIL="xxxxxx.xxxxxx@gmail.com" -e ADDRESS=“xxx.xxx.xx.xxx:28975” -e STORAGE=“15TB” --mount type=bind,source=“/volume10/STORJ_09/Identity/storagenode”,destination=/app/identity --mount type=bind,source=“/volume10/STORJ_09”,destination=/app/config --name STORJ_09 storjlabs/storagenode:latest

When I changed it to -p 28975:28975/tcp -p 28975:28975/udp does not work at all:

What would be your format?

I am going to try your:

docker run -d --restart unless-stopped --stop-timeout 300
–network host
-e WALLET=“…WALLET…”
-e EMAIL=“…EMAIL…”
-e ADDRESS=“…WAN_IP…:28981”
-e STORAGE=“xxTB”
–mount type=bind,source=“/volume1/Storj/Identity/storagenode/”,destination=/app/identity
–mount type=bind,source=“/volume1/Storj/”,destination=/app/config
–name storagenode storjlabs/storagenode:latest
–server.address=“:28981”
–console.address=“:14011”
–server.private-address=“127.0.0.1:14012”
–log.level=fatal
–filestore.write-buffer-size 4MiB
–pieces.write-prealloc-size 4MiB
–storage2.piece-scan-on-startup=true

Thanks

[Alexey]

I believe it’s related to a new version 1.90.2 this time:

QUIC misconfiguration in v1.86.1

My nodes updated to 1.90.2 two hours ago and all of them are now quic misconfigured. I won’t be able to look into regressing this again until evening. @Alexey, does storj test on FreeBSD as part of qa? This is so reliably broken — how come it was allowed to make it to the public?

@snorkel

In the docker run command, use the same ports for internal and external, not the default for all internals.

This will work ONLY if you also change the internal port in the config.yaml or with a flag --server.address, otherwise it must be 28967, because it’s a default port.

@snorkel

Dosen’t work on syno

because your docker network is configured to be shared between containers. This is not default for Linux setups.

@LordWigo

What would be your format?

sudo docker run -d --restart unless-stopped --stop-timeout 300 \
-p 28975:28975/tcp \
-p 28975:28975/udp \
-p 14010:14002 \
-e WALLET="0xbxxxxxxxxxxxxxxxxxxxxxxxxxxxxx00" \
-e EMAIL="xxxxxx.xxxxxx@gmail.com" \
-e ADDRESS="xxx.xxx.xx.xxx:28975" \
-e STORAGE="15TB" \
--mount type=bind,source="/volume10/STORJ_09/Identity/storagenode",destination=/app/identity \
--mount type=bind,source="/volume10/STORJ_09",destination=/app/config \
--name STORJ_09 storjlabs/storagenode:latest \
--server.address=:28975

If you going to switch to the host network, then you do not need -p options (because NAT will be disabled), and need to specify all ports to be unique, include internal ones.

[LordWigo]

So do you recommend to wait until update new version release?
I just want to be sure that my online score will fall under critical level.

[Alexey]

Not working QUIC doesn’t affect the online score. You may try these two options above to see, can it help or not. I do not think that a potential issue with QUIC will be resolved soon, it’s a low priority and appearing only on freeBSD so far.

[LordWigo]

I can confirm that issue is related to latest version. Another node updated to latest version 1.90.2 and QUIC misconfigured:

[snorkel]

I checked my only 2 nodes machine and I have Quick missconfig on both. Both are on 1.90.2. The one node machines don’t have this issue. I use network host mode, all different ports for both nodes, and it’s the first time since Quick implementation when I have this error and can’t get rid of it.

[Alexey]

Please try to specify the IP of your network adapter and the port in the server.address option.
This is very interesting, what’s OS/distro?

Windows docker node:

It’s still uses the default bridge, and I did not provide a local IP, but at least on freeBSD it was helpful (but not now).