QUIC stop working after SYNOLOGY DSM update

Hi Peem,

I am running latest DSM version:

Seems that I have latest version available for Synology:

Wigo

I was referring to this screenshot:

Depending on how synology implemented docker networking – classic bridge or macvlan the MAC address as seen by the gateway may have changed.

Oh, I missed that its’ on vm on synology. Unfortunately, this is probably still irrelevant – network stack for VM and docker is different. VM uses proper virtualization via QEMU and docker – depends on what sinology did there.

On the unrelated note, I remember having issues with either of them, which got resolved when I destroyed and created a new virtual switch. (This was on DSM6)

I would check that the port is actually open from your LAN – to rule out firewall/router/forwarding. you can use nmap or nc to probe the port from another machine.

BTW, on synology DSM a lot of useful tools (such as aforementioned lsof) are hidden by default. Do get access to them run synogear install. This will make them visible until you reboot the DSM.

So start with that – check if you can probe the port from one of the local machines. IF yes – the problem is with the routing. if not – synology networking shenanigans. In which case I would confirm that node is actually listening on the expected port (as described above with lsof) and if so – destroy and re-create the bridge.

Edit:

Also, maybe downgrade the docker if you can, or the whole DSM? (I ended up not updating DSM ever – they break way more things than they fix. Last reasonably unbroken version was 6.2.3 update 1. In the update 2 they broke samba… SMH). With Synology once you found the working version – stay on it and not give in to updater.

I mean Virtual DSM - as a virtual machine in VMM
e.g. Centrum pobierania - download | Synology Inc.
or
Centrum pobierania - download | Synology Inc.
not the latest

Hi arrogantrabbit,

Router has been changed and no difference:

image

I will check option bridge vs macvlan. Fact is that VM Win10 is connected physically via own LAN port to switch but rest of the route is the same residing behind the same router.

Ani-ddos has been disabled (no change after node reboot - will try whole NAS reboot latter)

I will try port check if my knowledge allow me that :slight_smile:

Unfortunately downgrade is not possible in this stage (I was checking that option). I will have to approach to updates with more caution.

Thank you very much for you effort

Wigo

So what do you suggest is try run VirtualDSM under physical SYNOLOGY or Windows10 machine?

Never play with virtualDSM before :blush:

Wigo

VirtualDSM under physical SYNOLOGY and on that Docker and some little test node

I would like to add that I tried PING tool on https://storjnet.info/ and get results for one of my Nodes which has QUIC Misconfiguration issue:

First PING:

And second is DIAL:

image

But still Misconfigured :frowning:

Not sure if this is somehow relevant but trying anything.

@LordWigo
Why do you change settings for network in Docker? I run 2 nodes in docker on latest DSM and latest Docker on the same machine. Never had a problem with ports and quick etc.
I just forward ports in router to the DS IP, than I set a fix IP in general Network settings of DSM, not in Docker. Firewall off, DDOS protection off in router and DSM, and that’s it. All works great. I have 7 DS220+ and DS216+ running nodes. I manualy keep them up to date, DSM and Docker.

1 Like

Hi snorkel,

What do you mean changed network settings in Docker?

ALL nodes on SYNOLOGY has the same IP of NAS, All network protections are turned off (like firewall etc).
Tried put NAS to DSM for a while and did not help. Issue is not connectivity but QUIC only. Same Windows machine on the same LAN behind the same router works OK (QUIC). VM on the same NAS with Storage Node installed works fine (both Win10). Only Docker SN had this issue.

Ports are properly configured (if not whole node(s) would be offline wasn´t it?).
Everything was shown in this thread. Only one think which was not done is downgrade DSM version because not possible. Reset NAS is also not possible for me.

But any ideas are welcome of course :slight_smile:

Thank you

Wigo

I think I missread something at a quick look. I will try reading the entire topic tonight, maybe something comes to mind. But if the experts didn’t come up with a solution, probably I won’t eider :slight_smile:

Hi LordWigo,

Just a long shot here. Have you tried to erase cookies, browsing history and whatnot from your browser?
I’m running (walking, right now!) nodes in synology and found that my QUIC problems were a browser illusion…

Hi humbfig,

I know that for lot of users is this only web browser “bug” which can be removed simply by clearing cache or browser restart but I have clearly this issue in logs from my Docker session as you can see in this thread upper :wink:

Wigo

Ok. No more browser suggestions. Sorry I didn’t read all your previous posts. I’m trying to catch up now.
It seems like you’re running 7 nodes in your syno, and I definitely want to talk to you about that later. But for now let’s focus on solving your problem.
Based on this image that you posted:

I’m a bit confused on what you’re doing on the port forwarding front.
I do this on my docker run command, modifying the ports:


EDIT: Obviously, in my docker run command I also specify:
-e ADDRESS=“my_IP:25678”
Otherwise the satellites wouldn’t know how to contact me.

Then I forward the modified ports in my router, without changing the port numbers, meaning 25678 will be forward to 25678. The router does not “translate ports”. Docker will “translate ports”.

Therefore I have something like this in my docker:

I find it odd how you come up with the same “source:destination” port numbers on your docker…
Maybe that’s just for your node #1, which would be alright…

I mean, I don’t touch the Storj configuration ports. It doesn’t matter how many nodes I’m running, they will all be listening on 14002 and 28967. Are you doing the same thing?

Hi humbfig,

Yes on screens and example is STORJ_1. Here are all SYNOLOGY nodes:

image

28968:28967 is dedicated to my Windows machine where all is running OK (behind the same router).

This is properly configured, guys from STORJ support confirmed this.
If there would be network problem (in configuration) probably my node/s would be OFFLINE and not Online/Misconfigured hmmm?

As you can see all nodes listen on 14002 and 28967.

Wigo

@LordWigo
OK. Now I read it all. I will point out a few things and I will tell you what is my config on multinode machine, that works fine, and never had QUICK errors, on any storagenode version, including 1.66.1, 1.67.1, 1.67.3. Please excuse me if I repeat some advises or you already did this, but I must share all I know, not skipping something important.

  1. Why are you setting “Bandwidth: 100TB”? I don’t have that, and maybe is better to be unlimited…

  2. Access DSM from a PC with putty, on SSH port, enter you credentials, then:

sudo -i

enter password

sudo netstat -tulpn

You should see all the ports opened; the ones used by the storagenodes should correspond to Docker on the last column PID… for IPV4 TCP and IPV4 UDP. They should not be used by something else.
The following commands don’t work on DSM:

sudo lsof -i -P -n
sudo ss -tulpn
sudo ss -tulwn
  1. I see on reply #12 that you typed wrong the ports in the router: 28979 instead of 28969. You changed the router, so maybe now they are typed correctly.
  2. Do you use 2 LAN ports on your NAS(DiskStation)? Where are they connected? To the same router/LAN or to 2 routers on 2 LANs, on 2 WANs? Check in Network settings, in DSM, what is your default Getway.
  3. In router, set the rules for the NAS’s IP, not Docker bridge IP. Never touch Docker’s network/bridge settings. Ignore them.
  4. You should use different internal and external ports for both TCP and UDP for every node. Using the same internal port for all nodes, dosen’t work on DSM.
  5. Install storagenode and run docker commands from a PC, with putty > SSH, in sudo su mode.
    I never used the Docker’s terminal, maybe it’s the same thing, I don’t know. I’m not a linux user eider.
  6. You should stop all nodes, rm all nodes, update everything DSM, Docker, all packages, storagenode, router, restart everything, and than docker run the nodes.
  7. Never use 2 routers, one after the other, creating double NAT. Put the first one that connects to ISP in bridge mode (call the ISP to do that), and second one use it as a normal router, and connect it to the first one like this: ISP > WAN port on bridge mode router > from port LAN 1 to WAN port on router 2 > from any port on router 2 (LAN1 or LAN2, maybe even 3 or 4) to LAN1 on NAS.
  8. DDOS protection off in router, DDOS protection and firewall off in DSM. The router should be the only NAT provider in your network.

You should have something like this on the LAN1 where your router is connected for the storagenodes:



Forward ports in router to 123.123.123.2, in my exemple.
Now, how do I setup my nodes, on Synology DS220+, with 2x16TB HDD.
I never open Docker, just install it and that’s it. I open it only if I want to check the logs. I always use putty > SSH to access DSM for storagenode management.
HDD and DSM settings:

RAID type: Basic
Filesystem: ext4
Record file access time: Never
Low capacity notification: 5%
Data Scrubbing schedule: Enable only for RAID with 2 or more disks
RAID Resync speed limits: lower impact on system performance
Fast Repair: Enable
Enable write cache: Yes on UPS.
Bad sector warning: Enable.
HDD hibernation off
DSM on manual update.
Packages: autoupdate to all versions, except Docker.

For a 16TB HDD, I allocate 14TB for storagenode. I want to point out that I only use these Synology NASes for Storj, so I don’t use the space for anything else.
SSH to DSM with putty on the administrator account (the user defined as admin on DSM installation).

sudo su
echo "net.core.rmem_max=2500000" >> /etc/sysctl.conf
sysctl -w net.core.rmem_max=2500000
docker pull storjlabs/storagenode:latest
docker run --rm -e SETUP="true" \
	--mount type=bind,source="/volume1/Storj/Identity/storagenode/",destination=/app/identity \
	--mount type=bind,source="/volume1/Storj/",destination=/app/config \
	--name storagenode storjlabs/storagenode:latest
docker run -d --restart unless-stopped --stop-timeout 300 \
	-p 28967:28967/tcp \
	-p 28967:28967/udp \
	-p 14002:14002 \
	-e WALLET="xxxxx" \
	-e EMAIL="xxxxx" \
	-e ADDRESS="WAN-IP:28967" \
	-e STORAGE="14TB" \
	--mount type=bind,source="/volume1/Storj/Identity/storagenode/",destination=/app/identity \
	--mount type=bind,source="/volume1/Storj/",destination=/app/config \
	--name storagenode storjlabs/storagenode:latest \
	--log.level=error

docker run --rm -e SETUP="true" \
	--mount type=bind,source="/volume2/Storj2/Identity/storagenode/",destination=/app/identity \
	--mount type=bind,source="/volume2/Storj2/",destination=/app/config \
	--name storagenode2 storjlabs/storagenode:latest
docker run -d --restart unless-stopped --stop-timeout 300 \
	-p 28968:28968/tcp \
	-p 28968:28968/udp \
	-p 14003:14002 \
	-e WALLET="xxxxx" \
	-e EMAIL="xxxxx" \
	-e ADDRESS="WAN-IP:28968" \
	-e STORAGE="14TB" \
	--mount type=bind,source="/volume2/Storj2/Identity/storagenode/",destination=/app/identity \
	--mount type=bind,source="/volume2/Storj2/",destination=/app/config \
	--name storagenode2 storjlabs/storagenode:latest \
	--log.level=error

docker pull storjlabs/watchtower
docker run -d --restart=always --name watchtower -v /var/run/docker.sock:/var/run/docker.sock storjlabs/watchtower storagenode storagenode2 watchtower --stop-timeout 300s

docker ps -a

Other usefull commands:

docker stop -t 300 storagenode
docker start storagenode
docker rm storagenode

docker stop -t 300 storagenode2
docker start storagenode2
docker rm storagenode2

To close putty, just type exit and Enter 2 times.
You can also check all my firsts topics; I got answers to all newbie questions.

1 Like

Please never run this command for the working node more than once, you can destroy it. This command should be used only once for the entire node’s life.

The second run was for the second node, storagenode2. Thanks! I know that, I read the docs carrefuly before I click anything.

This is for all readers. I saw some scripts which have included both commands and they used this script to start the node after boot instead just providing --restart unless-stopped.
This is very dangerous use case, if disk not mount for any reason after reboot, the setup step will allow to bypass all checks to do not start the node with empty storage, and node will be disqualified.

Hi snorkel,

Thank you for your effort. Going to try answer all here :slight_smile:

  1. Bandwidth was in node creation code/string as parameter by default 10TB so I just adjusted. Maybe I will increase it to 1PB and this will like unlimited :wink:

  2. Here is output of that command:


  1. This was as port range:

Now I have configured new router and for each port own rule for every TCP and UDP:

image

And I can say this is works because I would be Offline if not right?

  1. I am using two port…on for VM Win10 machien and one 10GB as default one. They are connected to the same router. That VM has also SN installed and there is not issue with QUIC. Anyway this issue started before this VM has been created for testing (basically under Windows (physical PC or VM) there is no issue with QUIC.

image

  1. Yes Port forwarding is set to SYNOLOGY IP not Docker. Do no touching Docker networking (tried but nothing changed).

  2. What do you mean Internal and External? Set on router or in SN config?
    This is config:

As you can see higher forwarding ports always to the same internal and external. This worked without issue before that DSM update. Still works fine with Win10 machine.

  1. Yes using Putti and SSH for creation/running nodes itself. After each change I am recreating nodes not just restart. In Docker Terminal only checking behavior after recreation/reboot of the node.

  2. This has been done many times after ich change in DSM/Node configuration.

  3. I have ISP router set to Bridge and use own router with DHCP and NAT functions etc…even only one router (from ISP) has been used. Issue was the same.

  4. Firewall and DDOS disabled on DSM. On Router did not found DDOS protection functionality.

My config looks same as your but rmem_max set as shceduled task:

That part which mentioned Alexey I have not in my config file.

Thank you once again.

Going to try reset networking on NAS…if not help I will try reinstall DSM.

Wigo

@LordWigo
Point 6. Those ports in docker run command… check my commands and see what ports I used. Don’t use 28967 more than once. They say use it, but in DSM dosen’t work. Each node must have in it’s run command external:internal TCP and external:internal UDP. Each external and internal must match what you have in router, in that picture on the new router.
First node - 28967:28967 TCP, 28967:28967 UDP, second node - 28968:28968 TCP, 28968:28968 UDP and so on. And also update for each node the line with wan-ip:port (28967, 28968… etc). I will try to find my older posts.

On point 3, I didn’t realize that it’s a range, sorry, my mistake.

About the LAN ports on DSM… maybe you should use only one for the same router; On my DS220+ I have to ports, and I think they are intended to be used on 2 different networks, like a backup. Here I’m not so sure, maybe I’m wrong. You can try disconnect one and let only one cable connected, just to see if it makes a difference. Let LAN1 connected, set it’s IP to SNs IP (200), and restart.

DDOS prevention is off by default in Asus routers, and not many has it. If you can’t find it, dosen’t matter. Is off or non existent.

You can also try remove that setting with rmem netcore etc etc, and run my first 2 commands once from putty.