Experimenting with UDP based protocols

Hi everyone, we are excited to inform you that we have been working on performance improvement in our network. One area that we have been exploring is UDP based transport protocols. For example, We have done some internal testing and have seen some good result from QUIC protocol.

Now we would like to gather more realistic performance metrics with UDP based protocols. In order to do that, we want to ask you, our storage node operators, to update your port-forwarding rules to also include UDP traffic.
If you are running your node using Docker, please make sure to also change
from
-p <host-port>:<container-port>
to
-p <host-port>:<container-port>/tcp -p <host-port>:<container-port>/udp.

Once you have set it up for UDP traffic, you can use our connect-test tool to verify the setup.
You can install our connect-test tool by running below command in your terminal:

# Please make sure you have `go` installed on your machine first.
git clone https://github.com/storj/storj.git storj
cd storj
go install ./cmd/connect-test/...

Thank you for your cooperation and patience with us.
Have a nice day!

8 Likes

-bash: git: command not found

-bash: go: command not found

apt install go

E: Unable to locate package go

So how can I test UDP?

Try
apt-get install golang-go

go install ./cmd/connect-test/…
cmd/connect-test/connecttest.go:13:2: cannot find package “golang.org/x/sync/errgroup” in any of:
/usr/lib/go-1.7/src/golang.org/x/sync/errgroup (from $GOROOT)
($GOPATH not set)
cmd/connect-test/connecttest.go:15:2: cannot find package “storj.io/common/identity” in any of:
/usr/lib/go-1.7/src/storj.io/common/identity (from $GOROOT)
($GOPATH not set)
cmd/connect-test/connecttest.go:16:2: cannot find package “storj.io/common/peertls/tlsopts” in any of:
/usr/lib/go-1.7/src/storj.io/common/peertls/tlsopts (from $GOROOT)
($GOPATH not set)
cmd/connect-test/connecttest.go:17:2: cannot find package “storj.io/common/rpc” in any of:
/usr/lib/go-1.7/src/storj.io/common/rpc (from $GOROOT)
($GOPATH not set)
cmd/connect-test/connecttest.go:18:2: cannot find package “storj.io/common/storj” in any of:
/usr/lib/go-1.7/src/storj.io/common/storj (from $GOROOT)
($GOPATH not set)
cmd/connect-test/connecttest.go:19:2: cannot find package “storj.io/storj/pkg/quic” in any of:
/usr/lib/go-1.7/src/storj.io/storj/pkg/quic (from $GOROOT)
($GOPATH not set)

Thank you for participating in this experiment. It looks like your go version is still on 1.7. Can you try to follow the instructions on here: https://golang.org/ to install the latest go version?

1 Like

got 4 of 6 nodes restarted with the new docker container start cmd. seems to be working okay. I ran the connect test on one of the nodes and that seemed fine…

Any change required for a non-dockerized node?

1 Like

I already have:
go version go1.16.3 linux/amd64

How do I run it after installation?

./connect-test 
panic: runtime error: index out of range [1] with length 1

goroutine 1 [running]:
main.main()
	/root/storj/cmd/connect-test/connecttest.go:23 +0x8c5

I have updated my configurations, but did not yet use testing tool. A will try to use it a bit later.

Sorry about that. You can run the tool by passing in your node’s address like so:

connect-test <ip-address>:<port-number>

I’ve just tried this tool. However I am getting UDP buffer error. Is it required to adjust some settings as well?

This is default setting in docker without UDP port

-p 28967:28967 \

go run ./cmd/connect-test < ip-address >:< port-number >

2021/04/23 20:24:02 failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See UDP Receive Buffer Size · lucas-clemente/quic-go Wiki · GitHub for details.
QUIC fail quic error: Timeout: No recent network activity
TCP success 1giLFB912zDNzxqUS4h2dJ7Dnxs3ituzeWDNE7nxoJXjvpuUb4

Second attempt after adding UDP port into docker.

-p 28967:28967/tcp -p 28967:28967/udp \

go run ./cmd/connect-test < ip-address >:< port-number >

2021/04/23 20:29:22 failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See UDP Receive Buffer Size · lucas-clemente/quic-go Wiki · GitHub for details.
QUIC success 1giLFB912zDNzxqUS4h2dJ7Dnxs3ituzeWDNE7nxoJXjvpuUb4
TCP success 1giLFB912zDNzxqUS4h2dJ7Dnxs3ituzeWDNE7nxoJXjvpuUb4

Once UDP port added I am getting success response but I am not sure if this is enough or some configuration should be fixed for the UDP-Receive-Buffer-Size?

Looks like I have default values on both my test ubuntu linux and the storj boxes.

> sysctl net.core.rmem_max
net.core.rmem_max = 212992
1 Like

I explain how i managed to get this work on my Ubuntu setup.
So as i see any older version of golang didnt work as expected, i purged my previous golang versions:
sudo apt-get purge golang-go
Then i followed go installation from this site: Download and install - The Go Programming Language
Downloaded the mentioned go for linux tar.gz to my home folder
Logged in to command line with ssh connection
But I was only able to install it as root user, sudo doesnt allowed me to do it.
So i did:
su root
rm -rf /usr/local/go && tar -C /usr/local -xzf go1.16.3.linux-amd64.tar.gz

Then i edited my home user .profile file and added the following to it:
export PATH=$PATH:/usr/local/go/bin

Logged in as user to shh again to get export path working.
test it by typing :
go version
go version go1.16.3 linux/amd64

Git cloned the folder mentioned by Yingrong:

git clone https://github.com/storj/storj.git storj
cd storj
As I remember i don`t installed it with:
go install ./cmd/connect-test/…

I simply used it with:
go run ./cmd/connect-test ip:port
Running it from previously created storj folder as user
It complained about “failed to sufficiently increase receive buffer size”
So i edited /etc/sysctl.conf
Addded to the end of the file:
net.core.rmem_max=2500000
net.core.rmem_default=2500000
Also run these 2 lines from command line, to get it work right away, not just from restart:
sudo sysctl net.core.rmem_max=2500000
sudo sysctl net.core.rmem_default=2500000
Run again:
go run ./cmd/connect-test ip:port
And get these results:
QUIC success 12nmGyHRiJDdZmZX4FxxxxYDPeE1nz2KoYdik5YnY2XuLKGyN
TCP success 12nmGyHRiJDdZmZX4FmYPH9pYDPeExxxxxnY2XuLKGyN
I hope it helped some people, who tried to get it work

3 Likes

What IP: LAN (192.168.x.x) or WAN (195.15.1x.x) ???
What port: 28967 or what ???

My result:

QUIC	fail	quic error: CRYPTO_ERROR (0x12a): tls peer certificate verification error: tlsopts error: node ID version error: no allowed peer identity versions specified
TCP	fail	rpc: tls peer certificate verification error: tlsopts error: node ID version error: no allowed peer identity versions specified

That was a mistake on our end, sorry about that. We have fixed this issue. If you pull in the latest code from storj/storj repo by running below command in storj folder

git pull

You can then install the tool again with

go install ./cmd/connecttest/...

You should be able to see success if the port is setup correctly

1 Like

Hi there, thank you for participating.
The message about increase receive buffer size is more of a warning. It would increase your node’s performance with QUIC if you do increase it, but it shouldn’t cause any operational issue if you don’t

And finally:

QUIC	success	1aw8uYQqEogwmshiigNaG3XkMbfgVgN36EahTVH7m74naXzyiH
TCP	success	1aw8uYQqEogwmshiigNaG3XkMbfgVgN36EahTVH7m74naXzyiH

It helped

git pull

As he understands it works for the TO traffic of the node …
Do I need to set something up for the FROM traffic of a node ???

I ran the connect test and also received success for QUIC and TCP along with the warning about the UDP buffer size.

I’m running the node in a docker container where the host has net.core.rmem_max = 212992 set but I’m unable to set this value on the docker container. Any suggestions on arguments I should be passing to docker to increase the buffer size?

Docker run has option --sysctl. My guess that it should be used. However since everything is working as is and connection can be opened via udp probably no reason to use it for now.
I don’t see any traffic difference on the node with enabled udp.

1 Like

Presumably, the nodes running natively with the Linux binary will just use UDP provided the port forwarding is set up for both TCP and UDP?

1 Like