Hi everyone, we are excited to inform you that we have been working on performance improvement in our network. One area that we have been exploring is UDP based transport protocols. For example, We have done some internal testing and have seen some good result from QUIC protocol.
Now we would like to gather more realistic performance metrics with UDP based protocols. In order to do that, we want to ask you, our storage node operators, to update your port-forwarding rules to also include UDP traffic.
If you are running your node using Docker, please make sure to also change
from -p <host-port>:<container-port>
to -p <host-port>:<container-port>/tcp -p <host-port>:<container-port>/udp.
Once you have set it up for UDP traffic, you can use our connect-test tool to verify the setup.
You can install our connect-test tool by running below command in your terminal:
# Please make sure you have `go` installed on your machine first.
git clone https://github.com/storj/storj.git storj
cd storj
go install ./cmd/connect-test/...
Thank you for your cooperation and patience with us.
Have a nice day!
go install ./cmd/connect-test/…
cmd/connect-test/connecttest.go:13:2: cannot find package “golang.org/x/sync/errgroup” in any of:
/usr/lib/go-1.7/src/golang.org/x/sync/errgroup (from $GOROOT)
($GOPATH not set)
cmd/connect-test/connecttest.go:15:2: cannot find package “storj.io/common/identity” in any of:
/usr/lib/go-1.7/src/storj.io/common/identity (from $GOROOT)
($GOPATH not set)
cmd/connect-test/connecttest.go:16:2: cannot find package “storj.io/common/peertls/tlsopts” in any of:
/usr/lib/go-1.7/src/storj.io/common/peertls/tlsopts (from $GOROOT)
($GOPATH not set)
cmd/connect-test/connecttest.go:17:2: cannot find package “storj.io/common/rpc” in any of:
/usr/lib/go-1.7/src/storj.io/common/rpc (from $GOROOT)
($GOPATH not set)
cmd/connect-test/connecttest.go:18:2: cannot find package “storj.io/common/storj” in any of:
/usr/lib/go-1.7/src/storj.io/common/storj (from $GOROOT)
($GOPATH not set)
cmd/connect-test/connecttest.go:19:2: cannot find package “storj.io/storj/pkg/quic” in any of:
/usr/lib/go-1.7/src/storj.io/storj/pkg/quic (from $GOROOT)
($GOPATH not set)
Thank you for participating in this experiment. It looks like your go version is still on 1.7. Can you try to follow the instructions on here: https://golang.org/ to install the latest go version?
got 4 of 6 nodes restarted with the new docker container start cmd. seems to be working okay. I ran the connect test on one of the nodes and that seemed fine…
./connect-test
panic: runtime error: index out of range [1] with length 1
goroutine 1 [running]:
main.main()
/root/storj/cmd/connect-test/connecttest.go:23 +0x8c5
I’ve just tried this tool. However I am getting UDP buffer error. Is it required to adjust some settings as well?
This is default setting in docker without UDP port
-p 28967:28967 \
go run ./cmd/connect-test < ip-address >:< port-number >
2021/04/23 20:24:02 failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See UDP Receive Buffer Size · lucas-clemente/quic-go Wiki · GitHub for details.
QUIC fail quic error: Timeout: No recent network activity
TCP success 1giLFB912zDNzxqUS4h2dJ7Dnxs3ituzeWDNE7nxoJXjvpuUb4
Second attempt after adding UDP port into docker.
-p 28967:28967/tcp -p 28967:28967/udp \
go run ./cmd/connect-test < ip-address >:< port-number >
2021/04/23 20:29:22 failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See UDP Receive Buffer Size · lucas-clemente/quic-go Wiki · GitHub for details.
QUIC success 1giLFB912zDNzxqUS4h2dJ7Dnxs3ituzeWDNE7nxoJXjvpuUb4
TCP success 1giLFB912zDNzxqUS4h2dJ7Dnxs3ituzeWDNE7nxoJXjvpuUb4
Once UDP port added I am getting success response but I am not sure if this is enough or some configuration should be fixed for the UDP-Receive-Buffer-Size?
Looks like I have default values on both my test ubuntu linux and the storj boxes.
I explain how i managed to get this work on my Ubuntu setup.
So as i see any older version of golang didnt work as expected, i purged my previous golang versions:
sudo apt-get purge golang-go
Then i followed go installation from this site: Download and install - The Go Programming Language
Downloaded the mentioned go for linux tar.gz to my home folder
Logged in to command line with ssh connection
But I was only able to install it as root user, sudo doesnt allowed me to do it.
So i did:
su root
rm -rf /usr/local/go && tar -C /usr/local -xzf go1.16.3.linux-amd64.tar.gz
Then i edited my home user .profile file and added the following to it:
export PATH=$PATH:/usr/local/go/bin
Logged in as user to shh again to get export path working.
test it by typing :
go version
go version go1.16.3 linux/amd64
I simply used it with:
go run ./cmd/connect-test ip:port
Running it from previously created storj folder as user
It complained about “failed to sufficiently increase receive buffer size”
So i edited /etc/sysctl.conf
Addded to the end of the file:
net.core.rmem_max=2500000
net.core.rmem_default=2500000
Also run these 2 lines from command line, to get it work right away, not just from restart:
sudo sysctl net.core.rmem_max=2500000
sudo sysctl net.core.rmem_default=2500000
Run again:
go run ./cmd/connect-test ip:port
And get these results:
QUIC success 12nmGyHRiJDdZmZX4FxxxxYDPeE1nz2KoYdik5YnY2XuLKGyN
TCP success 12nmGyHRiJDdZmZX4FmYPH9pYDPeExxxxxnY2XuLKGyN
I hope it helped some people, who tried to get it work
That was a mistake on our end, sorry about that. We have fixed this issue. If you pull in the latest code from storj/storj repo by running below command in storj folder
git pull
You can then install the tool again with
go install ./cmd/connecttest/...
You should be able to see success if the port is setup correctly
Hi there, thank you for participating.
The message about increase receive buffer size is more of a warning. It would increase your node’s performance with QUIC if you do increase it, but it shouldn’t cause any operational issue if you don’t
I ran the connect test and also received success for QUIC and TCP along with the warning about the UDP buffer size.
I’m running the node in a docker container where the host has net.core.rmem_max = 212992 set but I’m unable to set this value on the docker container. Any suggestions on arguments I should be passing to docker to increase the buffer size?
Docker run has option --sysctl. My guess that it should be used. However since everything is working as is and connection can be opened via udp probably no reason to use it for now.
I don’t see any traffic difference on the node with enabled udp.