Please enable TCP fastopen on your storage nodes

Product Discussions STORJLINGS
,
41

Since I just did this for my own Raspberry Pi, here are the instructions that can be used by others. This was done on Raspbian buster but should likely work for others.

The following displayed command prompts show an example of user pi on a system named storjpi.

Connect to your RPi via ssh or console and query your logs as indicated prior to see if you have the fastopen warning.

Check if your current config has fastopen defined:
pi@storjpi:~ $ grep fastopen /etc/sysctl.conf

If a line is returned and it is not set to 3, edit the file (sudoedit /etc/sysctl.conf) and change the value to 3.
If a line is not returned append fastopen to the end of the config:

pi@storjpi:~ $ sudo su -
root@storjpi:~# echo "
#Enabling fastopen for storj
net.ipv4.tcp_fastopen = 3" >> /etc/sysctl.conf
root@storjpi:~# exit

Note that when you paste or type the echo lines in there will be greater-than symbols shown at each new line. I’ve omitted them here for ease of copy/paste.

Shutdown and remove current storj docker containers:

pi@storjpi:~ $ docker stop -t 300 $(docker ps -a -q); docker rm $(docker ps -a -q)

Reboot:
pi@storjpi:~ $ sudo reboot now

After reboot reconnect and confirm the OS fastopen status shows 3:
pi@storjpi:~ $ sudo sysctl -a | grep "net.ipv4.tcp_fastopen ="

Recreate your storj containers adding the --sysctl net.ipv4.tcp_fastopen=3 parameter making sure you put it before storjlabs/storagenode:latest image name, then re-grep your logs to confirm the fastopen warning does not exist.

4 Likes
42

when I run command
$ grep fastopen /etc/sysctl.conf
it doesn’t display anything just goes like im ready to type another command
PI4 running ubuntu

43

Sanity check here… In my example all the pi@storjpi:~ $ and root@storjpi:~# strings are what you would see at the command prompt. Since you included the $ in your reply I want to make sure that is just copied from your command prompt and not something in the command you’re trying to execute.

No response text could be valid if there is no line with “fastopen” in the /etc/sysctl.conf file. On a non-arm Ubuntu 22.04.2 install I’ve got that is what happens when I run the command. I guess an additional test is to also confirm what your fastopen is currently set to per sysctl. If you run command sudo sysctl -a | grep "net.ipv4.tcp_fastopen =" you should get a line back. If this is anything but 3 then I believe you’ll want to edit /etc/sysctl.conf to add that in as applied at boot.

44

This file is not the only way to configure sysctl. Instead, grep output from sysctl:

sysctl -a | grep fastopen
1 Like
45

Should it be logging the fastopen stuff on node startup? Because I’ve restarted my node a few times, and I’m prepping the logs to see nothing

46

You need to have logs on info level, not error or warning.

47

Will support for windows be added later as well? Or has it been looked into and determined to be impossible?

48

This is the code for windows

My attempt for Windows

49

We don’t have TCP fastopen support for Windows nodes running the binary. So if you are running the Windows binary you can stop here.

From the initial post, hence I don’t expect it to work, instead I wanted to know if it will be supported later or not…?

50

How do you change this?

Edit: nevermind. It’s now been changed, but I’m not seeing any thing to do with fastopen being logged.

51

You may check:

sysctl net.ipv4.tcp_fastopen

and in the container:

docker exec -it storagenode cat /proc/sys/net/ipv4/tcp_fastopen
52

Under windows it shows

docker exec -it storagenode ls /proc/sys/net/ipv4
conf                               neigh
fib_multipath_use_neigh            ping_group_range
fwmark_reflect                     route
icmp_echo_ignore_all               tcp_base_mss
icmp_echo_ignore_broadcasts        tcp_ecn
icmp_errors_use_inbound_ifaddr     tcp_ecn_fallback
icmp_ignore_bogus_error_responses  tcp_fin_timeout
icmp_ratelimit                     tcp_fwmark_accept
icmp_ratemask                      tcp_keepalive_intvl
igmp_link_local_mcast_reports      tcp_keepalive_probes
igmp_max_memberships               tcp_keepalive_time
igmp_max_msf                       tcp_l3mdev_accept
igmp_qrv                           tcp_mtu_probing
ip_default_ttl                     tcp_notsent_lowat
ip_dynaddr                         tcp_orphan_retries
ip_early_demux                     tcp_probe_interval
ip_forward                         tcp_probe_threshold
ip_forward_use_pmtu                tcp_reordering
ip_local_port_range                tcp_retries1
ip_local_reserved_ports            tcp_retries2
ip_no_pmtu_disc                    tcp_syn_retries
ip_nonlocal_bind                   tcp_synack_retries
ipfrag_high_thresh                 tcp_syncookies
ipfrag_low_thresh                  vs
ipfrag_max_dist                    xfrm4_gc_thresh
ipfrag_time

Obvious result

docker exec -it storagenode cat /proc/sys/net/ipv4/tcp_fastopen
cat: /proc/sys/net/ipv4/tcp_fastopen: No such file or directory

53

You need to use an wsl2 engine to make it available. However, for Hyper-V engine there is a workaround as well:

But use

sysctl -w net.ipv4.tcp_fastopen=3

however, this change is not persistent and does not survive reboot.

In case of wsl2 you need to set it in the wsl2 shell for the docker-desktop distro as for Linux and use --sysctl net.ipv4.tcp_fastopen=3 option in the docker run command.

1 Like
54

Once enabled, what are the logs with fast open supposed to look like? What am I looking for to make sure it is working?

55

Dosen’t give you errors in info mode. See above.

56

It checks fastopen only on start, so you need to restart the container/service and check your logs.

1 Like
57

This is what I see after the work around

/ # sysctl -a | grep net.ipv4.tcp_fastopen
net.ipv4.tcp_fastopen = 3

After stop, remove, start of the container I see :point_down:

2023-06-17T00:27:35.282Z INFO server kernel support for tcp fast open unknown {“process”: “storagenode”}

58

You also need to use --sysctl net.ipv4.tcp_fastopen=3 in your docker run command.

59

Yes I did that too but it doesn’t seem to work :frowning:

docker: Error response from daemon: OCI runtime create failed: container_linux.go:344: starting container process caused “process_linux.go:424: container init caused "write sysctl key net.ipv4.tcp_fastopen: open /proc/sys/net/ipv4/tcp_fastopen: no such file or directory"”: unknown.

60

Then it’s not supported by the used kernel I guess, or there is a special docker software.

1 Like