Please enable TCP fastopen on your storage nodes

arrogantrabbit · June 6, 2025, 5:55pm

Has this been done but bug not updated, or not?

I don’t see fast open being used on FreeBSD:

I’m checking with

tcpdump -i epair0b -nn -vvv 'host 10.0.17.120 and port 28967 and tcp' | egrep 'flags \[.*S.*\]'

kernel is configured properly:

% sysctl -a | egrep 'fastopen.*enable'
net.inet.tcp.fastopen.server_enable: 1
net.inet.tcp.fastopen.psk_enable: 0
net.inet.tcp.fastopen.client_enable: 1

and I see this in the node log in start:

2025-06-06T11:00:02-07:00       INFO    server  kernel support for server-side tcp fast open enabled.   {"Process": "storagenode"}

@jtolio , @littleskunk , does it mean the support is in place, but I don’t see it on tcpdump because customers don’t request that?

Why don’t customers use that? That seems like a problem then. I’ve waited 10 min, not a single fast open connection on my 3 year old node.

Or perhaps vast majority of clients don’t create more than one connection to each node? In which case fastopen won’t help. But first post claims it does.

I’m confused. Please elaborate.

jtolio · June 30, 2025, 9:31pm

This is a good question, and honestly I’m not really sure.

You’re right that it appears we have FreeBSD support: storj/private/server/fastopen_freebsd.go at 4bea9e65d86f8ddc27d0ce7199c93fa80dee10d9 · storj/storj · GitHub and gained it 2 years ago.

I’ve checked a couple of things and I see at least two FreeBSD storage nodes that have identified themselves to the us1 Satellite as supporting TCP_FASTOPEN, and at least one that doesn’t. I don’t know if these are yours (I could confirm over DM), but if they are, there are many other hops in between that might kill the TCP_FASTOPEN chances. Just because a node supports TCP_FASTOPEN doesn’t mean the entire route does. Because of this, even for nodes that advertise support, the client software will always double dial, both with and without TCP_FASTOPEN concurrently, send requests down both channels (with a unique identifier), and then the storage node will respond to the first of that unique id that it hears about.

If you aren’t getting any TCP_FASTOPEN traffic, my next debugging question would be what happens on the client side when a client tries to talk to your node? Could Linux client operating systems be getting just enough of a difference from the FreeBSD stack to bail and not try?

TCP_FASTOPEN is supposed to have a sort of session cookie that gets established on first contact. I assume you haven’t seen any of those either?

arrogantrabbit · July 2, 2025, 5:23am

Thank you! This sent me to a rabbit hole of actually reading documentation… Indeed, the way I was checking is wrong. I shall be looking for SYN-ACK with cookie request, and client’s initial SYN shall contain both cookie and data.

So I tried grepping for “tfo” with context:

tcpdump -i epair0b -nn -vvv 'host 10.0.17.120 and port 28967 and tcp' | grep -C 5 'tfo'

This caught PLENTY of packets like these:

22:02:38.456947 IP (tos 0x0, ttl 55, id 7510, offset 0, flags [DF], proto TCP (6), length 652)
    109.61.92.70.54618 > 10.0.17.120.28967: Flags [S], cksum 0x51e6 (correct), seq 4282957032:4282957612, win 64240, options [mss 1460,sackOK,TS val 4145509149 ecr 0,nop,wscale 12,tfo  cookie 73fe999eb6d2f7e3,nop,nop], length 580
                                                  \./                                                                                                                               \                          /           \        / 
                               initial SYN packet -┘                                                                                                                                 \____________.___________/             \      /
                               tfo cookie being presented ----------------------------------------------------------------------------------------------------------------------------------------┘                          \__._/
                               Non-zero, actually, quite large, size of this SYN packet! ---------------------------------------------------------------------------------------------------------------------------------------┘

So, TFO is working!

I see the double-dials:

Regilar SYN (zero length):

 22:02:38.455225 ... 109.61.92.70.54620 > 10.0.17.120.28967: Flags [S], ... length 0

And one millisecond later the TFO one (with data):

22:02:38.456947 ... 109.61.92.70.54618 > 10.0.17.120.28967: Flags [S], ... options [...tfo cookie...], length 580

Also confirmed this works via my wireguard tunnel

root@storj:~ # tcpdump -i oracle_sj -nn -vvv 'host 10.148.251.56 and port 28967 and tcp' | grep 'tfo  cookie'
tcpdump: listening on oracle_sj, link-type NULL (BSD loopback), capture size 262144 bytes
    10.148.251.56.28967 > 10.148.251.55.34664: Flags [S.], cksum 0xb08c (correct), seq 3635447846:3635447898, ack 3115428476, win 65535, options [mss 1290,nop,wscale 11,sackOK,TS val 1648422202 ecr 685191259,tfo  cookie 0b00909d3d9c24e5,eol], length 52
    10.148.251.55.57750 > 10.148.251.56.28967: Flags [S], cksum 0xece3 (correct), seq 991295404:991295982, win 64240, options [mss 1460,sackOK,TS val 3852407911 ecr 0,nop,wscale 12,tfo  cookie 0b00909d3d9c24e5,nop,nop], length 578
    10.148.251.56.28967 > 10.148.251.55.57750: Flags [S.], cksum 0xacc3 (correct), seq 1056604353:1056604405, ack 991295983, win 65535, options [mss 1290,nop,wscale 11,sackOK,TS val 1303678033 ecr 3852407911,tfo  cookie 0b00909d3d9c24e5,eol], length 52
    10.148.251.55.36152 > 10.148.251.56.28967: Flags [S], cksum 0x84ea (correct), seq 3527844255:3527844833, win 64240, options [mss 1460,sackOK,TS val 3313657433 ecr 0,nop,wscale 12,tfo  cookie 0b00909d3d9c24e5,nop,nop], length 578
    10.148.251.56.28967 > 10.148.251.55.36152: Flags [S.], cksum 0x463a (correct), seq 3867278105:3867278157, ack 3527844834, win 65535, options [mss 1290,nop,wscale 11,sackOK,TS val 2317840120 ecr 3313657433,tfo  cookie 0b00909d3d9c24e5,eol], length 52
    10.148.251.55.57774 > 10.148.251.56.28967: Flags [S], cksum 0xa590 (correct), seq 273630421:273630999, win 64240, options [mss 1460,sackOK,TS val 3852408291 ecr 0,nop,wscale 12,tfo  cookie 0b00909d3d9c24e5,nop,nop], length 578
    10.148.251.56.28967 > 10.148.251.55.57774: Flags [S.], cksum 0xb600 (correct), seq 1933086348:1933086400, ack 273631000, win 65535, options [mss 1290,nop,wscale 11,sackOK,TS val 3807772045 ecr 3852408291,tfo  cookie 0b00909d3d9c24e5,eol], length 52
    10.148.251.55.31576 > 10.148.251.56.28967: Flags [S], cksum 0xae8d (correct), seq 1025325158:1025326408, win 64240, options [mss 1460,sackOK,TS val 1523425906 ecr 0,nop,wscale 12,tfo  cookie 0b00909d3d9c24e5,nop,nop], length 1250
    10.148.251.56.28967 > 10.148.251.55.31576: Flags [S.], cksum 0xea79 (correct), seq 1980191881, ack 1025326409, win 65535, options [mss 1290,nop,wscale 11,sackOK,TS val 1185713387 ecr 1523425906,tfo  cookie 0b00909d3d9c24e5,eol], length 0
    10.148.251.55.60462 > 10.148.251.56.28967: Flags [S], cksum 0x8ded (correct), seq 879419246:879419820, win 64240, options [mss 1460,sackOK,TS val 3285683914 ecr 0,nop,wscale 12,tfo  cookie 0b00909d3d9c24e5,nop,nop], length 574
    10.148.251.56.28967 > 10.148.251.55.60462: Flags [S.], cksum 0x360e (correct), seq 4282383616:4282383668, ack 879419821, win 65535, options [mss 1290,nop,wscale 11,sackOK,TS val 861036829 ecr 3285683914,tfo  cookie 0b00909d3d9c24e5,eol], length 52
    10.148.251.55.37048 > 10.148.251.56.28967: Flags [S], cksum 0xf633 (correct), seq 4258376682:4258377260, win 64240, options [mss 1460,sackOK,TS val 757673314 ecr 0,nop,wscale 12,tfo  cookie 0b00909d3d9c24e5,nop,nop], length 578
    10.148.251.56.28967 > 10.148.251.55.37048: Flags [S.], cksum 0x6bd2 (correct), seq 745264083:745264135, ack 4258377261, win 65535, options [mss 1290,nop,wscale 11,sackOK,TS val 519466014 ecr 757673314,tfo  cookie 0b00909d3d9c24e5,eol], length 52
    10.148.251.55.43522 > 10.148.251.56.28967: Flags [S], cksum 0xf809 (correct), seq 1924663895:1924664470, win 64240, options [mss 1460,sackOK,TS val 646979584 ecr 0,nop,wscale 12,tfo  cookie 0b00909d3d9c24e5,nop,nop], length 575
    10.148.251.56.28967 > 10.148.251.55.43522: Flags [S.], cksum 0x207f (correct), seq 1597437344:1597437396, ack 1924664471, win 65535, options [mss 1290,nop,wscale 11,sackOK,TS val 790892912 ecr 646979584,tfo  cookie 0b00909d3d9c24e5,eol], length 52

Side node: even though net.inet.tcp.fastopen.server_enable is a kernel feature, I had to add it to jails /etc/sysctl.conf manually as well.

Rijeho · July 5, 2025, 12:58pm

Hello Community

I don’t know if I should have created a new topic or continued here.

I wanted to activate TCPFastOpen on one of my 7 neuds, to test the difference between those where it’s activated and those where it’s not.

To do this, I followed the advice linked to my configuration, debian 12 arm with a kernel that supports TCPFastOpen (it seems to me).
and configuration on docker with flag --sysctl net.ipv4.tcp_fastopen=3.

It’s now been a little over 24 hours since the system has been up and running again, but I still have zero usage.

cat /proc/sys/net/ipv4/tcp_fastopen

which returns 3


cat /proc/net/netstat | awk '/^TcpExt: / { if (NR==1) {for(i=1;i<=NF;i++) if($i~/TCPFastOpen/) print i,$i} else {for(i=1;i<=NF;i++) if(prev[i]~/TCPFastOpen/) print prev[i],$i} if(NR==1) {for(i=1;i<=NF;i++) prev[i]=$i} }'

which returns : 

85 TCPFastOpenActive
86 TCPFastOpenActiveFail
87 TCPFastOpenPassive
88 TCPFastOpenPassiveFail
89 TCPFastOpenListenOverflow
90 TCPFastOpenCookieReqd
91 TCPFastOpenBlackhole
120 TCPFastOpenPassiveAltKey
TCPFastOpenActive 0
TCPFastOpenActiveFail 0
TCPFastOpenPassive 0
TCPFastOpenPassiveFail 0
TCPFastOpenListenOverflow 0
TCPFastOpenCookieReqd 34
TCPFastOpenBlackhole 0
TCPFastOpenPassiveAltKey 0

The question is, have I configured my server correctly?

TCPFastOpenCookieReqd: 34 : Your system has requested 34 TFO cookies
TCPFastOpenActive: 0 : No successful active TFO connection
TCPFastOpenPassive: 0 : No accepted passive TFO connection

What I understand :

tcp_fastopen = 3: TCP Fast Open is enabled on client and server side.
TCP Fast Open works at kernel level
My system is trying to use TFO
But the remote servers don’t seem to support TFO

can you confirm that this is what I understand and tell me if it’s normal that no one is using it?

arrogantrabbit · July 5, 2025, 3:43pm

Check node logs. Does it detect fastopen as enabled? You may need to set the sysctl both on the host and in the container.

There is no need to wait 24 hours, all clients double-dial, so you will see traffic immediately.

If you see node detecting fastopen on start (confirm in the log), then inspect actual packets with tcpdump, see if syn has non-zero size and check for cookies.

If node detect fastopen but you don’t see back and force traffic with cookies — only then investigate further. First, rule out the retched docker. Throw is away, run node directly. If still nothing — check your gateway, it may not understand fastopen traffic and block it as invalid. At this point it would be easier to test with separate small apps to create and check fastopen connection than messing with node.

Becuse clients always create both connections there is no downsize in enabling fastopen.