Please enable TCP fastopen on your storage nodes

Product Discussions STORJLINGS
,
184

Thank you! This sent me to a rabbit hole of actually reading documentation… Indeed, the way I was checking is wrong. I shall be looking for SYN-ACK with cookie request, and client’s initial SYN shall contain both cookie and data.

So I tried grepping for “tfo” with context:

tcpdump -i epair0b -nn -vvv 'host 10.0.17.120 and port 28967 and tcp' | grep -C 5 'tfo'

This caught PLENTY of packets like these:

22:02:38.456947 IP (tos 0x0, ttl 55, id 7510, offset 0, flags [DF], proto TCP (6), length 652)
    109.61.92.70.54618 > 10.0.17.120.28967: Flags [S], cksum 0x51e6 (correct), seq 4282957032:4282957612, win 64240, options [mss 1460,sackOK,TS val 4145509149 ecr 0,nop,wscale 12,tfo  cookie 73fe999eb6d2f7e3,nop,nop], length 580
                                                  \./                                                                                                                               \                          /           \        / 
                               initial SYN packet -┘                                                                                                                                 \____________.___________/             \      /
                               tfo cookie being presented ----------------------------------------------------------------------------------------------------------------------------------------┘                          \__._/
                               Non-zero, actually, quite large, size of this SYN packet! ---------------------------------------------------------------------------------------------------------------------------------------┘

So, TFO is working!
:partying_face:

I see the double-dials:

Regilar SYN (zero length):

 22:02:38.455225 ... 109.61.92.70.54620 > 10.0.17.120.28967: Flags [S], ... length 0

And one millisecond later the TFO one (with data):

22:02:38.456947 ... 109.61.92.70.54618 > 10.0.17.120.28967: Flags [S], ... options [...tfo cookie...], length 580
Also confirmed this works via my wireguard tunnel 
root@storj:~ # tcpdump -i oracle_sj -nn -vvv 'host 10.148.251.56 and port 28967 and tcp' | grep 'tfo  cookie'
tcpdump: listening on oracle_sj, link-type NULL (BSD loopback), capture size 262144 bytes
    10.148.251.56.28967 > 10.148.251.55.34664: Flags [S.], cksum 0xb08c (correct), seq 3635447846:3635447898, ack 3115428476, win 65535, options [mss 1290,nop,wscale 11,sackOK,TS val 1648422202 ecr 685191259,tfo  cookie 0b00909d3d9c24e5,eol], length 52
    10.148.251.55.57750 > 10.148.251.56.28967: Flags [S], cksum 0xece3 (correct), seq 991295404:991295982, win 64240, options [mss 1460,sackOK,TS val 3852407911 ecr 0,nop,wscale 12,tfo  cookie 0b00909d3d9c24e5,nop,nop], length 578
    10.148.251.56.28967 > 10.148.251.55.57750: Flags [S.], cksum 0xacc3 (correct), seq 1056604353:1056604405, ack 991295983, win 65535, options [mss 1290,nop,wscale 11,sackOK,TS val 1303678033 ecr 3852407911,tfo  cookie 0b00909d3d9c24e5,eol], length 52
    10.148.251.55.36152 > 10.148.251.56.28967: Flags [S], cksum 0x84ea (correct), seq 3527844255:3527844833, win 64240, options [mss 1460,sackOK,TS val 3313657433 ecr 0,nop,wscale 12,tfo  cookie 0b00909d3d9c24e5,nop,nop], length 578
    10.148.251.56.28967 > 10.148.251.55.36152: Flags [S.], cksum 0x463a (correct), seq 3867278105:3867278157, ack 3527844834, win 65535, options [mss 1290,nop,wscale 11,sackOK,TS val 2317840120 ecr 3313657433,tfo  cookie 0b00909d3d9c24e5,eol], length 52
    10.148.251.55.57774 > 10.148.251.56.28967: Flags [S], cksum 0xa590 (correct), seq 273630421:273630999, win 64240, options [mss 1460,sackOK,TS val 3852408291 ecr 0,nop,wscale 12,tfo  cookie 0b00909d3d9c24e5,nop,nop], length 578
    10.148.251.56.28967 > 10.148.251.55.57774: Flags [S.], cksum 0xb600 (correct), seq 1933086348:1933086400, ack 273631000, win 65535, options [mss 1290,nop,wscale 11,sackOK,TS val 3807772045 ecr 3852408291,tfo  cookie 0b00909d3d9c24e5,eol], length 52
    10.148.251.55.31576 > 10.148.251.56.28967: Flags [S], cksum 0xae8d (correct), seq 1025325158:1025326408, win 64240, options [mss 1460,sackOK,TS val 1523425906 ecr 0,nop,wscale 12,tfo  cookie 0b00909d3d9c24e5,nop,nop], length 1250
    10.148.251.56.28967 > 10.148.251.55.31576: Flags [S.], cksum 0xea79 (correct), seq 1980191881, ack 1025326409, win 65535, options [mss 1290,nop,wscale 11,sackOK,TS val 1185713387 ecr 1523425906,tfo  cookie 0b00909d3d9c24e5,eol], length 0
    10.148.251.55.60462 > 10.148.251.56.28967: Flags [S], cksum 0x8ded (correct), seq 879419246:879419820, win 64240, options [mss 1460,sackOK,TS val 3285683914 ecr 0,nop,wscale 12,tfo  cookie 0b00909d3d9c24e5,nop,nop], length 574
    10.148.251.56.28967 > 10.148.251.55.60462: Flags [S.], cksum 0x360e (correct), seq 4282383616:4282383668, ack 879419821, win 65535, options [mss 1290,nop,wscale 11,sackOK,TS val 861036829 ecr 3285683914,tfo  cookie 0b00909d3d9c24e5,eol], length 52
    10.148.251.55.37048 > 10.148.251.56.28967: Flags [S], cksum 0xf633 (correct), seq 4258376682:4258377260, win 64240, options [mss 1460,sackOK,TS val 757673314 ecr 0,nop,wscale 12,tfo  cookie 0b00909d3d9c24e5,nop,nop], length 578
    10.148.251.56.28967 > 10.148.251.55.37048: Flags [S.], cksum 0x6bd2 (correct), seq 745264083:745264135, ack 4258377261, win 65535, options [mss 1290,nop,wscale 11,sackOK,TS val 519466014 ecr 757673314,tfo  cookie 0b00909d3d9c24e5,eol], length 52
    10.148.251.55.43522 > 10.148.251.56.28967: Flags [S], cksum 0xf809 (correct), seq 1924663895:1924664470, win 64240, options [mss 1460,sackOK,TS val 646979584 ecr 0,nop,wscale 12,tfo  cookie 0b00909d3d9c24e5,nop,nop], length 575
    10.148.251.56.28967 > 10.148.251.55.43522: Flags [S.], cksum 0x207f (correct), seq 1597437344:1597437396, ack 1924664471, win 65535, options [mss 1290,nop,wscale 11,sackOK,TS val 790892912 ecr 646979584,tfo  cookie 0b00909d3d9c24e5,eol], length 52

Side node: even though net.inet.tcp.fastopen.server_enable is a kernel feature, I had to add it to jails /etc/sysctl.conf manually as well.

1 Like