Please enable TCP fastopen on your storage nodes

Product Discussions STORJLINGS
,
1

Let’s start with the bad news. We don’t have TCP fastopen support for Windows nodes running the binary. So if you are running the Windows binary you can stop here. Windows support will get implemented with this ticket: Windows support for TCP Fast Open · Issue #5990 · storj/storj · GitHub

Please grep your logs to see if you have any loglines with fastopen. Ideally, you have no loglines at all. That would indicate TCP fastopen is working fine on your node.

If you see something like this in your logs please enable TCP fastopen on your machine. The goal is to make these loglines go away.

INFO    server  server/fastopen_linux.go:44     kernel support for server-side tcp fast open remains disabled.  {"Process": "storagenode"}
INFO    server  server/fastopen_linux.go:50     enable with: sysctl -w net.ipv4.tcp_fastopen=3  {"Process": "storagenode"}

After the next storage node release v1.81 has been rolled out we want to run performance tests with TCP fastopen. The loglines above are already in the code for quite some time. The new release will just add some metrics for us to count the number of TCP fastopen nodes. More nodes supporting it will help us to get better test results.

1 Like
2

A small side note: if you followed along with the original TCP FASTOPEN design doc discussion, you may be worried that it is not always safe to enable TCP_FASTOPEN for your storage node. You no longer need to worry about that! We figured out how to make it always safe. Clients that try to use TCP_FASTOPEN will always concurrently try a normal TCP dial, so that at least one will succeed. The node also knows how to effectively “debounce” the incoming connections if both succeed.

In fact, there is now a substantial benefit to enabling TCP_FASTOPEN - if it is enabled, your node has a better chance of winning upload races.

4 Likes
3

Is there any way to confirm that fastopen is enabled, without restart of the node and check of startup logs?

4

There’s more discussion in Two new blueprints/design drafts seeking feedback: Replacing TLS with Noise and TCP_FASTOPEN if you want the details, but you should check net.ipv4.tcp_fastopen in your kernel. It should have the 0x2 bit set, (0x1 is default on), so most likely you should have the value of 3 there.

1 Like
5

On FreeBSD I see this:

# sysctl -a | egrep 'fastopen.*enable'
net.inet.tcp.fastopen.server_enable: 0
net.inet.tcp.fastopen.psk_enable: 0
net.inet.tcp.fastopen.client_enable: 1

Is the client_enable sufficient? Or is fastopne only supported on linux versions of storagenode for now?

6

Considering the partnership with ixsys, will it be automatically enabled on TrueNAS?

7

For the Linux Idiots like me, any chance you could give me the exact command I should type?
:flushed:

3 Likes
8

We need server-side enabled, but we also don’t support TCP FASTOPEN on FreeBSD yet. Set server_enable to true and hopefully a future release of the storage node on FreeBSD will take use of that.

2 Likes
9

@ACarneiro:

I suppose the command is:

sysctl -w net.ipv4.tcp_fastopen=3

But it needs to be run every boot. Alternatively, you could add

net.ipv4.tcp_fastopen = 3

to /etc/sysctl.conf which takes effect each boot. I don’t know how to translate this into your specific distribution, system, or container setup.

4 Likes
10

It’s in system settings → advanced in TrueNAS Scale but that alone doesn’t seem to take effect inside the kubernetes container. Haven’t rebooted yet but have restarted the app and still no fastopen.

1 Like
11

That is not enough for those running on docker

12

This configuration controls kernel behavior, and needs to be made on the host OS, docker has nothing do do with this*.

* unless you are using a platform, where docker engine itself is running in a VM (windows, macOS). Then you need to configure that in that VM.

13

I know that

On debian (RaspiOS) did the change, restarted the node and still got in the logs:

INFO kernel support for server-side tcp fast open remains disabled

You additionally need to recreate the container including the parameter --sysctl net.ipv4.tcp_fastopen=3

NOTE: I’m running the docker container as root.

2 Likes
14

@mgonzalezm
Where exactely do I must put this parameter in the docker run command?
Can you share your run command?

15

anywhere before the container name

2 Likes
16

This should be in the official docs.
I’m running Synology’s, with container as root, too. Thanks!

17

Thank you.
And when you did that you stopped getting the error messages?

18

I remember now… Synology dosen’t support that parameter. It gives this error:

docker: Error response from daemon: OCI runtime create failed: container_linux.go:367: starting container process caused: process_linux.go:495: container init caused: write sysctl key net.ipv4.tcp_fastopen: open /proc/sys/net/ipv4/tcp_fastopen: no such file or directory: unknown.

I just enabled it in sysctl.conf; I don’t see any log entries for tcp fast open or fastopen, in error log mode. Maybe I’m OK?! Should I switch to info mode?

19

Yes I believe it is log level Info

20

log level = info

INFO server kernel support for tcp fast open unknown {"process": "storagenode"}