Two new blueprints/design drafts seeking feedback: Replacing TLS with Noise and TCP_FASTOPEN

[BrightSilence]

@jtolio

Otherwise, my impression from this thread is that support is going to be hairy. Even though I haven’t figured out how to tell from the client side if a TCP_FASTOPEN connection was successfully established or not, it seems like from all of your experiences that even just timing the connection would be enlightening.

One update from my end. I’ve recently noticed a wider range of problems on my network… turns out my ISP switched on intrusion detection and ddos prevention settings in my router that I had previously turned off. This caused both the slowdown in response and remote TCP_FASTOPEN not working. After turning this back off again, TCP_FASTOPEN not only works, but seems to work instantly every time. Thanks ISP… very useful… sigh

[thelastspark]

How did you find out they did it?

[BrightSilence]

I certainly didn’t do it. And they’ve done this stuff before. They even turned wifi back on a while back, which I don’t use my ISPs router for. They think they know what’s best, it’s really annoying.

Edit: Take a wild guess when I switched these “features” off…

[jtolio]

A major update - we have implemented and tested a new approach to TCP_FASTOPEN that should have none of the downsides. We’re just going to do both!

See the details here: https://review.dev.storj.io/c/storj/storj/+/9933/1/docs/blueprints/tcp-fastopen.md

[vladro]

@jtolio

e case we’re worried about is when “complete” isn’t sent at all, and as far as I can tell from your experiments, “complete” was always sent, even when TCP_FASTOPEN didn’t work, which means it always gracefully fell back to normal TCP. Which is perfect! That’s exactly what we want to happen.

What I’ve been worried about is if “complete” never made it, because the packets weren’t delivered, and the connection timed out. The ideal case is that TCP_FASTOPEN is completely successful, but it’s okay if it’s not, as long as the connection itself still is. It looks like your connections were always successful! Hooray!

Let’s see what more tests look like, but if every test is like yours, I hav

Need help with windows server configuration, pls
Got fastopen configured on server:

but still getting multiply errors messages from nodes like this:

2023-03-27T21:22:29.977+0300	ERROR	piecestore	download failed	{Piece ID: AWNGMFE63HEOYWPFHUDYYZ5X7BSG5J7X2ZX7N4IMDFKR2B3YG7XQ, Satellite ID: 12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs, Action: GET, Offset: 232960, Size: 163840, Remote Address: 217.15.17.82:29916, error: write tcp 192.168.51.2:58001->217.15.17.82:29916: wsasend: An existing connection was forcibly closed by the remote host., errorVerbose: write tcp 192.168.51.2:58001->217.15.17.82:29916: wsasend: An existing connection was forcibly closed by the remote host.\n\tstorj.io/drpc/drpcstream.(*Stream).rawFlushLocked:401\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:462\n\tstorj.io/common/pb.(*drpcPiecestore_DownloadStream).Send:349\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func6.2:729\n\tstorj.io/common/rpc/rpctimeout.Run.func1:22}
2023-03-27T21:22:36.847+0300	ERROR	piecestore	download failed	{Piece ID: PCVL4BIJKCOW7Y5WS3MZC6XUGU6FHXDPZYMNNQUSS6UDRGCEMOAA, Satellite ID: 12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs, Action: GET, Offset: 1510912, Size: 65536, Remote Address: 217.15.17.82:25808, error: manager closed: read tcp 192.168.51.2:58001->217.15.17.82:25808: wsarecv: An existing connection was forcibly closed by the remote host., errorVerbose: manager closed: read tcp 192.168.51.2:58001->217.15.17.82:25808: wsarecv: An existing connection was forcibly closed by the remote host.\n\tgithub.com/jtolio/noiseconn.(*Conn).readMsg:183\n\tgithub.com/jtolio/noiseconn.(*Conn).Read:143\n\tstorj.io/drpc/drpcwire.(*Reader).ReadPacketUsing:96\n\tstorj.io/drpc/drpcmanager.(*Manager).manageReader:223}
2023-03-27T21:22:45.829+0300	ERROR	piecestore	download failed	{Piece ID: AA32QKOJNZWTZD54RYKK246ELCVPSY3OOPD653WL6LW5AW3KWPIQ, Satellite ID: 12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs, Action: GET, Offset: 103936, Size: 163840, Remote Address: 217.15.17.82:11338, error: write tcp 192.168.51.2:58001->217.15.17.82:11338: wsasend: An existing connection was forcibly closed by the remote host., errorVerbose: write tcp 192.168.51.2:58001->217.15.17.82:11338: wsasend: An existing connection was forcibly closed by the remote host.\n\tstorj.io/drpc/drpcstream.(*Stream).rawFlushLocked:401\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:462\n\tstorj.io/common/pb.(*drpcPiecestore_DownloadStream).Send:349\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func6.2:729\n\tstorj.io/common/rpc/rpctimeout.Run.func1:22}
2023-03-27T21:22:59.250+0300	ERROR	piecestore	download failed	{Piece ID: TD3QJRMY3ZBW7IX3T5HQR22JLSL34WBCPENZNE4437EWVSCYGZQA, Satellite ID: 12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs, Action: GET, Offset: 855296, Size: 163840, Remote Address: 217.15.17.82:13268, error: manager closed: read tcp 192.168.51.2:58001->217.15.17.82:13268: wsarecv: An existing connection was forcibly closed by the remote host., errorVerbose: manager closed: read tcp 192.168.51.2:58001->217.15.17.82:13268: wsarecv: An existing connection was forcibly closed by the remote host.\n\tgithub.com/jtolio/noiseconn.(*Conn).readMsg:183\n\tgithub.com/jtolio/noiseconn.(*Conn).Read:143\n\tstorj.io/drpc/drpcwire.(*Reader).ReadPacketUsing:96\n\tstorj.io/drpc/drpcmanager.(*Manager).manageReader:223}

should i

1. disable fastopen
2. configure nodes to forward 5996 to server?
   (server is not blocking any 5996 connection)
   Thx for any help

[jtolio]

“Connection was forcibly closed by the remote host” is another way of saying “Connection reset by peer”. This isn’t a TCP_FASTOPEN issue, but is instead a Noise issue. See Connection reset by peer errors for more details.

[vladro]

sry- it should be ignored atm? (i was spending a day trying to solve it)!

[jtolio]

Yep, ignorable. Once your node gets https://review.dev.storj.io/c/storj/drpc/+/9994 it should go away.

[vladro]

thx a lot! it made me really nervous !

[snorkel]

@BrightSilence

I’ve recently noticed a wider range of problems on my network… turns out my ISP switched on intrusion detection and ddos prevention settings in my router that I had previously turned off.

Why aren’t you puting the ISP’s router in bridge mode, and use your own router as the main router? I faced problems too with the double NAT setup, untill I asked ISP to switch it’s router to bridge mode and let me use my own router with my own settings and no backdoors. Also, they were enabling some protections in their servers (like malware protection, antivirus and stuff) to protect me from malware, and I “respectfuly” asked them to stop all the scanning and protections because I use my own. Now the internet is working great again.

[snorkel]

To the topic… I juse read about these 2 implementations. I have Synology Docker nodes, started with sudo su only. The only user is the admin one. I understand that for noise implementation, I don’t have to do anything. Should I setup the TCP Fast Open? Is it production ready? Is there a guide for Docker nodes?
Also @BrightSilence, I checked my sysctl.conf and the net.core option that I set last year didn’t disappeared from there, and I updated DSM like 2-3 times since than. I’m not sure about a major update though, like from 6 to 7. I did only uptades for DSM 7.

[snorkel]

Is this error related to noise? I see jtolio… github… noiseconn in there. Is noise already active on the network? Is the error normal? I see many of them, like 380 in 22 days, for a 4TB node.

2023-03-29T20:28:39.421873561Z	stdout	2023-03-29T20:28:39.421Z ERROR piecestore upload failed {"Process": "storagenode", "Piece ID": "G5L33ZSOFRYSIBOTEJQC5RFNUTNHMAT4PCLW3YRY5V5K2UPBF76A", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT", "error": "manager closed: unexpected EOF", "errorVerbose": "manager closed: unexpected EOF\n\tgithub.com/jtolio/noiseconn.(*Conn).readMsg:197\n\tgithub.com/jtolio/noiseconn.(*Conn).Read:143\n\tstorj.io/drpc/drpcwire.(*Reader).ReadPacketUsing:96\n\tstorj.io/drpc/drpcmanager.(*Manager).manageReader:223", "Size": 524288, "Remote Address": "172.17.0.1:33498"}

[Alexey]

See Connection reset by peer errors

[snorkel]

I see. Noise is out and that’s why these errors appear. Anyway, mine are “unexpected EOF”, not connection reset, but maybe they are normal too.

[Alexey]

This is the same deal, as with long tail cancelation - the cancel may happen in any time, and the error throws from the line, where is it happened.

[BrightSilence]

@snorkel

Also @BrightSilence, I checked my sysctl.conf and the net.core option that I set last year didn’t disappeared from there, and I updated DSM like 2-3 times since than. I’m not sure about a major update though, like from 6 to 7. I did only uptades for DSM 7.

It isn’t overwritten with every update, but with larger updates it can be. DSM 7.2 is coming up, which is a larger feature update (that also change the docker implementation to a new “container manager” interface. So reason to be careful with this update to begin with. But chances are this one will impact that file as well. I’d say just keep an eye on it after any big update. Or switch to a startup script that you can manage in the DSM interface itself to set these settings. That way you know it will never be overwritten.

I just use task scheduler to run these on boot:

sysctl -w net.core.rmem_max=2500000
sysctl -w net.ipv4.tcp_fastopen=3

[snorkel]

@BrightSilence
This looks good? I made them run as root…

[BrightSilence]

Yep, looks good. Though I recommend setting up the email notification only if there’s an error. (check both boxes and fill in your email address) There shouldn’t be one though, but it’s a nice extra check.

[snorkel]

@BrightSilence
DSM updates deleted some of my sheduled tasks, also. I had SMART scheduled and after an update, the task was gone. So, we should keep an eye on the tasks aswell.

[BrightSilence]

They redesigned how that worked in DSM7. Those tasks are set by other parts of the interface if I remember correctly. Your custom tasks should be safe from this.