How important is send buffer size?

Node Operators troubleshooting
1

I recently started 4 new nodes using my TrueNAS. The buffer related message looks like this:

INFO	failed to sufficiently increase send buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See https://github.com/quic-go/quic-go/wiki/UDP-Buffer-Sizes for details.

According to the github link this is about high-bandwidth connections. Running a number of storj nodes at home sharing a 100 mbit internet connection is obviously not a high-bandwidth thing.

So the question is does it matter or not?

2

Did you follow the Quick setup guide in official docs? You need to increase some buffers.

1 Like
3

I am knowing how to set the buffer size for linux but TrueNAS doesn’t want users to change such things directly. It is possible but not recommended. So I would prefer to ignore this message if the possible gain is negligible.

Is it negligible?

4

lot of consumer network cards like realtek, dont have over 128k sent buffer, as i seen only intell have up to 2048k buffer

5

Network card is 10 GBe Mellanox Connect-X 3, so not a consumer card. But as far as I know the buffer we are talking about is a software buffer within the os kernel.

6

I think this problem only affects Quic, and they said Quic is not crucial, not used by a big percent of clients. We are many with Quic missconfigured on some nodes. Not a big deal. I have 2 on the oldest hardware.

1 Like
7

It seems that I have that message on my Ubuntu Server node too. I think we need to increase both send and receive buffers, as QUIC docs say:
https://github.com/quic-go/quic-go/wiki/UDP-Buffer-Sizes

We need to use these comms for Linux systems:

sudo echo "net.core.rmem_max=2500000" >> /etc/sysctl.conf

sudo echo "net.core.wmem_max=2500000" >> /etc/sysctl.conf

sudo sysctl -w net.core.rmem_max=2500000

sudo sysctl -w net.core.wmem_max=2500000

Can you check the logs after you set these and reboot the machine?

I believe TrueNAS Scale is non-BSD type and TrusNAS Core is BSD type. So, I don’t know what version you use, but if it’s Core, you need to use the proper commands from those docs.

8

I just realised it is the send buffer. Why should a SNO care about the send buffer? Egress doesn’t matter much since latest payment cut.

I would rather throttle egress if it helped ingress… :wink:

9

Because SNO should please the customers using the network and not throttle them. There will be no data to store if there will be no happy customers.
Sometimes I’m just amazed by how people are thinking.

3 Likes
10

I am thinking as if it was a business. Storj is optimizing their stuff (cutting egress payment). I am optimizing my stuff. You get what you pay.

1 Like
11

Well, I guess if this will affect your 95th percentile or CIR you have with the carrier, then fair enough. But I doubt this for obvious reasons. Thus now, you are paying your ISP for a bandwidth you are not using. Smart.

1 Like
12

My internet is a poor 100/40 Mbit copper line. Nothing else available here. Sometimes upload is close to saturation. So in fact it could make sense to reduce egress.

13

Nope. It would not. I have /20. Storagenode is allowed to consume all of it. Why not? Unused bandwidth – money down the drain.

If saturating upstream affects your other applications – then throttling anything is definitely is not the solution. You need some sort of smart queue management/fair queuing protocol on the upstream side.

14

Actually I have a pfSense box which probably could do this. But I am still learning to use all its features.

15

Oh great, then you already have everything needed: pfSense® software Configuration Recipes — Configuring CoDel Limiters for Bufferbloat | pfSense Documentation

2 Likes
16

Thanks @arrogantrabbit. I will give it a try. This is where we come from…

bufferbloat1
bufferbloat11766×1332 325 KB

17

This is not bad. Perhaps the issue is elsewhere.

Assuming you measured it from wired connection. The unloaded latency is quite high, and wifi may have its own qos mechanisms.

18

There is no real issue. Upload success rate for my nodes is 99.x%, download success is like 95%.

I was just asking how imortant this buffer setting is. I guess I could improve download percentage a little bit…

19

I figured out how to set the buffer size in the TrueNAS Scale GUI: System Settings->Advanced->Sysctl

20

Can’t you SSH to it and do the usual CLI commands?