How important is send buffer size?

I recently started 4 new nodes using my TrueNAS. The buffer related message looks like this:

INFO	failed to sufficiently increase send buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See for details.

According to the github link this is about high-bandwidth connections. Running a number of storj nodes at home sharing a 100 mbit internet connection is obviously not a high-bandwidth thing.

So the question is does it matter or not?

Did you follow the Quick setup guide in official docs? You need to increase some buffers.

1 Like

I am knowing how to set the buffer size for linux but TrueNAS doesn’t want users to change such things directly. It is possible but not recommended. So I would prefer to ignore this message if the possible gain is negligible.

Is it negligible?

lot of consumer network cards like realtek, dont have over 128k sent buffer, as i seen only intell have up to 2048k buffer

Network card is 10 GBe Mellanox Connect-X 3, so not a consumer card. But as far as I know the buffer we are talking about is a software buffer within the os kernel.

I think this problem only affects Quic, and they said Quic is not crucial, not used by a big percent of clients. We are many with Quic missconfigured on some nodes. Not a big deal. I have 2 on the oldest hardware.

1 Like

It seems that I have that message on my Ubuntu Server node too. I think we need to increase both send and receive buffers, as QUIC docs say:

We need to use these comms for Linux systems:

sudo echo "net.core.rmem_max=2500000" >> /etc/sysctl.conf

sudo echo "net.core.wmem_max=2500000" >> /etc/sysctl.conf

sudo sysctl -w net.core.rmem_max=2500000

sudo sysctl -w net.core.wmem_max=2500000

Can you check the logs after you set these and reboot the machine?

I believe TrueNAS Scale is non-BSD type and TrusNAS Core is BSD type. So, I don’t know what version you use, but if it’s Core, you need to use the proper commands from those docs.

I just realised it is the send buffer. Why should a SNO care about the send buffer? Egress doesn’t matter much since latest payment cut.

I would rather throttle egress if it helped ingress… :wink:

Because SNO should please the customers using the network and not throttle them. There will be no data to store if there will be no happy customers.
Sometimes I’m just amazed by how people are thinking.


I am thinking as if it was a business. Storj is optimizing their stuff (cutting egress payment). I am optimizing my stuff. You get what you pay.

1 Like

Well, I guess if this will affect your 95th percentile or CIR you have with the carrier, then fair enough. But I doubt this for obvious reasons. Thus now, you are paying your ISP for a bandwidth you are not using. Smart.

1 Like

My internet is a poor 100/40 Mbit copper line. Nothing else available here. Sometimes upload is close to saturation. So in fact it could make sense to reduce egress.

Nope. It would not. I have /20. Storagenode is allowed to consume all of it. Why not? Unused bandwidth – money down the drain.

If saturating upstream affects your other applications – then throttling anything is definitely is not the solution. You need some sort of smart queue management/fair queuing protocol on the upstream side.

Actually I have a pfSense box which probably could do this. But I am still learning to use all its features.

Oh great, then you already have everything needed: pfSense® software Configuration Recipes — Configuring CoDel Limiters for Bufferbloat | pfSense Documentation


Thanks @arrogantrabbit. I will give it a try. This is where we come from…

This is not bad. Perhaps the issue is elsewhere.

Assuming you measured it from wired connection. The unloaded latency is quite high, and wifi may have its own qos mechanisms.

There is no real issue. Upload success rate for my nodes is 99.x%, download success is like 95%.

I was just asking how imortant this buffer setting is. I guess I could improve download percentage a little bit…

I figured out how to set the buffer size in the TrueNAS Scale GUI: System Settings->Advanced->Sysctl

Can’t you SSH to it and do the usual CLI commands?