I’m running 10 storagenodes on my network, which saturate my upload bandwidth, when not limited. Since this causes lag spikes, I resorted to limit the upload bandwidth of my ethernet interface of my Storj server to a little bit less than my maximum upload bandwidth. However, I noticed that the ingress (which I haven’t limited) is less than half of what it is, when no upload/egress limit is in place. I tested this multiple times and see this phenomenon every time. Is there something I can do to ensure, that I get the usual egress when limiting my ingress?
The Storj server is running on Ubuntu Server 24.04.3
I use “tc” to limit the upload bandwidth. The script I am using:
#!/bin/bash
# limit-egress.sh
# Example: ./limit-egress.sh enp4s0 25600
# Only limiting traffic which is not routed to 192.168.0.0/24 and 192.168.178.0/24
IFACE="$1"
RATE="$2"
if [ -z "$IFACE" ] || [ -z "$RATE" ]; then
echo "Usage: $0 <interface> <rate-in-kbit>"
exit 1
fi
if [ "$EUID" -ne 0 ]; then
echo "Dieses Script muss mit Root-Rechten ausgeführt werden (sudo)."
exit 1
fi
# Remove old qdiscs
tc qdisc del dev "$IFACE" root 2>/dev/null
# Add root qdisc
tc qdisc add dev "$IFACE" root handle 1: htb default 30
# Class for limited traffic
tc class add dev "$IFACE" parent 1: classid 1:1 htb rate "${RATE}kbit" ceil "${RATE}kbit"
# Everything that does not go into the internal network, in class 1:1
# First exclude the internal networks (RETURN)
tc filter add dev "$IFACE" protocol ip parent 1:0 prio 1 u32 match ip dst 192.168.0.0/24 action pass
tc filter add dev "$IFACE" protocol ip parent 1:0 prio 2 u32 match ip dst 192.168.178.0/24 action pass
# Everything else
tc filter add dev "$IFACE" protocol ip parent 1:0 prio 10 u32 match ip dst 0.0.0.0/0 flowid 1:1
I followed your posted tutorial and set the limits to 95% of my maximum upload/download bandwidth. Unfortunately, I see no difference whatsoever. The lagspikes are still present.
I measured my bandwidth with no nodes running (and no other relevant traffic) and used that to determine the values for the limiters.
CPU usage is sitting at 10% according to the pfSense dashboard. It’s an Intel N100. Should be enough for my small network.
Waveform bufferbloat test shows grade C at the moment. Note: All following values are from the test while all my nodes were running.
Unloaded: 28ms
Download Active: +96ms
Upload Active: +114ms
I have set the limits to 50% now. I get a B now (again with nodes running):
Unloaded: 25ms
Download Active: 2ms
Upload Active: 52 ms
I talked to my ISP last week about this problem. They pointed me to my saturated uplink and told me nothing they can do, if I satured it. They also told me, that there is no limiter or throttle on their side (I specifically asked).
No wireless link. All involved clients are on LAN.
What confuses me is the fact, that the Storj Server still uses 100% of my upload bandwith, even thought the limiter is set to 50%.
Something does not work then – but I don’t have experience with pfsense to point out what could be wrong. I played with a plugin on OpenWRT, it worked there, and now I use ubiquiti gateways that have it built-in (technically they have fq_codel + HTB)
(IIRC there was something with firewall rules “match” vs “pass”, their placement (shall be very top rule) and direction, and make sure you turn off all other QoS.)
Well, you need both – fq (to create several “fair” queues") and CoDel – which is an “active queue management” that absolutely does drop packets. This is how TCP is supposed to work, moderate traffic by dropping packets.
Not dropping packets is what poorly behaving equipment does. and this is precisely what causes bufferbloat in the first place.
This ia by design: When CoDel used along with fair queuing, packets will be getting dropped from the most obnoxous larget flows – exactly what you want.
Just using codel is not enough, because indeed, it may drop packets you don’t want dropped.
Often people say “codel” when they mean “fq_codel” and this causes confusion.
