Are you sure it’ll not hurt audits/online score when tc starts dropping packets? Did you implement it? What are your results?
No; TCP is designed to use packet loss as a congestion signal and respond by backing off and retransmitting.
Yes, I did, it works rather well – credits get accumulated when load is lower, then spent when usage picks up. It’s pretty cool. Most of the time went into designign recovery procedure (ie if instance dies following bad OS update, I woudl need to recover it from backup, and apply current statistics and quotas (that are stored to external cloud storage every 5 min for this purpose).
My internet lines are at the upload limit atm and I see online scores going down but audit is still ok.
You probably have bufferbloat - important packets sit in queue forever and timeout.
Consider some sort of queue management – like fq_codel – on your wan link.
If you use Ubiquiti gateways (and why would you use anything else? 
) it’s built in, called SmartQueues.
PfSense has it also built in and OpenWRT has packages (sqm-scripts and luci-app-sqm)
I have testet a lot with pfsense settings but no sort of queue management can magically overcome the missing bandwidth. I settled with just prioritizing the none storj traffic.
Then keep testing, until you make it work. because fq_codel + HTB (which ubiquity is using) is absolutely magically overcomes missing bandwidth – latency goes back to below 10ms while you are fully saturating connection.
I used to have 400down/12up (yes, twelve) internet for a while, it was saturated all the time – bandwidth graph was a straight line for months (I like doing backups) – and this made it imperceptible.. I tried it on OpenWRT, got blown away, and bought USG3. This was my gateway drug to ubiquity ecosystem. Yes, it’s magic. (ok, math. Same thing)
I don’t know how to do it on pfSense – I tried it, and I absolutely despise the interface, I refused to use it.
This is a half-measure. F in FQ mean Fair. And it’s dynamic, unlike whatever static priorities you assign, manually and have to maintain
Edit, curiously, it’s (been rnamed?) now “Flow Queueing” not “Fair Queuing”: RFC 8290: The Flow Queue CoDel Packet Scheduler and Active Queue Management Algorithm | RFC Editor
FQ-CoDel is a hybrid of DRR [DRR] and CoDel [RFC8289], with an optimisation for sparse flows similar to Shortest Queue First (SQF) [SQF] and DRR++ [DRRPP]. We call this "flow queueing" rather than "fair queueing", as flows that build a queue are treated differently from flows that do not.
Even though the document still talks a lot about fairness.
I have acceptable latency for none storj traffic and internet lines are already saturated with current pfsense settings. A “fair” approach can’t work in my opinion when the overload is massive and 90+ percent of all traffic is storj.
What’s your upload bandwidth, and how many concurrent outgoing Storj transfers do you have when your online score starts dropping?
You probably want to prioritize traffic instead of hard limiting it, but I do not know if pfSense can do it. I use pfSense in some places, but never had to use its shaping. Here’s how I do it on my Linux router for my backup connection (which only has about 11mbps upload).
ext0.4093 is the interface that connects to that ISP. I skipped some classes that are specific for my uses.
tc qdisc del dev ext0.4093 root
tc qdisc add dev ext0.4093 root handle 10: htb default 106
tc class add dev ext0.4093 parent 10:0 classid 10:1 htb rate 10mbit ceil 11mbit burst 64k cburst 2k
#TCP ACK
tc class add dev ext0.4093 parent 10:1 classid 10:101 htb rate 1mbit ceil 11mbit burst 64k cburst 2k prio 1
tc qdisc add dev ext0.4093 parent 10:101 fq_codel
#ICMP
tc class add dev ext0.4093 parent 10:1 classid 10:103 htb rate 1mbit ceil 11mbit burst 64k cburst 2k prio 3
tc qdisc add dev ext0.4093 parent 10:103 fq_codel
#udp
tc class add dev ext0.4093 parent 10:1 classid 10:105 htb rate 1mbit ceil 11mbit burst 64k cburst 2k prio 5
tc qdisc add dev ext0.4093 parent 10:105 fq_codel
#default
tc class add dev ext0.4093 parent 10:1 classid 10:106 htb rate 1mbit ceil 11mbit burst 64k cburst 2k prio 6
tc qdisc add dev ext0.4093 parent 10:106 fq_codel
#Storj norm
tc class add dev ext0.4093 parent 10:1 classid 10:108 htb rate 1mbit ceil 11mbit burst 64k cburst 2k prio 8
tc qdisc add dev ext0.4093 parent 10:108 fq_codel
#filters
tc filter add dev ext0.4093 parent 10:0 prio 1 protocol ip u32
#TCP ACK
# IHL = 5 (no options)
# length <=63B
# ACK flag and nothing else
tc filter add dev ext0.4093 parent 10:0 prio 1 protocol ip u32 \
match ip protocol 6 0xff \
match u8 0x05 0x0f at 0 \
match u16 0x0000 0xffc0 at 2 \
match u8 0x10 0xff at 33 \
flowid 10:101
#ICMP
tc filter add dev ext0.4093 parent 10:0 prio 1 protocol ip u32 \
match ip protocol 0x01 FF \
flowid 10:103
#Storj norm
#TCP/UDP
#src port 42000
tc filter add dev ext0.4093 parent 10:0 prio 1 protocol ip u32 \
match ip protocol 6 0xff \
match ip sport 42000 FFFF \
flowid 10:108
tc filter add dev ext0.4093 parent 10:0 prio 1 protocol ip u32 \
match ip protocol 0x11 0xff \
match ip sport 42000 FFFF \
flowid 10:108
#UDP
tc filter add dev ext0.4093 parent 10:0 prio 1 protocol ip u32 \
match ip protocol 0x11 FF \
flowid 10:104
This makes TCP ACK packets the highest priority, followed by ICMP and then UDP packets. Storj node gets the lowest priority, so it can use whatever bandwidth is left from everything else.
The important part is to set the bandwidth limit slightly lower than what you get from the ISP.
It seems to work fine, the 10mbps upload is saturated but I can use other stuff without problems.
Prioritizing works better in this case IMO. The reason is that, say, a video chat is more important that the Stroj traffic, so it should have higher priority. With just “fair queueing”, when stuff gets overloaded, everything slows down, but with priorities, the things that are more important remain responsive, while the less important things slow down.
Well I think he wants hard limiting it, because of free quota on oracle.
But it seems that heavy repair trafic is gone anyway, it falls as a stone 
If its only the node traffic going there then it is very easy - just a single class (I am sure there are other QDs that can do this, but I am used to htb, so I use it everywhere).
I think you’re right - but it was fun while it lasted! 
This is not entirely true. fq_codel already gives preference to sparse flows – those that don’t maintain a standing queue. Bulk transfers normally have separate queues, so their backlog doesn’t sit ahead of every call packet. codel controls persistent queueing delay within those queues. This keeps video call responsive on a saturated link without explicitly prioritizing the application, while fqcodel controls the bottlenecked queue. This also works for long-running connections. A flow can return to the preferred queue list after its queue drains.
From RFC 8290, paragraph 1.3, in the end:
.. it will effectively get priority over queues that do not empty out each round (a minor caveat is required here to protect against starvation, see below). Exactly how little data a flow has to send to keep its queue in this state is somewhat difficult to reason about, because it depends on both the egress link speed and the number of concurrent flows. However, in practice, many things that are beneficial to have prioritised for typical internet use (ACKs, DNS lookups, interactive Secure Shell (SSH), HTTP requests, Voice over IP (VoIP)) tend to fall in this category, which is why FQ-CoDel performs so well for many practical applications. However, the implicitness of the prioritisation means that for applications that require guaranteed priority (for instance, multiplexing the network control plane over the network itself), explicit classification is still needed.
But I guess if storj traffic results in a lot of flows this can leave a video insufficient bandwidth, which will make it backlogged persistently and lose the sparse-flow treatment then explicit classification will help. When storj can do that on my connection – that would be a happy day for everyone 
But I don’t’ like manually classifying things unless I have it, and so far – I did not need to, even on slower connection, that was always saturated. I’m not sure if Unifi does some other magic in addition to fq_codel though. They totally might.
Good to know. Storj traffic does not saturate a normal connection, but it would saturate my backup connection right now (egress is about 15mbps), that’s why I use priorities on that connection. Priorities also help in case of similar looking streams. For example, if I’m uploading a large file somewhere, it should take priority over Storj, even though both would look similar as “bulk transfers”, but a video call should take priority over both.
Was it already with Storj traffic?
I wonder. fq_codel considers each connection separately, and Storj has a large number of them. It cannot really act on short-lived connections, and most of connections Storj makes are short-lived. And even if a connection uploads/downloads more data, these connections on each own would be often pretty slow too, because the customer would also have to spread their traffic across many nodes—hence potentially being treated preferentially by fq_codel.
This is a similar case to bittorrent’s LEDBAT traffic shaping. This paper describing the issue is even cited by fq_codel’s RFC8290 acknowledging the issue.
So quite curious now how well does fq_codel actually work with Storj…
No, I think it was before storj, but with bit-torrent.
Codel cannot, but fq_codel can. From the same rfc:
Additionally, flow isolation radically improves the transient behaviour of the network when traffic or link characteristics change (e.g., when new flows start up or the link bandwidth changes); while CoDel itself can take a while to respond, FQ-CoDel reacts almost immediately.
I’m not sure it’s the same issue with LEBAT, I need to read that carefully, it seems it has to do AQM keeping delays below the level that makes LEDBAT back off.