If you want to be hard core about it, my favorite router is openbsd. It can run on very minimal hardware, can handle an entire university’s worth of traffic, and just in general is very secure.
Where PFSense on the cluster comes in nice is when you need multiple people managing the router (i.e. adding a dns entry or a ssl cert). With obsd and pf, i’m afraid someone will make a mistake.
Two notes-
L2Arc takes some memory too, so that’s a factor you may need to attribute for.
I prefer to run my pfsense instances without any plugins aside from either zabbix or something to export to influxDB/prometheus. BandwidthD, netdata, ntopng, etc… they all have to touch the disk to save things and they build aggregations some times that may be taking RAM or CPU cycles away from routing- so I do all those, essentially, meta aggregations elsewhere to prevent from bogging the devices primary function. I’ve seen SG-3100’s reboot due to watchdog and it turns out it was just bandwidthD or suracatta chugging down and bottlenecking the system to a point the watchdog timer expired and rebooted it.
yeah i completely agree running other stuff in pfsense is most likely just a bad idea… atleast in many cases… but i will assume most of the sensible features it has is a safe bet…
and in regard to the memory… granted the server hasn’t been running for nearly as long… but to just kick it a bit extra, i spun up a couple of windows vm’s on top, after i put the l2arc back online.
now it’s at 91.70% and still dropping… yes i know l2arc will require some allocation to keep track of it… but from what i’ve seen thus far it’s not a problem… had it up to just shy of 1tb and running for weeks… and ofc took like 6-8 weeks to get there…
but i really like it and it being a fast ssd at many things… not perfect in any aspect but good in all aspects… so when it’s really been saturated over a few months, everything that has been run will still be there… and be snappy as on an ssd… sure something minute might be needed from the hdd’s but 99% of the time when it’s vm’s, scrubs, programs, browsers, streaming content, iso’s… even the filewalker or storj databases end up in the l2arc…
and if something needs memory, the memory just drops it immediately because the data is already in the l2arc and if you change what you are doing… it can reload the entire allowed memory capacity with it immediately…
thus far it seems to work great… it was also why i kinda just let it be crashed for so long… to get a sense of what i was getting out of it… and i can say i did notice a lot of stuff being affected by my l2arc being down.
sure it’s not a perfect solution, but its much cheaper than tons of ram… and seem to get kinda close…
ofc much less will most likely do nearly as well… but figured it would be a fun experiment.
and with such a huge l2arc it will have a memory like an elephant.
if it’s been used this year… it’s most likely there 
atleast for my tiny system
and now in a few months persistent l2arc… which should be a game changer
Got a huge amount of repair egress today. I guess someone big blew it up
28GB was what I got yesterday, now
Just a few days ago I had similar repair egress- was yours greater than these?
the ratio seems about the same, on the 18th my repair to regular egress ratio is even lower… wprst than 5 to 1
but egress is known to be quite random…
my egress haven’t been over my repair egress since the 5th of December.
and may dashboard claims i’m over 8% downtime
the ingress looks low because i go two nodes on that connection and ingress is split and both where down for like 4-5 hours i think yesterday.
wasn’t the memory thing with pfsense, now testing out checksum offloading which might be the cause… i’m sure the running out of memory thing didn’t help…
and memory usage is slowly dropping… sadly i need to do a reboot after xmas to try and fix my netdata… but memory utilization looks stable with l2arc online, even if kinda high… but it’s usually like that… now i’m just more paranoid about it hitting 99.9%… but doesn’t seem to do want to do that…
ofc makes sense if it cannot offload the ARC the ram would fill and not respond to incoming allocations.
your graphs are so much more stable lol, mine makes me sad.
no comments about the 17th lol wooops
the memory doesn’t look to bad because it’s weekly avg
worst point is less than 500MiB free of 50.6GiB aka 48GB
or is it the other way around…50.6GiB the number formerly known as 48 Gigabytes, before all the hipsters lost their minds about not understanding data bandwidth math…
add a data lane and you basically double the throughput even if it’s more like an octave or something… it doesn’t work in decimal for santas sake, gosh damnit!!!
but oh no… now they want it all in decimal numbers… and it’s not like they can make up their own numbers… oh no they rename already existing numbers ofc… makes perfect sense… why make new names for stuff you can just steal… deep breaths i blame Mr T
pretty sure it’s the other way around… 48GiB is 48 GibiBytes which was what was called 48 GigaBytes, but is now 50.6GigaBytes
i do love how proxmox seems to just jump around in those terms…
and then ofc to my understanding the former should be 48GiB because that was what 48GB is called in the new system… i mean windows will show it as flat 48… proxmox can’t even agree with itself what to call it… and i can’t even figure out what number it’s trying to say…
even tho i’ve been working with pretty nitty gritty computer stuff for decades lol
and how is storage set in GiB when that should be in Gigabytes, since it’s storage and not affected by the bandwidth math in the same way as ram is… ram should be in binary because else the bandwidth math doesn’t make sense when calculating lanes / data transfer rates…
granted i can accept that storage might not have much purpose being in octal, just for doing the math in decimal, and sure i guess RAM could be that way to… if we found some numbers that made sense on either end of the engineering / utilization spectrum… but no…
now it’s just chaos and it seems the numbers are wrong more often than they are right…
and all of them are just … decimal numbers…
hey i got 47.21GiB or 48GiB or maybe 50.69G whatever that is and i bought them as 48GB heheh facepalm
oh will tis the season to be jolly.
cheers
just got to really looking at those numbers and … gets my blood boiling…
@Pentium100
pretty sure i figured out the pfsense weirdness…
i was using emulated drivers and i hadn’t turned checksum offloading to off…
it magically fixed every single issue… the internet speed being slow, the routing in pfsense stopped working, haven’t switch back to emulated drivers…
they said it would fix the virtio nic drivers, on proxmox wiki… and it did…
have run 90MB/s up and 90MB/s down at the same time, tho not over the internet, but through the router…
it’s so nice when trying to fix one problem, ends up fixing multiple issues.
kinda makes me wonder if it always just was that checksum offloading being enabled.
but really happy i finally found a fix.
now i guess i can start turning off all the STP stuff i got turned on everywhere…
the internet speed seems a bit on the low side still tho… ofc pushing around 90MByte/s in and 90Mbyte/s out on a 1gbit nic might cause that, which was more a long the lines of what i would have expected.
a good patch solution and since the fiber if just one strand… then it cannot do full duplex and thus then maximum bandwidth is 1gbit, while the lan is 2gbit and thus i can more or less forced it twice over a cable without a ton of loss.
IT WERKS!! IT’S ALIVE, IT’S ALIVE… so far only for a couple of days, but had like 5-7 crashes in total over 5 days… so seems like it’s finally fixed…
and the l2arc back online seems to have fixed the ram issue…
can’t believe everything is working… so weird
ofc i didn’t get my isp to give me multiple ip addresses with was the whole cause of all this restructuring of my setup… lol
Oh yeah, I think even pfsense recommends turning offloading off for virtio.
Sure it can. With one fiber, two different wavelengths are used for the directions, you can even see them printed on the SFP transceiver. For example: Tx1310/R1550 (fun fact - the transceiver on the other end of the cable has to have these wavelengths reversed).
A single fiber can also do 10G, you just need different transceivers.
yeah, but aren’t they much more expensive and most often with SFP it has a dual cable setup, because in most cases for short runs it’s cheaper… so i sort of assume that they used an efficient solution… the fiber might be able to support faster speeds… but the gear seems to be around the 1gbit mark, must be some sort of local nexus nearby… which reroutes the network… the weird thing is that these guys could give me much higher internet speed… but it was basically at the push of a button…
it was setup and running like 12 hours or less from when i ordered it… how can some of my isp’s only offer 500mbit syncronous while another offer 1gbit… using the same hardware and clearly didn’t have to do anything to change it…
kinda makes it seem like most of them just artificially set the speed…
you really think my 50$ a month 1 gbit fiber internet can do full duplex…? i suppose i could test it,
wouldn’t it make more sense that it was the cheaper kind that can’t do full duplex…
but yeah i was wrong saying it can’t, it’s just very complex and not much used to my understanding… ofc on very long lines i’m sure the price ratio to fiber costs will make it more sensible.
very limited knowledge on fiber, have been trying to get more familiar with it for a while tho.
such a complex subject.
would be nice if i can get 10gbit + tho… not that i need it… would need a lan 40gbit first or so
My 600/600mbit movistar fiber connections in Barcelona are routinely fully saturated.
For example right now
eno1: 77.32 MB/s 5.80 MB/s 83.12 MB/s
This is just one of the modems, where we are currently pulling approx 35TB zfs snapshot backup from a server at hetzner.
Note - evidently Movistar promised all 600/600 fiber would be upgraded to gigabit on Jan 6th, it appears that’s already happened because 77.32MB/sec is for sure higher than 600mbit.
Also i’m pretty sure i only pay 35 euro/month for 600mbit, then an extra 5/month for it to be symetric, but don’t quote me on this.
Maybe it can, maybe it can’t, depending on how you ISP does the shaping, but 1G SFPs support full duplex. One fiber vs two is a tradeoff - one fiber (BiDi) SFPs are more expensive, but you only need one fiber - cheaper for longer runs, while two fiber SFPs are cheaper (especially multimode, which cannot be used for long distances), but you need two fibers.
I should also mention that origionally I had 4x300mbit movistar’s, which all shared one fiber. When they upgraded them to 600mbit, they ran an individual fiber for each.
600mbit/s = 75MB/sec so I’m not sure 77.34MB/s is really “higher” than 600mbit/s
been trying to test my connection… but that has proven problematic… doesn’t seem like a lot of people have 1gbit bandwidth to spare… lol tried speed tests to like 15 + different servers, even multiple ones at the same time, and still it’s a bit random…
was going to note that 600mbit thing…
@joesmoe i paid more for my 500mbit which was initially 400mbit i think, upgrading to 1gbit got cheaper, but it’s not really a surprise that everything here is expensive
really wanted to test the connection… but thus far i’ve just given up… when you can download with 85mb/s and upload with also like 600mbit then i guess it has to be full duplex… but pretty sure if it was able to do that it wouldn’t go through the network i get setup locally… so 1gbit total will be my limit for now… even if my internet might be able to do more…
might try and test the internet more some other time… but for now it seems like a waste of time since i’m unable to saturate it no matter what i try of online services and it’s not the connection that’s saturated… i’ve had it to 90mb/s and i think that was actually the write speed of the hdd’s that was being downloaded to.
my old cpu’s in the server is certainly also getting a bit of work done when running at near 1gbit … pfsense will take like 10% of the total cpu time across both cpu’s, but that’s not to bad… i was fearing worse.
I believe there’s some EU 10G endpoints there.
Appears that the new year is off to a pretty nice start. Download traffic is just plugging right along, on par with last month. Upload traffic has slowed once again, but hey, I can’t be too picky, and I’m ever hopeful that new ingress from some of these newly announced partnerships starts picking up in the near future.
Node 1:
Node 2:
Node 3:
Excel Tracker with everything combined (daily basis, not all inclusive):
Excel Tracker with everything combined (monthly basis, all inclusive):
Well your excel sure is impressive but wouldn’t it be a lot less work to use the prometheus exporter and make those charts with grafana?
