Got a huge amount of repair egress today. I guess someone big blew it up
28GB was what I got yesterday, now
Just a few days ago I had similar repair egress- was yours greater than these?
the ratio seems about the same, on the 18th my repair to regular egress ratio is even lower… wprst than 5 to 1
but egress is known to be quite random…
my egress haven’t been over my repair egress since the 5th of December.
and may dashboard claims i’m over 8% downtime
the ingress looks low because i go two nodes on that connection and ingress is split and both where down for like 4-5 hours i think yesterday.
but been a few days since i had any extended downtime… so i guess now my numbers are improving again… pretty close to having it all solved i think…
wasn’t the memory thing with pfsense, now testing out checksum offloading which might be the cause… i’m sure the running out of memory thing didn’t help…
and memory usage is slowly dropping… sadly i need to do a reboot after xmas to try and fix my netdata… but memory utilization looks stable with l2arc online, even if kinda high… but it’s usually like that… now i’m just more paranoid about it hitting 99.9%… but doesn’t seem to do want to do that…
ofc makes sense if it cannot offload the ARC the ram would fill and not respond to incoming allocations.
your graphs are so much more stable lol, mine makes me sad.
no comments about the 17th lol wooops
the memory doesn’t look to bad because it’s weekly avg
worst point is less than 500MiB free of 50.6GiB aka 48GB
or is it the other way around…50.6GiB the number formerly known as 48 Gigabytes, before all the hipsters lost their minds about not understanding data bandwidth math…
add a data lane and you basically double the throughput even if it’s more like an octave or something… it doesn’t work in decimal for santas sake, gosh damnit!!!
but oh no… now they want it all in decimal numbers… and it’s not like they can make up their own numbers… oh no they rename already existing numbers ofc… makes perfect sense… why make new names for stuff you can just steal… deep breaths i blame Mr T
pretty sure it’s the other way around… 48GiB is 48 GibiBytes which was what was called 48 GigaBytes, but is now 50.6GigaBytes
i do love how proxmox seems to just jump around in those terms…
then 2 graphs down…
and then ofc to my understanding the former should be 48GiB because that was what 48GB is called in the new system… i mean windows will show it as flat 48… proxmox can’t even agree with itself what to call it… and i can’t even figure out what number it’s trying to say…
even tho i’ve been working with pretty nitty gritty computer stuff for decades lol
and how is storage set in GiB when that should be in Gigabytes, since it’s storage and not affected by the bandwidth math in the same way as ram is… ram should be in binary because else the bandwidth math doesn’t make sense when calculating lanes / data transfer rates…
granted i can accept that storage might not have much purpose being in octal, just for doing the math in decimal, and sure i guess RAM could be that way to… if we found some numbers that made sense on either end of the engineering / utilization spectrum… but no…
now it’s just chaos and it seems the numbers are wrong more often than they are right…
and all of them are just … decimal numbers…
hey i got 47.21GiB or 48GiB or maybe 50.69G whatever that is and i bought them as 48GB heheh facepalm
oh will tis the season to be jolly.
cheers
just got to really looking at those numbers and … gets my blood boiling…
@Pentium100
pretty sure i figured out the pfsense weirdness…
i was using emulated drivers and i hadn’t turned checksum offloading to off…
it magically fixed every single issue… the internet speed being slow, the routing in pfsense stopped working, haven’t switch back to emulated drivers…
they said it would fix the virtio nic drivers, on proxmox wiki… and it did…
have run 90MB/s up and 90MB/s down at the same time, tho not over the internet, but through the router…
it’s so nice when trying to fix one problem, ends up fixing multiple issues.
kinda makes me wonder if it always just was that checksum offloading being enabled.
but really happy i finally found a fix.
now i guess i can start turning off all the STP stuff i got turned on everywhere… 
the internet speed seems a bit on the low side still tho… ofc pushing around 90MByte/s in and 90Mbyte/s out on a 1gbit nic might cause that, which was more a long the lines of what i would have expected.
a good patch solution and since the fiber if just one strand… then it cannot do full duplex and thus then maximum bandwidth is 1gbit, while the lan is 2gbit and thus i can more or less forced it twice over a cable without a ton of loss.
IT WERKS!! IT’S ALIVE, IT’S ALIVE… so far only for a couple of days, but had like 5-7 crashes in total over 5 days… so seems like it’s finally fixed…
and the l2arc back online seems to have fixed the ram issue…
can’t believe everything is working… so weird
ofc i didn’t get my isp to give me multiple ip addresses with was the whole cause of all this restructuring of my setup… lol
Oh yeah, I think even pfsense recommends turning offloading off for virtio.
Sure it can. With one fiber, two different wavelengths are used for the directions, you can even see them printed on the SFP transceiver. For example: Tx1310/R1550 (fun fact - the transceiver on the other end of the cable has to have these wavelengths reversed).
A single fiber can also do 10G, you just need different transceivers.
yeah, but aren’t they much more expensive and most often with SFP it has a dual cable setup, because in most cases for short runs it’s cheaper… so i sort of assume that they used an efficient solution… the fiber might be able to support faster speeds… but the gear seems to be around the 1gbit mark, must be some sort of local nexus nearby… which reroutes the network… the weird thing is that these guys could give me much higher internet speed… but it was basically at the push of a button…
it was setup and running like 12 hours or less from when i ordered it… how can some of my isp’s only offer 500mbit syncronous while another offer 1gbit… using the same hardware and clearly didn’t have to do anything to change it…
kinda makes it seem like most of them just artificially set the speed…
you really think my 50$ a month 1 gbit fiber internet can do full duplex…? i suppose i could test it,
wouldn’t it make more sense that it was the cheaper kind that can’t do full duplex…
but yeah i was wrong saying it can’t, it’s just very complex and not much used to my understanding… ofc on very long lines i’m sure the price ratio to fiber costs will make it more sensible.
very limited knowledge on fiber, have been trying to get more familiar with it for a while tho.
such a complex subject.
would be nice if i can get 10gbit + tho… not that i need it… would need a lan 40gbit first or so
My 600/600mbit movistar fiber connections in Barcelona are routinely fully saturated.
For example right now
eno1: 77.32 MB/s 5.80 MB/s 83.12 MB/s
This is just one of the modems, where we are currently pulling approx 35TB zfs snapshot backup from a server at hetzner.
Note - evidently Movistar promised all 600/600 fiber would be upgraded to gigabit on Jan 6th, it appears that’s already happened because 77.32MB/sec is for sure higher than 600mbit.
Also i’m pretty sure i only pay 35 euro/month for 600mbit, then an extra 5/month for it to be symetric, but don’t quote me on this.
Maybe it can, maybe it can’t, depending on how you ISP does the shaping, but 1G SFPs support full duplex. One fiber vs two is a tradeoff - one fiber (BiDi) SFPs are more expensive, but you only need one fiber - cheaper for longer runs, while two fiber SFPs are cheaper (especially multimode, which cannot be used for long distances), but you need two fibers.
1 Like
I should also mention that origionally I had 4x300mbit movistar’s, which all shared one fiber. When they upgraded them to 600mbit, they ran an individual fiber for each.
1 Like
600mbit/s = 75MB/sec so I’m not sure 77.34MB/s is really “higher” than 600mbit/s
2 Likes
been trying to test my connection… but that has proven problematic… doesn’t seem like a lot of people have 1gbit bandwidth to spare… lol tried speed tests to like 15 + different servers, even multiple ones at the same time, and still it’s a bit random…
was going to note that 600mbit thing…
@joesmoe i paid more for my 500mbit which was initially 400mbit i think, upgrading to 1gbit got cheaper, but it’s not really a surprise that everything here is expensive
really wanted to test the connection… but thus far i’ve just given up… when you can download with 85mb/s and upload with also like 600mbit then i guess it has to be full duplex… but pretty sure if it was able to do that it wouldn’t go through the network i get setup locally… so 1gbit total will be my limit for now… even if my internet might be able to do more…
might try and test the internet more some other time… but for now it seems like a waste of time since i’m unable to saturate it no matter what i try of online services and it’s not the connection that’s saturated… i’ve had it to 90mb/s and i think that was actually the write speed of the hdd’s that was being downloaded to.
my old cpu’s in the server is certainly also getting a bit of work done when running at near 1gbit … pfsense will take like 10% of the total cpu time across both cpu’s, but that’s not to bad… i was fearing worse.
I believe there’s some EU 10G endpoints there.
Appears that the new year is off to a pretty nice start. Download traffic is just plugging right along, on par with last month. Upload traffic has slowed once again, but hey, I can’t be too picky, and I’m ever hopeful that new ingress from some of these newly announced partnerships starts picking up in the near future.
Node 1:
Node 2:
Node 3:
Excel Tracker with everything combined (daily basis, not all inclusive):
Note: dips in values (mainly in diskspace used, although egress really did dip quite a lot) from yesterday are due to the fact that the dashboard resets at the end of month, so I captured the info a few hours before hand just so that I’d have something there for 31Dec2020.
Excel Tracker with everything combined (monthly basis, all inclusive):
4 Likes
Well your excel sure is impressive but wouldn’t it be a lot less work to use the prometheus exporter and make those charts with grafana?
I’m sure it would be, and I’d love to use those instead of updating my excel sheet every day. Although, I’ve yet to figure out how to actually use those…I get as far as running exporter, but have absolutely no idea how to use/make it work with grafana…
1 Like
I’m in exactly the same situation, never figured out how to get grafana working.