Mines restart at 00:00 GMT time as well so we should be good
1 Like
i think mine uses GMT… it does seem keep counting far past the day shift… but i’m only +1 or +2 so not easy to really see all that clearly… GMT doesn’t have summer and winter time… 
so it changes if i’m +2 or +1
as a famous indian chief said… only government can convince people to cut the bottom of a blanket just to sow it back on the top in hope of getting more blanket.
or something like that
xD
2 Likes
@dragonhogan inet 200/20 - 2 nodes (11 and 4 months) - Midwest Region, USA
@SGC - inet 400/400 - 1 node (4 months) - Denmark
@TheMightyGreek 30/10 - 1 node
@Mark unk / unk - 1 node (age approx 1 year)
@striker43 unk/unk - 3 nodes (4 months) - Frankfurt, Germany
@zagg 1000/500 - 1 node (age unk) boulogne sur mer, France
@kevink unk/unk - 1 node maybe (1-2 months) - Germany
additional notes : i’ve extrapolated and made some logical deductions from place to place… the margin of error should be within 10% or much less most of the time, tho with the smaller nodes due to lack reliable stored data numbers, i’ve moved their later added number upwards.
added approx to most of those or notes to the place where there are issues with the data.
July 3rd
dragonhogan - ingress 17.75 - egress 19,58 = 1,89 ‰ of stored 10,31 TB
Mark - ingress 18.17 - egress 3,14 = unknown
SGC - ingress 17.40 - egress 19,34 = 1,77 ‰ of stored 10,9 TB
(used the july 4th for data stored)
striker43 - ingress 25,2 - egress 36,93 = 3,62 ‰ of stored 10,2 TB
(had to extrapolate here so striker numbers are highly inaccurate)
4th July
Mark - ingress 57,97 - egress 3,14 = ‰ of stored
SGC - ingress 55,32 - egress 23,58 = 2,16 ‰ of stored 10,9 TB - (still with stability issues)
kevink - ingress 48.05 - egress 9,73 = 2.86 ‰ of stored - 3,4 TB approx
(seems slightly off, but have been changing/tinker numbers of nodes)
striker43 - ingress 57,75 (i will assume mark and striker are the two accurate numbers here)
dragonhogan - ingress 57,03 - egress 22,63 = 2,20 ‰ of stored 10,31 TB approx
July 5th
striker43 - ingress 105,81
SGC - ingress 104,20 - egress 32,39 = 2,97 ‰ of stored 10,9 TB
(again used data from the 4th.... thx past me for not including my total disk space dash xD)
Kevink - ingress 106,07 - egress 17,61 = 5,18 ‰ of stored 3,4 TB approx
Krystof - ingress 106,15 - egress 38,24 = 3,55 ‰ of stored 11,85 TB
dragonhogan - ingress 101,62 - egress 34,74 = 3,37 ‰ of stored 10,31 TB approx
the mighty geek - ingress 105,45 - egress 6,27 = 5,22 ‰ of stored 1,2 TB
6th july
striker43 - ingress 112,83 - (missing egress)
dragonhogan - ingress 108,38 - egress 34,33 = 3,32 ‰ of stored 10,31 TB
TheMightyGeek - ingress 113,09 - egress 6,87 = 5,72 ‰ of stored 1,2
SCG - ingress 108,71 - egress 32,64 = 2,93 ‰ of stored 11,11TB
zagg - ingress 108,77 - egress 6,52 = 4,93 ‰ of stored 1,32 TB
7th july
TheMightyGeek - ingress 109,30 - egress 6,06 = 5.0 ‰ of stored - 1,3 TB
SGC - ingress 109,96 - egress 29,61 = 2.6 ‰ of stored - 11,21 TB
kevink - ingress 109,60 - egress 15,69 = 4.6 ‰ of stored - 3.4 TB approx
dragonhogan - ingress 104,21 - egress 29,11 = 2,8 ‰ of stored - 10,31TB
striker43 - ingress 109,76 - egress 25,58 = 2.5 ‰ of stored - 10,2TB
July 8th
TheMightyGeek - ingress 113,82 - egress 7,13 = 5.50 ‰ of stored 1,4 TB
SGC - ingress 114,14 - egress 36,49 = 3.20 ‰ of stored 11,33 TB
Dragonhogan - ingress 108,71 - egress 41,79 = 4,03 ‰ of stored 10,35 TB
striker43 - ingress 114,25 - egress 36,31 = 3,52 ‰ of stored 10,31 TB
PS : while going through this i did note that it doesn’t seem that the Used from the Total Disk Space dash doesn’t seem to grow in correlation with to the previous day’s ingress…
[went through the data… didn’t my own numbers did seem to follow it… so i duno… most likely just a irrelevant flux of incomplete datasets and my mind seeing patterns… ]
the stored TB are also very unreliable and most of them before the 6th have just been pushed back to like the 3rd or such to calculate the ‰ of stored
will have to take a look at that sometime… but it’s late here and making this summery took way more time than i had expected…
but i will regard this as the end of the current version of this, ofc people are allowed to continue, alas i’ll be looking into better ways of extracting and collecting this data in a database or such…
Ideas and help welcome hope you enjoy…
and sorry if i missed a post here and there… maybe…
3 Likes
@dragonhogan
nothing amazing here… but just found out my node is like 4 months now… not 5
lol … so i’m betting that you are seeing the benefit of your node being nearly a year
you did break the record… that should be atleast past 6 ‰… ill add the exacts to my summery tomorrow
oh wait i forgot to press update on the dashboard 
still behind on the egress tho, but ahead on ingress… was taking a glance at it… and i was like…
why isn’t my ingress correct lol because it was lower than yours… should be atleast equal… so i feared my spinning up more vm’s and other internet using stuff, had a major impact on the node… but all seems fine tho…
you might be seeing a bit of the same as themightygeek… where it seems his egress might be choking his ingress… just a bit…
1 Like
Could be true, although to be honest, I’d rather the egress move as freely as possible versus prioritizing ingress…
1 Like
got nice egress too yesterday, can’t complain:
And that’s with 3.6TB stored and you have 8TB.
1 Like
Egress is a function of node’s age and it does not depend on ingress function. My full old nodes have more egress (and no ingress ofc) than newer ones with free TBs and good ingress.
1 Like
idk about that, all that extra egress seems to come from the saltlake test satellite. So node age could not be the important factor here because most nodes are 5 month on that satellite as it joined very late.
All the ingress in July comes from europe-north.
1 Like
Same for me, bulk of egress from europe-north and yesterday there was a spike of egress for saltlake
1 Like
i’m not saying you should bandwidth limit, i mean if you have trouble already… how would it help if you set your upload bandwidth to 90%… creates more problems than it solves… but you would get max download speeds…
you could maybe prioritize overhead over actual usage… ofc if you are having 24mbytes/s aka 200mbit/s then you will be using about 10-20mbit for overhead…so overhead bandwidth is essentially worth 10x maybe better of what regular data gives you…
and most people do have 99% egress successrates… so maybe it would be worth it… but really… its not a solution to anything… the only real solution would be having less asynchronous internet, like if you could get 200/50… the real problem is that overhead is about 10% max… thus they say… people don’t upload they just download mainly… so to give you full download you need to have 200/20… so really your isp being idiots and not giving you 200/30 instead of assume you won’t be uploading anything on a regular basis…
with just 200/30 you might not even feel it…
but hey i got a super connection… maybe i should bandwidth limit myself… or make a new test node for it…been wanting to spin up a test node
then i could test all the different ratios or minimum connection speeds… see how it will work…
i mean with 20mbit you should be able to get 2.4mb/s which is a ton more than we use… so lets just set aside 1mb/s and then we got ingress which is like less than 2 mb/s easy. but hell lets set aside 3mb/s avg atm… and then + 10% on both… ofc its reversed… so the 10% of 1mb/s egress becomes + 0.1mb/s to ingress and vice versa… so 0.3mb/s + to egress so 1.3mb/s and 3.1mb/s which is like not even close to 30mbit full download and nowhere near 15mbit full upload… more like 11mbit avg
so your connection isn’t even that high in utilization… must be the peak downloads / uploads that runs into the max speeds…
1 Like
I got only 12Mbit/s upload at the moment and it is not maxed out.
1 Like
yeah in my netdata is spikes a lot…
has like a slow avg is rarely goes below … like at present it seems to be 2.5mbit and then it seems to spike anywhere from near 30mbit… with much lower avg tho… jumps around a ton…
but already now in just about 1 minute of watching my outbound, it has peaked above 20mbit 3 times already… and sometimes it will totally crazy and it has been near 20mbit many times… but it’s just below most often…
download is like ofc much higher… i think the highest inbound i’ve seen in netdata was like 120mbit…spike
i’m not saying there is a big problem… i’m saying there seems to be a direct correlation between low upload to download internet ratios… and lower than expected ingress…
also i was overestimating the numbers, so to be sure there was ample bandwidth for everything to see if it was a limitation or a problem with the peak’s
seems to be peak related… which sort of explains why it’s such a limited effect… usually if one chokes bandwidth completely… it will die… and it DIES HARD
you go from like 90% utilization to super high pings and 30% “utilization” / broken unstable throughput because everything just fails and retries, which increases bandwidth usage, which chokes overhead… and… and… its terrible… becomes like a cascade problem… and can basically put one offline…
1 Like
I’ll monitor the network usage and see it it saturates my connection when it spikes
1 Like
understand what you’re saying. All I was merely trying to say was if one is going to bottleneck the other, I’d rather the egress bottleneck the ingress instead of the other way around.
1 Like
totally agree on that, maybe i didn’t formulate that clearly enough… the ingress needs to be at like 24mb/s to eat most the ingress, so anything less than 50mbit and you would barely be able to measure it with any confidence in the numbers…
egress however well even at 10mbit you might start to see an noticeable effect, and since we work with very inconsistent activity … granted i got like 200kbyte running on other stuff both up and down right now…
but my inbound / outbound measured by netdata is all over the place… 40mbit peaks and maybe an avg of 13-15mbit and lows of 2 mbit…on inbound and outbound is peaks of 25mbit avg of 8mbit and lows at also like 2.5mbit…
so very wide ranges… if it was a clean avg out and in then i doubt you would see any restriction… and because it’s just peaks in many cases it might just slow it down slightly…
if we measure latency, then the bandwidth not having enough throughput isn’t really a huge issue… because like if i send a network packet to the United States then thats like 60ms travel time, + maybe 25ms search latency on disk or thats what my zfs iostat tells me… and then because it’s a customer in the states, then it will take 60ms to send the request to me, 25ms for me to answer, maybe slightly more… and 60ms the other way… so thats 150ms delay…
at 20mbit thats enough that you could send maybe 1/5th if we say 200ms latency and we assume you are nearby… so thats 500kbyte you in through could have sent before i might even respond from europa… also the bandwidth i get to the states isn’t my full 400mbit… more like 200mbit when i’ve tested it…
ofc my iops is super fast… and i got ssd caches… + synchronous fiber so really afaik i’m like the worst case scenario… and still you might be 25% done before my node starts…
before i got my system setup right i would have like 60-100ms disk latency… maybe even more if it was really busy… then the disk latency might become a bigger factor than the internet bandwidth.
so the egress bandwidth is certainly not the only factor in the calculation…
1 Like
hmm I do see a lot of spikes to 12Mbit/s in netdata so might be the case that my upload is limited… But the egress only recently got this high so I have to think about spending 5€/mo for a 50Mbit/s upload… But if egress drops again, I would be wasting money.
1 Like
yeah thats where cost vs benefit calculation comes in…
you might be missing 10% … i forgot what your connection speed is… again lol
but in anycase… it suppose it’s a matter of node age and node size… meaning we could in theory just say 50% utilization of upload might be where one runs into substantial issues… especially if one wants to use the internet connection for other things as well…
so if we say 10mbit as the 50% for ease of math… which is 1.2mb * 3600sec an hour * 24
which gives us a total of 104gb pr day possible egress using 10mbit.
and we are lucky to see 5 ‰ which is 5gb pr day pr 1tb stored data.
so 20mbit upload should be decent until around the 20tb mark… but then one would really start to see some limits… ofc at 20tb the egress ratio seems to drop, … lets check dragonhogans 1 year node for the ratio he is getting… 3 ‰ on the 7th for his old node… 4.54 ‰ for the 8th, and 7.2 ‰ for the 9th.
so for a year old node, at 10tb 20 mbit might start to begin to be a real limitation… and at 20tb i would expect seeing a substantial drop…
so you won’t be seeing much limitation because of it until you fill up your node…
and if you got 16tb i would say… when that is full and you upgrade next time, then internet upload upgrade should be very much a prime point to atleast look at the number for…
so yeah long story short… 
you are good
atleast when taking the general numbers into account… not really considering the impact of peaks… not sure how to do that math on that… but i would suspect take the avg bandwidth used by the node… and then say that should be the 50% of the bandwidth… or close … max…
also the people with low bandwidth connection on a 1/10 ratio of upload to download… has the overhead issue… if ingress over 50% it takes a substantial part of the egress, but when one has a better ration like 1/5 then it’s not really a big problem… because even at max bandwidth ingress, it would only take 50% of the egress
1/10 connections is mainly focused for max download speeds… they are not well suited for uploads… atleast comparatively …
according to a simple average calculation I should be good with 12Mbit/s upload and 200Mbit/s download, that’s true. The question is probably more interesting when thinking about download successrates since a capped upload at 1MB/s might lose some races. At the moment my download successrate went town to 95% from over 98%.
But not sure that extra 3% are worth 5€/mon for more upload