the current total network ingress for all satellites add up to like 137PB
this is 137000TB meaning if there are 10000 nodes each storagenode should be seeing 13.7TB of ingress, even changing the nodes to 20k doesn’t fix it… not even close…
but using 20k nodes does make it 6.85TB pr node, and the general ingress we usually have is like a bit over 500GB pr node, so i would assume these numbers in the dashboard are for whatever reason off by a decimal place…
Sorry for the late reply, hadn’t time to check it before. The values in the Dashboard are shown as reported by Storj. The shown ingress values are taken from here: https://stats.storjshare.io/data.json
(bandwidth_bytes_uploaded - > number of bytes uploaded (ingress) to the network for the last 30 days).
As example let’s take Satellite AP1. It reports: "bandwidth_bytes_uploaded":26354933339446650
Which would mean on AP1 the Ingress for the last 30 Days was 23.4 Petabytes. I wondered myself about the value but i haven’t questioned them for now. I think it contains all uploaded data (inclusive aborted uploads, the expansionfactor etc.)
to me it seems highly unlikely to be anything else than a decimal error as the gross estimate seems to be off by about 10x
to big a variation to be random flux, and its a simple mistake to make.
has happened even to NASA many times…
sure does make the network ingress numbers look much more impressive tho… so there is that… lol
you can run through the numbers quite easily… add the satellite ingress together and divide them across the number of nodes… doesn’t even have to be that accurate to see it’s way way way off.
what do you think @BrightSilence ?
Well, can’t say much other than this has to be a wrong number. The total stored on that satellite is "storage_total_bytes":801912361571328. That’s 802TB (@Arkina, Storj uses decimal byte notation, I suggest you align with that for consistency)
Maybe the stat actually displays what has been uploaded in total, not just last month. Either way those numbers can’t possibly be right.
i guess the next step then would be to wait and see what happens overtime…
ill try to track it a bit see, if i can make some better sense of it.
pretty sure i got it to about 137PB last but wasn’t super accurate… now its like 164PB so it’s gone up by over 20% over the last 4 days…
but ingress has also been a bit higher than usual…
15-07 its now a total of 173.5TB
16-07 total of 185,3TB
19-07 total ingress of 253,7TB
it does seems to change each day… and yes it does seems to move in both directions.
so has to be counting over some sort of time scale… where the granularity is pretty big.
if it jumps by 10PB in a day that does make it pretty close to the 4PB avg change pr day would be… think i saw it dropped 20 a while back between…
but it jumping around like that would also fit with how ingress behaves as it sometimes is 100kb pr node and other days it might be near 500kb
so that would make a 30 day moving avg be all over the place…
Hi @pdwelly thanks for letting us know! The team is aware of this issue and we are working on fixing it. It’s just on the metrics collection side for EU-N1 and the satellite itself is running as expected.
I have now found time to rework the Dashboards based on your Feedback. I have now Split it into two Dashboards, one for the whole Network with Stats for PRD and Dev Satellites and one with the Data for each Satellite. I have focused for now on the Networkstats, the per Satellite has not yet got any changes.
During this, I stumbled across some strange numbers for storage_remote_bytes. Those value did not really changed over the last 3 days on the PRD satellites even when other numbers as object count etc. change hourly. Is this intended? (@rafael, @jennifer)
I already cross-checked if that might be an scrapingissue, but it seems those are the provided values form storj. Above printscreen accumulates storage_remote_bytes/storage_total_objects for AP1, EU1 and US1.
graphs like the total objects one isn’t very useful because they is no reference to what the graph shows… is the uptake in the example you show from 213 mil to 214 mil or is it from 50 mil to 214 mil
one hasn’t a clue… so it would be much more useful with some numbers on the graphs like the top one for data usage
Great catch. It looks like reporting of those metrics was broken in the latest satellite code deployment. I found one or two other things that needed fixing along the way, so yay. I expect we’ll want to get these numbers back as soon as we can.
No, it’s just being computed wrong. I should have the min_healthy counts fixed soon.
Reporting of storage_inline_bytes, storage_inline_segments, storage_remote_bytes, and storage_remote_segments will be fixed in the next satellite deployment. For now, at least storage_total_bytes and storage_total_segments are working.
