this is normal… this graph was suppose to be more stable… the reason it’s unstable is because it’s updated each time the satellites complete a computation cycle or something like that…
so the reason it goes to 0 on the 18th is that the satellites didn’t finish their calculation, that was finished at the 19th and which is why the 19th is higher than the others.
i’ve complained about this a lot, and it’s like its gotten better… but yeah it still happens… really since this is a space used graph it should be nearly flat for most of us… because the variation each day is less that 1-3%
also the 19th peak seems kinda low… i would expect to see more of that when and if it finishes on the 20th … it’s really sort of a graph of satellite cpu latency rather than anything else…
the graph is right, if one takes the avg… but the day to day and even week to week is very rough…
as you can see the valley is much larger than the additional peak… which is why i expect there to be more plotted today, it has an avg to uphold.
anyways, long story short, nothing to worry about… maybe a satellite being overloaded, but it has no effect for SNO’s, in the past satellite latency have caused late payments because of the processing being months behind, but it’s been a while since we have seen these spikes, so i’m guessing … maybe because they restarted the next test data sequence and the system started generating test data or whatever, putting high load on the cpu’s slowing down the space computations.
No, this is different because it is the total not the individual satellite this time.
edit to say the asia satellite seems to be unaffected.
I have the same and I thought the same we will se at the end of the day because technically the 2th should be double the height of the average of the other days, I’m interested too
This is absolutely nothing new see here. Saltlake used space wrong calculation on dashboard that can affect payout calculation... again
saltlake satellite.
So while I hear all of this, and I do remember the Salt Lake issue and what SGC is talking about, I take severe issue with it reporting close to the average, and then there after the fact it “correcting” to much less than it should be. This looks to this time to be not a single SAT issue (see Salt Lake) thought- more like a broad underreport of information from storage_usage.db/piece_space_used.db .
Call me crazy, I just expect my histogram of things to not suddenly move historical data around.
Thats true I didn’t think of that, There should be a spike of data the next time and there isnt.
it’s not because it’s inaccurate, it’s just slow and i believe that when littleskunk explained it, there was also something with how the timing of it was done…
i think there was some time schedules that was overlapping… my initial guess of what was wrong was kinda right, it was ofc just a lot more complex.
basically goes like this… the graph is updated twice a day, lets say noon and midnight, and a full tally for the satellite usually takes less than that to do it’s tally of the data stored, when the satellite is done the numbers are published and used by the graph.
but if the tally isn’t done, because the satellite isn’t done with it’s tally, because maybe it was dealing with other workloads like say deletions, then the number isn’t there for the graph to plot.
and then when the tally is done it might be a day late and then it will be counted in the next graph plot…
which makes the graph basically useless… it plots satellite latency… mostly with the avg number being our stored data tally…
but littleskunk talked something about that once a month or so there was some counters / tallys that would overlap and thus create that big gap… i’m sure it will be fixed one day…
duno why they don’t just smooth it out tho… would be pretty simple to do…
and really it is live… which is part of the problem… it plots the data when it has them… instead of just guestimating.
I guess, yes, the live part is my biggest issue with the whole thing. The sat verifies you’ve stored X TB for Y hrs, but it’s not like that changes days later- you still held onto X TB for Y hrs, regardless of it being one day later or 10 days later. I had a node go from 187TBh to 14.9TBh on the 16th with 221.22 the day following… so obviously the 187TBh was right but due to “funny math” I get 14.9 and 221.22 that needed to be hand calculated at, roughly, 118TBh per day [ (14.9+221.22)/2 ]… if the assumption of delayed reporting is taken so the spacetime attribution would be on the following day. The issue is that is also quite far from the possible truth given the riding average is in the 150-160’s- so effectively, that specific node “lost” some of its attributed storage time.
eg, in the accounting world, that sort of historical manipulation is the type of thing people get collared for- it reeks of manipulation and corruption… unless it can be accounted for and explained. Right now, neither can be done.
it’s what feature request voting is for… 
i’ve already tried to get it fixed with little luck… maybe i phrased it wrong or didn’t understand the problem, or didn’t supply a good enough solution.
you have my vote for basically any solution that fixes it because i feel the graph in its current state is basically useless filler.
Exactly what I thought: it doesn’t just look like a display issue due to late calculations. This time, it just looks like some data is missing, because we should see the same spike (but inverted) the day after, which is far from being the case. Here what my graph looks like:
(I added the dotted red line which represents what I would have expected)
Same expected the same
well you can go check the numbers, i’m sure it’s okay… does look kinda off… tho if you squint and tilt your head you can see actually on the high end of the avg from the other side of the dip, could be that its actually right…
can be very difficult to tell from this graph… it’s shit, but there are real numbers behind it you can go access, and i do believe some people more or less do the accounting for this from time to time…
doubt they do it every month tho… and it does happy they find mistakes, but that gets rarer and rarer all the time ofc…
but yeah it’s a representation of numbers ofc which really only the avg can be used for anything…
facepalms i hate this graph
same issue here:
not corrected next day either
I see the same issue:
Only europe-north-1 is tried to recover, but another 3 just loose data.
Is it Storj-Labs have a problem with satellites databases?
So I guess the other issue is this seems to be affecting only non-asian SATs, based on your graphs.
Even with Euro-N trying to correct itself… there’s something afoot with those four.
I confirm that europe-north tried to recover, asia-east didn’t have this issue.
doesn’t loose data, the calculation is delayed and then it either catches up or just slowly slips further and further behind… and because it’s most likely running on some sort of vm then if one vm uses more cpu the rest of the vm’s have less to work with…
which might explain why it seems to shift be the different satellites… it’s only because there is so much accounting data that it’s difficult for them to process it… last time before they noticed it was running behind it was like more than a month behind if not 6 months… i forget the actual number
was only on one satellite tho, last time… to my knowledge, but i’m not really that well informed on this.
would make sense that they had problem with processing, moved it to some sort of shared cloud thing and now a few months later ran out of processing power again…
that would make sense if it’s being run from one cluster, ofc each continent would need its own, but in some regions multiple satellites might be easier to just keep on the same hardware… or it was part of the patch implemented last time.
maybe we should hear what @littleskunk has to say
but it’s been an ongoing issue, and has no detrimental impacts aside from at worst late payouts because the space accounting has not been processed or whatever…
Personally, I don’t get that much payout yet, so I don’t have any issue even if couple of cents get lost or are delayed.
My goal here is to only report and confirm that I also have this issue in order for Storj team to be aware of scale.