I was working on a fix for my earnings calculator that resulted in faulty low Disk Last Report usage from the satellite, due to 0 records being added and had just implemented a fix when I encountered something strange in the last report from EU1.
The second to last line was causing the initial issue, which I fixed by ignoring those lines. That works for past results just fine, but for some reason the next report suddenly shows significantly lower storage usage for EU1. Like down to a third of what it used to be. Even though is should report on about 26 hours of usage, just like the few lines before it.
Is this a clean up initiated by Storj? Did something go wrong in storage usage calculation? Or am I interpreting the data wrong?
Note: at_rest_total is the storage used in byte*hours between the interval_end_time of the previous none-0 record and the current record. At least that’s how I’ve always interpreted it and that seemed to have always been correct. Fluctuations in at_rest_total are usually caused by different time intervals being reported on, but with that last EU1 record, the time interval is similar to previous records, yet at_rest_total has dropped to a third, indicating removal of 2/3rds of the data.
Edit: For context, here is a screenshot of such a 0 record happening in the past, but here you can clearly see the following record accounts for all storage usage since the record prior to that 0 record.
My stats are confusing again too. I don’t know if it’s the same error the last month or not but it makes me have doubts about the entire project and makes me want to turn off the node.
I wouldn’t have posted a new topic if it was the same error. Previous times it was just a display glitch related to the 0 records or missing reports. This time there seems to be an actual usage report with too low usage.
There’s no reason to have doubts about the project in my opinion. I’ve been around a long time and serious issues have always been addressed. Display glitches may stick around for a while as they don’t have any material impact, but if this is more than that, I’m sure it will be addressed appropriately. Let’s just wait for Storj to respond. Its perfectly possible that I’m misinterpreting things, so I’m not ready to jump to conclusions just yet.
Thanks @Knowledge . Let me know if someone needs a node ID to look into it. I’ll help any way I can from my end.
@Aitor : See. Took them barely 2 hours to respond and take it seriously. Now let’s just give them a bit of time to investigate, as the cause of these things might not be immediately obvious and may need several people to be involved.
An attitude of “trust but verify” has always worked just fine for me here.
I see the same dip on all my nodes in NA. I don’t think it’s specific to the node. I can’t check which or how many satellites are reporting this (the drop down list hasn’t worked for ages) but here is the screenshot,
The issue in this topic has coincided with a day of skipped reporting. Unfortunately the screenshots of the dashboard aren’t useful as result of that, as they may just point to the display glitch we’ve had before.
If you want to check you’ll have to get into the storage_usage.db file. I recommend working on a copy so you’re not interrupting your storagenode. You can install sqllite and use CLI or use https://sqlitebrowser.org/ like I did.
Satellite ID’s are binary, so it’s easiest to run
select *, hex(satellite_id) from storage_usage
Then scroll all the way down for the latest storage reports. Also please read my top post for how to interpret the data.
At this point just posting dashboard screenshots is unfortunately not useful.
API data is derived from the databases, but there is calculation inbetween to summarize per day. The bold lines are a result of the 0 records not being handled well by that calculation. However, Saltlake seems to be showing a similar issue on your node as I saw on mine, but without the raw data in the database it’s hard to say if it’s actually a report that is too low or a miscalculation in the API.
I’m less concerned if the node does a miscalculation locally vs the satellite miscalculating. As a satellite miscalculation would impact payouts.
Summarizing - It looks like there may have been an issue with tallies loading on the 6th for EU1. They recently (today) upgraded the EU1 cluster. It may take some time for the SNO graphs to catch up.
Thanks for the update!
Got quite a few questions, though I understand if you don’t have the answers just yet.
Is it possible Saltlake was effected as well, because @wildwaffle’s last screenshot seems to show a similar issue on Saltlake?
Since incomplete tallies were sent to nodes, will they be replaced in the node db? Because if not, the graph won’t fix itself when new tallies come in.
If tallies won’t be replaced, will the next tally compensate for missed Bh? Resulting in the graph showing a valley and a peek.
If neither 2 or 3 are answered with yes, how else will it be corrected?
Will any of this effect payouts?
Ps. Just to be clear this is very much not about a delay in storage usage reports or a graph issue. But about actually received reports that are too low.
I’m having the same issue for both of my nodes. On the 6th of May the disk space used in the graph went to zero for the us1 and eu1 satellites. The saltlake and ap1 seemed to be unaffected.
But when I woke up this morning I was pleasantly surprised that for the eu1 the graph seems to be fixed (but not for the us1 yet).
I have to check whether the earnings report will be corrected. For the eu1 alone, I have $ 0.79 for 7 days of 2.5 Tb used. That seems to be correct now.
So the issue seems to be fixed on the eu1, but not the us1 yet.
In the interest of providing my own answers whenever possible. This seems to be the case. On this node my storage_usage.db now looks like this for EU1.
Two missed days and then a record for the 8th that is way too low. Since I saw it resolve itself on EU1, I’m not too worried anymore about this.
But I would like to ask how we are supposed to interpret this data? I understand not having a new calculation because it took a little longer, but I’m really confused about receiving new records with numbers that are too low. I would like to get an accurate correct measure of the latest storage usage according to the satellite for my earnings calculator, but I have no clue how to derive that from this mess. Help would be appreciated.
