Storj nodes reporting a sharp decrease in month usage after March 24

Hello,

All of my Storj nodes are showing a sharp decline in “Average Disk Space Used This Month” in the Node Dashboard. Did I do something wrong?

This isn’t limited to new nodes — older nodes show the same drop. “Space Used” still shows the same value, but I’m not sure if data was moved to trash or if this is just a reporting/accounting change.

Here are some node dashboard screenshots:



This is normal; reporting is usually a few days behind.

Come back in a day or two, and it will be perfect :slight_smile: Nice project

Yep — Storj just gave me a mini heart attack :sweat_smile:

Everything is back to normal now. Sorry for the noise, and thanks everyone!

Storj is great… but a bit mischievous sometimes.

Is it though? We have pointed out these issues for years now and it seems like they treat us like we don’t matter. There is a word for software like this and storj is full of it

2 Likes

We’ve filed this issue as a bug, and we’ve definitely heard your feedback:

While it’s a low priority for our team, we’d be happy to accept PRs from the Сommunity to fix it.

1 Like

That’s the point, is it?

A low bug priority has nothing to do with how operators are treated.
There are few important factors: team capacity, the cost of development for the business, and the financial benefit it will bring.
This dashboard chart doesn’t affect payouts; it’s just a nice estimate, nothing more.
Unfortunately right now this bug is not a priority, but a good challenge for the Community to fix it.

1 Like

This thread is an excellent illustration of why I always push for removing information from the dashboard. Not adding more details.

Every detail, every bit of data surfaced to operators must be scrutinized and fiercely fought off, until it’s clear that it is absolutely necessary. Because every feature comes with support and maintenance requirements. These plots, along with the 99% of the rest of current dashboard — 100% waste of both.

On the other side — I agree, that if operators wanted this fixed, they could have submitted PR. Now, when AI leveled the field in terms of skill, it would have taken less time for anyone to do this than to read this thread.

The fact that we don’t see this happen means nobody from the community cares about dashboard enough to vibe code the fix. That also means storj needs to stop wasting time on it as well and focus on high priority features that improve QoS, not tickle SNOs egos. I look at recent changes in the dashboard and all I see wasted engineering time that could have been better used for something, anything else.

Or simply started using storj-node-exporter and grafama or some other collector graphing tool.

1 Like

It doesn’t matter what monitoring tool SNOs are using when the problem is missing reports from satellites.

Only storj can know the root cause and fixing it. Could be a code bug or insufficent hardware or just operator laziness.

+1. I only need a report showing changes in used-space: per-node and total. The UI “Bandwidth Utilization” and “Average Disk Space Used This Month” graphs aren’t useful.

1 Like

The root cause is known and described in the bug report. However, the only needed thing here is to ignore nulls and don’t plot them as zeroes. It would be nice also to ignore not complete reports as well, when the previous and next values are known and they are greater.

1 Like

Now I see where you are coming from. No, it is not just a plotting issue as far as I can see. The wrong values are also in the API averageUsageBytes. This goes down when the satellite calculations are missing and showing wrong values.

That’s why I believed it cannot be solved be community it must be solved by Storj in the way the satellite data gets calculated.

This is the same info, used for plotting this graph. So, the consumers should also be adapted this way, or the node’s API should filter them to allow all consumers to get the information with values, not nulls or half-filled.
And it can be solved by the Community.

The fix for the root cause is not as easy especially considering a low or non-existing benefit. If that would affect something, I’m sure, it will be fixed. But currently it’s not.

Yes probably. But it is not just a plotting issue then, but a problem of how the underlying data is handled for plotting and calculations?

It’s the same issue and it’s visible to operators.
I think, that right now it could be used as an indicator, that something might be wrong with other chores. However, it’s a some kind of a second line of monitoring, the main one is tracking these processes closely to allow the team to fix the issue before it could affect customers. So, notifying the team about an externally visible issue with this graph could be useful in my opinion to confirm that there is no other issues.

Yes. This chore was not designed to be 100% reliable (it just happen to be reliable for a long time, but now it’s not), so the API should ignore missing data, because it’s used mostly only for this graph. It’s also used in some estimations, but again the estimation is not precise anyway.

All I can say it is terribly annoying and shocking every month over and over again as a drop of average data stored could indicate a problem of the node with availability.

This only to learn that again and again the node(s) are fine and only the data from the satellites are wrong resp. mishandled.
Actually it is more than annoying.

When you know that it can happen unrelated to the node’s performance you start to ignore it :person_shrugging:
If your node has a good reputation, is online and you do not have any FATAL or Unrecoverable errors in your logs - there is no need to panic.

Then I may only suggest that the Community might fix this issue and do not wait when the queue would be empty to take it into work by the team.

There’s 26 pull requests opened, many of them over a year waiting for review. It doesn’t seem like a priority to deal with them, this also takes time.

Most of each either Storjlings or with failed checks.
However, I notified the team to check what’s going on with storj/storj PRs.

1 Like