Cheers.
Where is the bug?
3 Likes
The “Not vetted” state. The node has been up, as you can see from the other dates, for more than enough time for vetting. Plus the earnings script shows vetting completed:
Payout and held amount by satellite:
┌────────────────────────────────┬─────────────┬──────────────────────────┬─────────────────────────────────────────────────────────────┐
│ SATELLITE │ HELD AMOUNT │ REPUTATION │ PAYOUT THIS MONTH │
│ Joined Month │ Total │ Disq Susp Down │ Storage Egress Repair/Aud Held Payout │
├────────────────────────────────┼─────────────┼──────────────────────────┼─────────────────────────────────────────────────────────────┤
│ ap1.storj.io:7777 (OK) │ │ │ $ 1.49/TBm $ 2.00/TB $ 2.00/TB 25% 75% │
│ 2025-01-04 7 │ $ 0.09 │ 0.00% 0.00% 0.09% │ $ 0.0117 $ 0.0004 $ 0.0064 -$ 0.0046 $ 0.0139 │
├────────────────────────────────┼─────────────┼──────────────────────────┼─────────────────────────────────────────────────────────────┤
Cheers.
You may subscribe to this issue:
Some data from one new node from me (same as on my GitHub post):
┌────────────────────────────────┬─────────────┬──────────────────────────┬─────────────────────────────────────────────────────────────┐
│ SATELLITE │ HELD AMOUNT │ REPUTATION │ PAYOUT THIS MONTH │
│ Joined Month │ Total │ Disq Susp Down │ Storage Egress Repair/Aud Held Payout │
├────────────────────────────────┼─────────────┼──────────────────────────┼─────────────────────────────────────────────────────────────┤
│ ap1.storj.io:7777 (OK) │ │ │ $ 1.49/TBm $ 2.00/TB $ 2.00/TB 50% 50% │
│ 2025-03-31 5 │ $ 0.01 │ 0.00% 0.00% 0.00% │ $ 0.0000 $ 0.0000 $ 0.0002 -$ 0.0001 $ 0.0001 │
├────────────────────────────────┼─────────────┼──────────────────────────┼─────────────────────────────────────────────────────────────┤
│ eu1.storj.io:7777 (OK) │ │ │ $ 1.49/TBm $ 2.00/TB $ 2.00/TB 50% 50% │
│ 2025-03-31 5 │ $ 0.01 │ 0.00% 0.00% 0.83% │ $ 0.0001 $ 0.0001 $ 0.0003 -$ 0.0002 $ 0.0002 │
├────────────────────────────────┼─────────────┼──────────────────────────┼─────────────────────────────────────────────────────────────┤
│ us1.storj.io:7777 (OK) │ │ │ $ 1.49/TBm $ 2.00/TB $ 2.00/TB 50% 50% │
│ 2025-03-31 5 │ $ 0.23 │ 0.00% 0.00% 0.00% │ $ 0.0025 $ 0.0009 $ 0.0001 -$ 0.0017 $ 0.0017 │
├────────────────────────────────┼─────────────┼──────────────────────────┼─────────────────────────────────────────────────────────────┤ +
│ TOTAL │ $ 0.25 │ │ $ 0.0026 $ 0.0009 $ 0.0007 -$ 0.0021 $ 0.0021 │
│ ESTIMATED END OF MONTH TOTAL │ $ 0.25 │ │ $ 0.2469 $ 0.0023 $ 0.0017 -$ 0.1254 $ 0.1254 │
└────────────────────────────────┴─────────────┴──────────────────────────┴─────────────────────────────────────────────────────────────┘
Data from du for used disk space and the number of files used by hashstore:
Node ID: 1V3piqkkxVNu4dxup3ACddaDAFHhor1fXAXGtHjf7EYxgapUgd
Version: 1.131.7
Total Storage used by Hashstore: 291GBHashstore: AP1 (6%)
Storage used: 18GB
Files: 38Hashstore: EU1 (13%)
Storage used: 37GB
Files: 66Hashstore: US1 (81%)
Storage used: 236GB
Files: 258
US1 holds 81% of the used storage, but the node is not vetted?
Evening
Following the recent dashboard update I now see this - that my node has never been vetted in eu1, despite being up over a year. Not sure what to make of it really - node has always been online, and I am in eu. Maybe makes sense why the node took so long to fill.
Thanks
CC
This is a classic example of “less is more”.
There was absolutely no need to put vetting status into the dashboard. Zero. “Getting vetted” is a one-time event, that happens to every node within a month, and never changes since. Now majority of operators has this stupid green date, that will never change, and carries zero actionable information, only sticks out like a sore thumb, and minority have another windmill to fight with faulty reporting. No UX expert was inconvenienced by discussing of this feature before it was rushed in production. It’s bad.
My recommendation is to ignore it. Completely. It’s bogus, broken, useless, and non-actionable. If you node is sending and receiving data (check logs) whatever nonsense dashboard shows is 100% irrelevant noise.
I’d take a step further and suggest ignoring entire dashboard altogether. It’s inaccurate at best, and blatantly wrong most of the time. Just look at the bandwidth graph. People who designed it skipped the class when units were discussed, especially part where you cannot put values in different units on the same plot. I’m referring to average per day vs today-so-far reporting. Let alone interpolating between the two. Or the fact that time zones exist. This shows sheer and blatant incompetence or negligence – especially since that was pointed out many times by now. There is no reason to assume the rest of the dashboard was implemented with more care.
If your node is offline – you will receive an email from satellite. Then do something. Not before. DOn’t waste your time obsessing over unreliable data.
/rant.
2 Likes
Why not handling the vetting thing by email as well? The satellite could just send one email on vetting finished event… 
2 Likes
There are indications that it’s not faulty reporting, and instead the satellite actually believes the node is not vetted and directs less traffic to the node as if it was not vetted.
5 Likes
Oof, bit harsh, dude. Having a bad day?
Kind of agree with this part though. Vetting status is only really interesting during vetting. But then mostly to see progress, which doesn’t seem to be part of this status display either.
Anyway, if you think the dashboard is useless, by all means, don’t use it. But I think many SNOs like to see what’s happening with their node. Even if information isn’t always directly actionable.
5 Likes
Right, since the vetting status is reported by satellite, the issue is not in node’s code, it’s satellite issue – regardless of whether it’s reporting or actually vetting process.
The bigger issue is why did it take implementation of a useless UI gimick and SNO involvement to detect these alleged issues with vetting process?
In either case, this information is of no interest to SNO – it’s non-actionable (telling Storj that they have an issue with vetting should not have required SNO involvement – it’s in Storj’s interest to vet good nodes) and shall not be present in the dashboard.
1 Like
A bit, yes, more sensitive to nonsense than usual.
I would agree if information presented was self-coherent and trustworthy. It isn’t. The only useful indicator is Online status. At least it hasn’t lied so far. Every single other UI item did.
I’ve already suggested before an overhaul of dashboard to only show big green or red online status and in the latter case – a text box with logs. That would be exactly what SNO needs – All OK vs Here are logs. Because the first thing is asked here “show me 20 lies from the logs”
So much time was wasted by SNO chasing ghosts just because the dashboard hallucinated – it’s just not acceptable.
Thanks to this label in Dashboard looks like we finally understand why things like this are happening. And now it can be fixed. If so, it’s certainly was not useless.
1 Like
See this:
Is annoying 2500 SNO better than having one storj engineer audit vetting process? It’s a very inefficient way to triage issues and waste of everyone’s time.
And no, we still don’t know why, we just collectively made 1 step towards triage – we know vetting is broken. Storj still needs to get an engineer triage it.
That engineer should have sat down before, and did the audit from the satellite side. This roundtrip though permanent UI feature as an aid to debugging is obnoxious.
What next? why dont’ we just dump debug info from go profiler into the dashboard – it may be useful too. Let sno deal with triaging and reporting. Come on. I dont’ mind helping to triage but not this way. This approach is broken.
That’s a pretty good question. Per the github thread this was even reported before the reporting was implemented, with no investigation from Storj and a statement:
Still, it’s good that now we have at least a better tool to investigate this problem.
I like to see the statistics from the power of two choices somewhere in the dashboard in the same way…
4 Likes
We were unable to investigate the lower usage due to a choice of two rule and random choice of nodes depending on the customers, I still do not know how to check that - too many factors, including daily fluctuations in demand.
But now when the Community noticed a correlation between the Unvetted state on the dashboard and less usage from the customers of that satellite, it might be a root cause of such behavior. So, I’m agree, this addition to the dashboard could become useful.
1 Like
Except before you annoyed those 2500 SNOs, that one storj engineer didn’t know they had to audit the vetting process.
But you have helped to triage the issue, because without this who knew there was an issue. Serendipity works in very strange ways.
Cheers.
2 Likes
Lol. I have one node that says it is 37 months old and not vetted on EU1.
I don’t know what to say.
2 Likes
While this bug investigated, i hope there will be small chance to implement vetting progress, which will be showing until vetting ended 
1 Like
I think it is not a sequential value but just a date or NULL. So the progress is not available on the node except for the implementation in the BS’s script. Which, obviously doesn’t match the real one as turned out.