Bug in new Vetted Date

[alpharabbit]

I wonder if there is a false positive bug as well? This node was started 19.06.2025. So according to dashboard US1 vetting was done in 5 days.

[jammerdan]

@alpharabbit

I wonder if there is a false positive bug as well?

If it is not a bug, maybe vetting process shall be re-adjusted so it takes at least 1 month again?
What is the point of vetting if it completes in 5 days?

[arrogantrabbit]

Fellas, hear me out, how about we fix all the churn and turmoil caused by vetting date reporting (along with UX mishap of having transient face tattoo one-time event status displayed forever in contrasting blob) with one simple trick:

git revert --no-edit eaaee83

[Toyoo]

I would like to request anyone who observes anomalously long vetting (let say, >3 months) and who also has access to full node logs for the whole period to compute the number of successful GET_AUDIT responses from the affected satellites. It will be helpful to figure out whether the problem lies in the node not getting audits, or successful audits not being counted properly.

(also, please ignore @arrogantrabbit for a bit)

[arrogantrabbit]

But why? Satellite knows how many audits it sent to what nodes and how many responses it received. There is no need to give SNO another fools errand.

Storj has full access to satellites. Let them review the stats, identify problematic nodes, debug, and fix. There is absolutely no need to involve SNO: if nodes were not responding to audits they would be disqualified. So the problem is not with nodes/

[Alexey]

satellite/reputation/config.go

So, I’m waiting for an explanation of this is a little bit cryptic description.

I have an idea though. They should be consecutive. So, if for example, the node passed 10 audits and the next one was not answered (timeout, etc.), then it resets.

[jammerdan]

@Alexey

I have an idea though. They should be consecutive. So, if for example, the node passed 10 audits and the next one was not answered (timeout, etc.), then it resets.

The question is what is the goal.
If the goal is that the node has to prove some stability and durability then vetting within of 5 days is wrong.
There is no time setting in the code piece that you have posted similar to this one:

@Alexey

"the time period that must pass before suspended nodes will be disqualified" releaseDefault:"168h" devDefault:"1h"

Maybe like this:
Passing audit should consist of a minimum number of successful audits + a minimum number of consecutive successful audits + a minimum of ((consecutive) online) days before a node leaves vetting phase and can receive full data.

So it could be for example: 1 count for every X consecutive audits, if fail, then reset.
After Y counts = vetted unless minimum (online) time z has not passed. After minimum (online) time z has passed = vetted.

[snorkel]

From 18 nodes, I found one with the unvetted status on EU1, and the incoming traffic is in accordance with an unvetted node; so the tag is correct. Question is what to do about it?
Could Storj trigger another vetting session? Like for all unvetted nodes older than 2 months, just restart the vetting process, but with more data, proportionaly with the number of nodes.

On the same machine, a younger node (-1 month) stores 25% more data. I didn’t understand the culprit until now.
Something must be done, because now that 131.7 is live, this forum will fill soon with unvetted nodes complaints.
And is very unfair for the node operators that ran nodes for months/years not knowing that they are in the vetting pool.

[Alexey]

But now you know. However, I do not know the reason yet.
Accordingly this

satellite/satellitedb/reputations.go

it should become vetted if the total amount of audits more or equal to configured amount of expected audits to mark the node as vetted.
And the config is

satellite/reputation/config.go

By the way, could you please post the amount of audits and successful audits from that satellite for that node?

I need an example of NodeID, which satellite and the amount of audits and the amount of successful audits.
Please post this info or you may send me it via DM. By the way, NodeID is not a secret information, it’s publicly available.

[BrightSilence]

@Alexey

I think it is not a sequential value but just a date or NULL. So the progress is not available on the node except for the implementation in the BS’s script. Which, obviously doesn’t match the real one as turned out.

It didn’t. Btw, feel free to mention me to get my attention if clarification is useful. Before the changes that added the vetting status from the satellite to the node API/databases, there was no other way to determine vetting status than simply counting successful audits. So that’s what the earnings calculator up to version 14.1.0 did. Clearly this wasn’t sufficient to determine whether the satellite considers the node vetted.

I have created a new version that now considers the satellite’s reported status as well. Info can be found here and I’m looking for some testing help as well.

Earnings calculator (Update 2024-07-28: v14.1.0 - Now shows garbage collection progress! - Detailed earnings info and health status of your node, including vetting progress and garbage collection status)

I’ve added a feature that considers the new vetting status as reported by the satellite. This will help show whether your node is considered vetted by the satellite independent of the number of reported successful audits. This prevents the earnings calculator from just assuming your node has been successfully vetted after 100 successful audits. If for some reason your node is not considered vetted after 100 audits are completed it will show up like this. [image] However, as I don’t have any n…

[snorkel]

How can I check those? There’s to much stuff in my notes, and to little time to search for how to…

[alpharabbit]

@snorkel

How can I check those?

Search for “reputation” in log file. Log level must be info.

[alpharabbit]

This node is not vetted at EU1.

Node ID: 12JdvQaAYtxk53yJDHkQoMhBFyUaGBNu4MHaJ3kpkVZncoSPxQx

2025-07-18T08:40:35+02:00       INFO    reputation:service      node scores updated     {"Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Total Audits": 56974, "Successful Audits": 56409, "Audit Score": 1, "Online Score": 0.9975501170493515, "Suspension Score": 1, "Audit Score Delta": 0, "Online Score Delta": 0, "Suspension Score Delta": 0}
2025-07-18T08:40:35+02:00       INFO    reputation:service      node scores updated     {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Total Audits": 15595, "Successful Audits": 15459, "Audit Score": 1, "Online Score": 0.9976702679717343, "Suspension Score": 1, "Audit Score Delta": 0, "Online Score Delta": 0, "Suspension Score Delta": 0}
2025-07-18T12:38:46+02:00       INFO    reputation:service      node scores updated     {"Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Total Audits": 7730, "Successful Audits": 7662, "Audit Score": 1, "Online Score": 0.9960743512858218, "Suspension Score": 1, "Audit Score Delta": 0, "Online Score Delta": 0, "Suspension Score Delta": 0}
2025-07-18T12:38:46+02:00       INFO    reputation:service      node scores updated     {"Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Total Audits": 64532, "Successful Audits": 63916, "Audit Score": 0.9999999995460614, "Online Score": 0.9958201419850893, "Suspension Score": 1, "Audit Score Delta": 0.000000000026159407973125326, "Online Score Delta": 0, "Suspension Score Delta": 0}

[snorkel]

Any other way? For no info level logs? Is it in the debug port output?

[flo]

I remember when people were reporting nodes that receive less traffic a long time ago… nobody investigated.

Aren’t there basic sanity checks on the satellite? Nodes with 50’000 audits certainly shouldn’t be unvetted any more.

[seanr22a]

12u9Sbesa5nAtmLCc1yu7p51qQhTqb2fb3mzS3bNpXcvZonshpE NOT vetted at eu1 and saltlake
1kNJtLuSY3AvRjpTLufn5Cf7ZwBRjTauzuQiWWyYTzB5HUKhTF NOT vetted at eu1
1N67qTSwEdthr6pkbuqthmRwWQPLgFYnqepvtpJqMBz8grYbky NOT vetted at eu1
12oNsFhs6G6wNFFMQCuw3MNzpANtnNvnZM5eApVDFLufqaeE5Aj NOT vetted at eu1 and saltlake
12X9c1HrJ6beBXW84WwLr9ZV5SgwQjioYx39JedfsEThGmfbpcp NOT vetted at ap1
124NM5z5BkfKJJPkiT3R3mAmKd2QcQSeL7ZAPu8arg85gCuneDN NOT vetted at eu1
1j59S2HSp5nveeCQcg9vB7uLHZgSXjw89aBtetCBhFg5pccnTs NOT vetted at ap1

Total 7 nodes out of 24 nodes not vetted.

[edit] They are all old nodes more than a year.

[Balage76]

nodeID: 1n6XVttjSbrA8UQUpXBMKxPD1Za2Am9TiQYvXhURDGfppNE171

I have other nodes on the same PC, behind the same IP. This unvetted node has 2/3 of ingress for EU1 versus all the other, vetted nodes.

[DisaSoft]

2 nodes I mentioned here - started at the same day, at the same location, on the same hardware and sharing the same IP address. Almost 1 year old at this moment and nothing is changed since 9 month after that post. Still has similar traffic from US1 and very different from AP1 and EU1.

12BdvHjDkZFp4NZeX18yTHU4NKcwc6iU4maaH4YvjWyHu9SE6H2 - Lucky (vetted on all satellites)
12ATDR6JehMzhhadqWh1wBXn4iFu2iDVbu8YEgEGVgPHSfWeLsF - Unlucky (not vetted on ap1 and eu1)

Lucky one (12BdvHjDkZFp4NZeX18yTHU4NKcwc6iU4maaH4YvjWyHu9SE6H2):

2025-07-19T00:50:56Z    INFO    reputation:service      node scores updated     {"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Total Audits": 56478, "Successful Audits": 55996, "Audit Score": 1, "Online Score": 0.9995033282130057, "Suspension Score": 1, "Audit Score Delta": 0, "Online Score Delta": 0, "Suspension Score Delta": 0}
2025-07-19T00:50:56Z    INFO    reputation:service      node scores updated     {"Process": "storagenode", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Total Audits": 241823, "Successful Audits": 240179, "Audit Score": 1, "Online Score": 0.9994583271522524, "Suspension Score": 1, "Audit Score Delta": 0, "Online Score Delta": 0, "Suspension Score Delta": 0}
2025-07-19T00:50:57Z    INFO    reputation:service      node scores updated     {"Process": "storagenode", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Total Audits": 113676, "Successful Audits": 113140, "Audit Score": 1, "Online Score": 0.9990754597933748, "Suspension Score": 1, "Audit Score Delta": 0, "Online Score Delta": 0, "Suspension Score Delta": 0}
2025-07-19T00:50:57Z    INFO    reputation:service      node scores updated     {"Process": "storagenode", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Total Audits": 447936, "Successful Audits": 445260, "Audit Score": 1, "Online Score": 0.9993424640656474, "Suspension Score": 1, "Audit Score Delta": 0, "Online Score Delta": 0, "Suspension Score Delta": 0}

Unlucky one (12ATDR6JehMzhhadqWh1wBXn4iFu2iDVbu8YEgEGVgPHSfWeLsF):

2025-07-19T00:56:32Z    INFO    reputation:service      node scores updated     {"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Total Audits": 52855, "Successful Audits": 52592, "Audit Score": 1, "Online Score": 1, "Suspension Score": 1, "Audit Score Delta": 0, "Online Score Delta": 0, "Suspension Score Delta": 0}
2025-07-19T00:56:33Z    INFO    reputation:service      node scores updated     {"Process": "storagenode", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Total Audits": 42454, "Successful Audits": 42252, "Audit Score": 1, "Online Score": 0.9996666666666667, "Suspension Score": 1, "Audit Score Delta": 0, "Online Score Delta": 0, "Suspension Score Delta": 0}
2025-07-19T00:56:33Z    INFO    reputation:service      node scores updated     {"Process": "storagenode", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Total Audits": 114490, "Successful Audits": 114196, "Audit Score": 1, "Online Score": 0.9987033680338737, "Suspension Score": 1, "Audit Score Delta": 0, "Online Score Delta": 0, "Suspension Score Delta": 0}
2025-07-19T00:56:34Z    INFO    reputation:service      node scores updated     {"Process": "storagenode", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Total Audits": 62439, "Successful Audits": 62201, "Audit Score": 1, "Online Score": 0.9998015873015873, "Suspension Score": 1, "Audit Score Delta": 0, "Online Score Delta": 0, "Suspension Score Delta": 0}

Screenshots

[Alexey]

via API, but seems only using an audit history.

PowerShell

((Invoke-WebRequest http://localhost:14002/api/sno/satellite/12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs).Content | ConvertFrom-Json).auditHistory.windows | Measure-Object -Property totalCount, onlineCount -Sum

bash

curl -s http://localhost:14002/api/sno/satellite/12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs | \
    jq '[.auditHistory.windows[]]
        | {
            totalCount: (reduce .[] as $item (0; . + $item.totalCount)),
            onlineCount: (reduce .[] as $item (0; . + $item.onlineCount))
        }'

[Alexey]

@flo

I remember when people were reporting nodes that receive less traffic a long time ago… nobody investigated.

Bug in new Vetted Date

We were unable to investigate the lower usage due to a choice of two rule and random choice of nodes depending on the customers, I still do not know how to check that - too many factors, including daily fluctuations in demand. But now when the Community noticed a correlation between the Unvetted state on the dashboard and less usage from the customers of that satellite, it might be a root cause of such behavior. So, I’m agree, this addition to the dashboard could become useful.

I would add, I believed that if a node got >= 100 audits, it was vetted, since that’s what the code said, so the vetted state wasn’t even considered.