Something happened with audits?

Awful_Charlie · December 7, 2025, 10:56am

It’s not just audits - it’s certainly the online score that is not getting updated either, but I’ve never seen a suspension score other than 1 (100%) so can’t comment on that.

I had a terrible October (one disk popped, SATA cable problem, router firmware update), and was waiting for my online scores to improves before doing some more remedial work. After 33 days I gave up waiting and just did it, and the online scores didn’t go down either. I tested that on the QA satellite, and after 6 hours down there, the dashboard was still reporting 100% online

Maybe related, but seems to be in the period for updates from 1.139.6 to 1.141.2
I still see the ‘reputation:service node scores updated’ messages come in, but now the delta is consistently 0

Aleksman4o · December 7, 2025, 1:34pm

done
saltlake - last contact 2025-11-03
ap1 - last contact 2025-11-15
us1 - last contact 2025-11-19
eu1 - last contact 2025-11-12

Summary

{
  "id": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE",
  "auditHistory": [
    {
      "windowStart": "2025-10-12T00:00:00Z",
      "totalCount": 139,
      "onlineCount": 59
    },
    {
      "windowStart": "2025-10-12T12:00:00Z",
      "totalCount": 113,
      "onlineCount": 89
    },
    {
      "windowStart": "2025-10-13T00:00:00Z",
      "totalCount": 106,
      "onlineCount": 87
    },
    {
      "windowStart": "2025-10-13T12:00:00Z",
      "totalCount": 24,
      "onlineCount": 15
    },
    {
      "windowStart": "2025-10-14T00:00:00Z",
      "totalCount": 114,
      "onlineCount": 101
    },
    {
      "windowStart": "2025-10-14T12:00:00Z",
      "totalCount": 131,
      "onlineCount": 109
    },
    {
      "windowStart": "2025-10-15T00:00:00Z",
      "totalCount": 142,
      "onlineCount": 101
    },
    {
      "windowStart": "2025-10-15T12:00:00Z",
      "totalCount": 141,
      "onlineCount": 113
    },
    {
      "windowStart": "2025-10-16T00:00:00Z",
      "totalCount": 111,
      "onlineCount": 73
    },
    {
      "windowStart": "2025-10-17T00:00:00Z",
      "totalCount": 119,
      "onlineCount": 61
    },
    {
      "windowStart": "2025-10-17T12:00:00Z",
      "totalCount": 97,
      "onlineCount": 94
    },
    {
      "windowStart": "2025-10-18T00:00:00Z",
      "totalCount": 131,
      "onlineCount": 88
    },
    {
      "windowStart": "2025-10-18T12:00:00Z",
      "totalCount": 78,
      "onlineCount": 70
    },
    {
      "windowStart": "2025-10-23T12:00:00Z",
      "totalCount": 136,
      "onlineCount": 133
    },
    {
      "windowStart": "2025-10-24T12:00:00Z",
      "totalCount": 152,
      "onlineCount": 148
    },
    {
      "windowStart": "2025-11-02T12:00:00Z",
      "totalCount": 93,
      "onlineCount": 90
    },
    {
      "windowStart": "2025-11-03T12:00:00Z",
      "totalCount": 135,
      "onlineCount": 132
    }
  ]
}
{
  "id": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6",
  "auditHistory": [
    {
      "windowStart": "2025-10-23T12:00:00Z",
      "totalCount": 280,
      "onlineCount": 276
    },
    {
      "windowStart": "2025-10-25T00:00:00Z",
      "totalCount": 309,
      "onlineCount": 307
    },
    {
      "windowStart": "2025-10-26T12:00:00Z",
      "totalCount": 276,
      "onlineCount": 275
    },
    {
      "windowStart": "2025-10-27T00:00:00Z",
      "totalCount": 495,
      "onlineCount": 494
    },
    {
      "windowStart": "2025-11-07T12:00:00Z",
      "totalCount": 263,
      "onlineCount": 261
    },
    {
      "windowStart": "2025-11-14T12:00:00Z",
      "totalCount": 369,
      "onlineCount": 362
    },
    {
      "windowStart": "2025-11-15T00:00:00Z",
      "totalCount": 387,
      "onlineCount": 386
    }
  ]
}
{
  "id": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S",
  "auditHistory": [
    {
      "windowStart": "2025-10-21T00:00:00Z",
      "totalCount": 2796,
      "onlineCount": 2791
    },
    {
      "windowStart": "2025-10-21T12:00:00Z",
      "totalCount": 1453,
      "onlineCount": 1451
    },
    {
      "windowStart": "2025-10-23T00:00:00Z",
      "totalCount": 2103,
      "onlineCount": 2100
    },
    {
      "windowStart": "2025-10-23T12:00:00Z",
      "totalCount": 3602,
      "onlineCount": 3596
    },
    {
      "windowStart": "2025-10-24T12:00:00Z",
      "totalCount": 2324,
      "onlineCount": 2274
    },
    {
      "windowStart": "2025-10-25T00:00:00Z",
      "totalCount": 3088,
      "onlineCount": 3087
    },
    {
      "windowStart": "2025-10-26T12:00:00Z",
      "totalCount": 4536,
      "onlineCount": 4533
    },
    {
      "windowStart": "2025-10-27T12:00:00Z",
      "totalCount": 1461,
      "onlineCount": 1460
    },
    {
      "windowStart": "2025-10-28T00:00:00Z",
      "totalCount": 3195,
      "onlineCount": 3194
    },
    {
      "windowStart": "2025-10-29T00:00:00Z",
      "totalCount": 3205,
      "onlineCount": 3204
    },
    {
      "windowStart": "2025-10-30T00:00:00Z",
      "totalCount": 3437,
      "onlineCount": 3436
    },
    {
      "windowStart": "2025-10-31T12:00:00Z",
      "totalCount": 3822,
      "onlineCount": 3821
    },
    {
      "windowStart": "2025-11-02T00:00:00Z",
      "totalCount": 3857,
      "onlineCount": 3854
    },
    {
      "windowStart": "2025-11-02T12:00:00Z",
      "totalCount": 4905,
      "onlineCount": 4853
    },
    {
      "windowStart": "2025-11-03T00:00:00Z",
      "totalCount": 5263,
      "onlineCount": 5262
    },
    {
      "windowStart": "2025-11-03T12:00:00Z",
      "totalCount": 2168,
      "onlineCount": 1634
    },
    {
      "windowStart": "2025-11-06T00:00:00Z",
      "totalCount": 2582,
      "onlineCount": 2580
    },
    {
      "windowStart": "2025-11-06T12:00:00Z",
      "totalCount": 3629,
      "onlineCount": 3610
    },
    {
      "windowStart": "2025-11-08T00:00:00Z",
      "totalCount": 1370,
      "onlineCount": 1367
    },
    {
      "windowStart": "2025-11-08T12:00:00Z",
      "totalCount": 2132,
      "onlineCount": 2130
    },
    {
      "windowStart": "2025-11-10T00:00:00Z",
      "totalCount": 3999,
      "onlineCount": 3998
    },
    {
      "windowStart": "2025-11-10T12:00:00Z",
      "totalCount": 2811,
      "onlineCount": 2810
    },
    {
      "windowStart": "2025-11-12T00:00:00Z",
      "totalCount": 2587,
      "onlineCount": 2583
    },
    {
      "windowStart": "2025-11-13T00:00:00Z",
      "totalCount": 4280,
      "onlineCount": 4279
    },
    {
      "windowStart": "2025-11-14T12:00:00Z",
      "totalCount": 3659,
      "onlineCount": 3658
    },
    {
      "windowStart": "2025-11-15T00:00:00Z",
      "totalCount": 12,
      "onlineCount": 9
    },
    {
      "windowStart": "2025-11-17T00:00:00Z",
      "totalCount": 2762,
      "onlineCount": 2761
    },
    {
      "windowStart": "2025-11-17T12:00:00Z",
      "totalCount": 3207,
      "onlineCount": 3206
    },
    {
      "windowStart": "2025-11-18T00:00:00Z",
      "totalCount": 2774,
      "onlineCount": 2772
    },
    {
      "windowStart": "2025-11-19T00:00:00Z",
      "totalCount": 2713,
      "onlineCount": 2711
    }
  ]
}
{
  "id": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs",
  "auditHistory": [
    {
      "windowStart": "2025-10-21T12:00:00Z",
      "totalCount": 982,
      "onlineCount": 981
    },
    {
      "windowStart": "2025-10-22T00:00:00Z",
      "totalCount": 975,
      "onlineCount": 973
    },
    {
      "windowStart": "2025-10-23T12:00:00Z",
      "totalCount": 607,
      "onlineCount": 602
    },
    {
      "windowStart": "2025-10-25T00:00:00Z",
      "totalCount": 1396,
      "onlineCount": 1370
    },
    {
      "windowStart": "2025-10-25T12:00:00Z",
      "totalCount": 923,
      "onlineCount": 916
    },
    {
      "windowStart": "2025-11-04T00:00:00Z",
      "totalCount": 1508,
      "onlineCount": 1507
    },
    {
      "windowStart": "2025-11-05T12:00:00Z",
      "totalCount": 1473,
      "onlineCount": 1472
    },
    {
      "windowStart": "2025-11-07T00:00:00Z",
      "totalCount": 675,
      "onlineCount": 674
    },
    {
      "windowStart": "2025-11-08T12:00:00Z",
      "totalCount": 773,
      "onlineCount": 772
    },
    {
      "windowStart": "2025-11-10T12:00:00Z",
      "totalCount": 701,
      "onlineCount": 700
    },
    {
      "windowStart": "2025-11-12T00:00:00Z",
      "totalCount": 689,
      "onlineCount": 676
    }
  ]
}

digitalfrank · December 7, 2025, 5:21pm

Confirmed yet. All my nodes are 0 audits/48 hours

Alexey · December 8, 2025, 2:12am

The script doesn’t show the last contact, it shows when was a discrepancy between a total amount of audits and answered audits.
To see when was the last audit you need to request a full history without a filter, i.e.

PowerShell

foreach ($item in ((Invoke-WebRequest http://localhost:14002/api/sno).Content | ConvertFrom-Json).satellites.id) {
    ((Invoke-WebRequest http://localhost:14002/api/sno/satellite/$item).Content | ConvertFrom-Json).auditHistory.windows | 
        Where-Object{$_.totalCount -ne 0} | ForEach-Object{Write-Host $item; $_ | Format-Table -AutoSize}
}

bash

for item in `curl -sL http://localhost:14002/api/sno | jq '.satellites[].id' -r`; do
    curl -s http://localhost:14002/api/sno/satellite/$item | \
    jq '{id: .id, auditHistory: [.auditHistory.windows[] | select(.totalCount != 0]}'
done

Please do not post full results , only when the last window was for each satellite

Alexey · December 8, 2025, 2:19am

I shared this with the team as soon as saw the first post Something happened with audits? - #33 by hwm.land, however, there are weekends.

Aleksman4o · December 8, 2025, 4:59am

saltlake - last contact 2025-11-10
ap1 - last contact 2025-11-18
us1 - last contact 2025-11-19
eu1 - last contact 2025-11-19

Alexey · December 8, 2025, 7:20am

So you do not have any audit windows since these dates or have you used the script to show only when there was a discrepancy?

agente · December 8, 2025, 7:21am

Same problems here. These are not isolated cases

Aleksman4o · December 8, 2025, 7:23am

It’s a new version of your bash (there missed bracket in select(.totalCount != 0)). No audits from that dates were shown.

Alexey · December 8, 2025, 7:27am

Yes, this is what I wanted to confirm.
Shared this maybe related issue too.

EddieA · December 8, 2025, 7:08pm

Same here for me…

Toyoo · December 9, 2025, 1:09am

Yep, I can confirm as well, 2025-11-19 being the last date. Inconvenient, as I started some tinkering with hashtbls and now I don’t know if I haven’t erased the whole node already

hwm.land · December 9, 2025, 6:09am

Just note: audits are back (yesterday ~16:00 UTC). Thanks.
Edit: it seems that I have audits (~3k node1, ~2k node2) just from US1. Other nodes still 0B → 0 audits (I suppose)

Alexey · December 9, 2025, 6:37am

What about online checks? Do you have an audit history back too?

hwm.land · December 9, 2025, 7:15am

Do you mean http://storj1.internal:14001/api/sno/satellite/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S ?
Here my ‘last’ window is "windowStart": "2025-11-19T00:00:00Z" even for US1 node

Alexey · December 9, 2025, 7:27am

Yes, so online checkers are not restored yet. Shared with the team, thank you!

RecklessD · December 9, 2025, 8:15am

Still no Audits here, or online checks.

Tue Dec  9 07:09:55 PM AEDT 2025
Number of GET_AUDIT - 2025-12-1 : 0
Number of GET_AUDIT - 2025-12-2 : 0
Number of GET_AUDIT - 2025-12-3 : 0
Number of GET_AUDIT - 2025-12-4 : 0
Number of GET_AUDIT - 2025-12-5 : 0
Number of GET_AUDIT - 2025-12-6 : 0
Number of GET_AUDIT - 2025-12-7 : 0
Number of GET_AUDIT - 2025-12-8 : 0
Number of GET_AUDIT - 2025-12-9 : 0

alpharabbit · December 9, 2025, 1:30pm

Date       |  GET_AUDIT | GET_REPAIR | PUT_REPAIR |        GET |        PUT
===========+============+============+============+============+============
2025-11-30 |     154835 |     574405 |     121308 |    5567007 |    1206143
2025-12-01 |     163012 |     322076 |      69666 |    5423152 |    1204811
2025-12-02 |      49387 |     729818 |     132703 |    4755813 |    1097049
2025-12-03 |      58736 |     564826 |     126548 |    4553236 |    1318987
2025-12-04 |      50614 |     998196 |     228868 |    4677372 |    1104113
2025-12-05 |        116 |     622224 |     126139 |    4243282 |    1196342
2025-12-06 |          0 |     747880 |     205778 |    4418055 |    1436886
2025-12-07 |          0 |     420862 |      92182 |    4628472 |    1341749
2025-12-08 |      10866 |     385003 |      87192 |    5270981 |    1188083
2025-12-09 |      39242 |     223974 |      49832 |    3203257 |     877202

Andrew · December 11, 2025, 3:50am

Same for me. 0 audits since 2025-11-19

therealmatrix · December 14, 2025, 8:55pm

i had an serverfault about 1.5 month ago. but my onlinescore not better again.

with the script it shows this.

{
“id”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”,
“auditHistory”: [
{
“windowStart”: “2025-10-14T12:00:00Z”,
“totalCount”: 419,
“onlineCount”: 415
},
{
“windowStart”: “2025-10-20T12:00:00Z”,
“totalCount”: 546,
“onlineCount”: 316
},
{
“windowStart”: “2025-10-21T00:00:00Z”,
“totalCount”: 435,
“onlineCount”: 185
},
{
“windowStart”: “2025-10-26T00:00:00Z”,
“totalCount”: 483,
“onlineCount”: 470
},
{
“windowStart”: “2025-11-01T12:00:00Z”,
“totalCount”: 421,
“onlineCount”: 411
}
]
}
{
“id”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”,
“auditHistory”: [
{
“windowStart”: “2025-10-20T12:00:00Z”,
“totalCount”: 310,
“onlineCount”: 220
},
{
“windowStart”: “2025-10-21T00:00:00Z”,
“totalCount”: 509,
“onlineCount”: 92
},
{
“windowStart”: “2025-10-22T12:00:00Z”,
“totalCount”: 374,
“onlineCount”: 338
},
{
“windowStart”: “2025-10-27T00:00:00Z”,
“totalCount”: 658,
“onlineCount”: 656
},
{
“windowStart”: “2025-11-01T12:00:00Z”,
“totalCount”: 523,
“onlineCount”: 490
}
]
}
{
“id”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”,
“auditHistory”: [
{
“windowStart”: “2025-10-20T12:00:00Z”,
“totalCount”: 732,
“onlineCount”: 731
},
{
“windowStart”: “2025-10-21T00:00:00Z”,
“totalCount”: 3502,
“onlineCount”: 1669
},
{
“windowStart”: “2025-10-28T12:00:00Z”,
“totalCount”: 3221,
“onlineCount”: 3220
},
{
“windowStart”: “2025-11-01T12:00:00Z”,
“totalCount”: 3000,
“onlineCount”: 2999
},
{
“windowStart”: “2025-11-02T00:00:00Z”,
“totalCount”: 3067,
“onlineCount”: 3052
},
{
“windowStart”: “2025-11-08T12:00:00Z”,
“totalCount”: 3073,
“onlineCount”: 3072
},
{
“windowStart”: “2025-11-09T00:00:00Z”,
“totalCount”: 2515,
“onlineCount”: 2514
}
]
}
{
“id”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”,
“auditHistory”: [
{
“windowStart”: “2025-10-21T00:00:00Z”,
“totalCount”: 1345,
“onlineCount”: 667
},
{
“windowStart”: “2025-10-22T12:00:00Z”,
“totalCount”: 694,
“onlineCount”: 686
},
{
“windowStart”: “2025-10-29T00:00:00Z”,
“totalCount”: 1397,
“onlineCount”: 1396
},
{
“windowStart”: “2025-11-01T12:00:00Z”,
“totalCount”: 994,
“onlineCount”: 934
},
{
“windowStart”: “2025-11-10T12:00:00Z”,
“totalCount”: 780,
“onlineCount”: 779
}
]
}

there are still the entries from october.