Reputation error

Node Operators troubleshooting
1

Started seeing the following errors on 2 of my nodes:

2025-09-25T03:05:52Z    INFO    reputation:service      node scores updated     {"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Total Audits": 72678, "Successful Audits": 70976, "Audit Score": "NaN", "Online Score": 0.8748070515389789, "Suspension Score": "NaN", "Audit Score Delta": "NaN", "Online Score Delta": 0, "Suspension Score Delta": "NaN"}
2025-09-25T03:05:52Z    ERROR   reputation:chore        failed to store reputation      {"Process": "storagenode", "error": "reputation: NOT NULL constraint failed: reputation.audit_reputation_score", "errorVerbose": "reputation: NOT NULL constraint failed: reputation.audit_reputation_score\n\tstorj.io/storj/storagenode/storagenodedb.(*reputationDB).Store:101\n\tstorj.io/storj/storagenode/reputation.(*Service).Store:72\n\tstorj.io/storj/storagenode/reputation.(*Chore).GetAndCacheStats:118\n\tstorj.io/storj/storagenode/reputation.(*Chore).RunOnce:81\n\tstorj.io/storj/storagenode/reputation.(*Chore).Run.func1:61\n\tstorj.io/common/sync2.(*Cycle).Run:163\n\tstorj.io/storj/storagenode/reputation.(*Chore).Run:56\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:51\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).add.func1:130"}
2025-09-25T03:05:52Z    INFO    reputation:service      node scores updated     {"Process": "storagenode", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Total Audits": 21554, "Successful Audits": 21286, "Audit Score": 1, "Online Score": 0.8647058823529411, "Suspension Score": 1, "Audit Score Delta": 0, "Online Score Delta": 0, "Suspension Score Delta": 0}
2025-09-25T03:05:53Z    WARN    reputation:service      node scores worsened    {"Process": "storagenode", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Total Audits": 540296, "Successful Audits": 510922, "Audit Score": 1, "Online Score": 0.8757167679571498, "Suspension Score": 1, "Audit Score Delta": 0, "Online Score Delta": -0.0001393534002229746, "Suspension Score Delta": 0}
2025-09-25T03:05:53Z    INFO    reputation:service      node scores updated     {"Process": "storagenode", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Total Audits": 33100, "Successful Audits": 33028, "Audit Score": 1, "Online Score": 0.8649337997141444, "Suspension Score": 1, "Audit Score Delta": 0, "Online Score Delta": 0, "Suspension Score Delta": 0}
2025-09-25T03:05:53Z    ERROR   reputation:chore        reputation chore failed {"Process": "storagenode", "error": "reputation: NOT NULL constraint failed: reputation.audit_reputation_score", "errorVerbose": "reputation: NOT NULL constraint failed: reputation.audit_reputation_score\n\tstorj.io/storj/storagenode/storagenodedb.(*reputationDB).Store:101\n\tstorj.io/storj/storagenode/reputation.(*Service).Store:72\n\tstorj.io/storj/storagenode/reputation.(*Chore).GetAndCacheStats:118\n\tstorj.io/storj/storagenode/reputation.(*Chore).RunOnce:81\n\tstorj.io/storj/storagenode/reputation.(*Chore).Run.func1:61\n\tstorj.io/common/sync2.(*Cycle).Run:163\n\tstorj.io/storj/storagenode/reputation.(*Chore).Run:56\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:51\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).add.func1:130"}

Any ideas as to what’s going on.

Cheers.

1 Like
2

I found today same thing in logs.

3

Node version? Operating system?

4

Both on v1.137.5

One on Rocky Linux 9.6, the other TrueNAS SCALE, which is based on Debian 12.9.

Cheers.

5

version 1.137.5 , windows

6

Same here, 1.137.5


2025-09-25T10:15:58Z	ERROR	reputation:chore	reputation chore failed	{"Process": "storagenode", "error": "reputation: NOT NULL constraint failed: reputation.audit_reputation_score", "errorVerbose": "reputation: NOT NULL constraint failed: reputation.audit_reputation_score\n\tstorj.io/storj/storagenode/storagenodedb.(*reputationDB).Store:101\n\tstorj.io/storj/storagenode/reputation.(*Service).Store:72\n\tstorj.io/storj/storagenode/reputation.(*Chore).GetAndCacheStats:118\n\tstorj.io/storj/storagenode/reputation.(*Chore).RunOnce:81\n\tstorj.io/storj/storagenode/reputation.(*Chore).Run.func1:61\n\tstorj.io/common/sync2.(*Cycle).Run:163\n\tstorj.io/storj/storagenode/reputation.(*Chore).Run:56\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:51\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).add.func1:130"}
7

I wonder if this caused the high amount of repair traffic my nodes showing today? :thinking:

8

Now seeing them on a v1.136.4 node.

Cheers.

10

please dont make doble topiks on same thing, then you will get something like this, this mean some one flaged it.

12

Looks to be related to an experiment in Saltlake.

Supposed to be fixed from now…

4 Likes
13

Seeing a lot of repair upload here, too.
Seems our nodes are to slow? :grimacing:

14

I was seeing 2 errors each time. One was Saltlake, the other EU1.

Cheers.