Bug in new Vetted Date

I got the email that the node is unsuspended, but the dashboard still shows the message with supended; maybe it will go away someday.

It should be. However, I think there is a combination of Suspended and Offline. And you got a confirmation, that now your node is unsuspended because being online.. If I’m right you should receive a confirmation of unsuspended because of a suspension score is now greater than 60% :person_facepalming:

Apparently, time depends on ingress and with it on the number of audits. Waiting already for 4 hours, checked the correctness of email 15 times, nothing yet ))
P.s: after 5 hours got node offline email and stopped experiment: Suspension 95.25 %, too slow.

I’m happy it helped! Though, honestly I didn’t expect it to be that fast. This probably means some other parameters of your node were at risk already, yet not communicated to your node correctly. This is actually quite alarming, how many other nodes are considered by the system as healthly, even if they aren’t?

With the focus Storj has had on performance: I’m sure they’ll get this fixed quickly. If there are many nodes that should be considered for traffic, but aren’t (because of mistakenly still being unvetted)… then customer data isn’t being spread across as many nodes as it could be (and some of them are certainly fast SNOs).

As far as I know there is no vetting for select nodes. So maybe we need a community PR to get this fixed… :wink:

Interesting… you did the steps I described, right? Not shuting down the entire node, only blacklisting the unvetted one?
I imagine it depends on the number of nodes under the same /24 network that are unvetted and the data gattered so far. I have around 300GB from EU1.

The node is strong, it dosen’t seem to have any problems. The scores are great, the system is Ubuntu server+Docker, Exos 22TB, 32 RAM, N100 cpu. Nothing else is runing on it. And the suspension score already recovered. I think the explanation is what Alexey said: the sat is seeing the node online, but because it dosen’t respond to audits, it is considering that it’s ill.

Still no info from the company when this is getting fixed? I am not experimenting or shutting down my nodes hoping to fix it.

You posted here twice: on a Saturday and Sunday. Are you not receiving the after-hours weekend support you expected?. Just chill :squinting_face_with_tongue:

3 Likes

Thanks to snorkel’s experiment we know more about the problem. No need to do it again, I think. If the hypothesis that the bug comes from missing updates to the nodes table is true, Storj engineers will be able to update that table and fix any existing cases with no intervention by node operators. So please just wait a bit more.

Sorry, wasn’t my intention to sound impatient. I just skimmed through messages posted here over the weekend, and seeing experimental user-induced workarounds I was a bit worried, so I decided to ask again.
I am not messing around with production software running 24/7 on a homeserver, that’s for sure. Company is paying for it, company will fix it, that’s my opinion.

1 Like

It’s not like it’s putting any customer data at risk.

It must respond on audits with the unknown error to get suspension score to drop and being suspended for that.
If it wouldn’t respond, then the satellite will consider it offline instead and the online score will drop.

Two different suspension reasons, they are independent too.

1 Like

Yup. I Started that thread and nowt has changed. Have 12 nodes, got fobbed off with β€œit’s just a luck thing” and yet, STILL, I have the same delta of bandwidth… Same IP, same drive models, same host. And now, funilly enough, the node that’s lagging shows unvetted on EU1.

However earnings.py shows…

Payout and held amount by satellite:
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ SATELLITE                      β”‚ HELD AMOUNT β”‚        REPUTATION        β”‚                       PAYOUT THIS MONTH                     β”‚
β”‚              Joined     Month  β”‚      Total  β”‚    Disq    Susp    Down  β”‚    Storage      Egress  Repair/Aud        Held      Payout  β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ ap1.storj.io:7777 (OK)         β”‚             β”‚                          β”‚  $  1.49/TBm $  2.00/TB  $  2.00/TB        0%        100%   β”‚
β”‚              2024-10-27    10  β”‚   $   0.78  β”‚   0.00%   0.00%   0.68%  β”‚  $  0.2125   $  0.0139   $  0.0956  -$  0.0000   $  0.3220  β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ eu1.storj.io:7777 (OK)         β”‚             β”‚                          β”‚  $  1.49/TBm $  2.00/TB  $  2.00/TB        0%        100%   β”‚
β”‚              2024-10-27    10  β”‚   $   0.72  β”‚   0.00%   0.00%   0.48%  β”‚  $  0.1827   $  0.0074   $  0.1025  -$  0.0000   $  0.2926  β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ saltlake.tardigrade.io:7777 (OK)             β”‚                          β”‚  $  1.49/TBm $  2.00/TB  $  2.00/TB        0%        100%   β”‚
β”‚              2024-10-27    10  β”‚   $   0.00  β”‚   0.00%   0.00%   0.58%  β”‚  $  0.0004   $  0.0001   $  0.0002  -$  0.0000   $  0.0008  β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ us1.storj.io:7777 (OK)         β”‚             β”‚                          β”‚  $  1.49/TBm $  2.00/TB  $  2.00/TB        0%        100%   β”‚
β”‚              2024-10-27    10  β”‚   $   9.97  β”‚   0.00%   0.00%   0.35%  β”‚  $  3.0464   $  0.2502   $  0.1185  -$  0.0000   $  3.4151  β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€ +
β”‚ TOTAL                          β”‚   $  11.47  β”‚                          β”‚  $  3.4420   $  0.2716   $  0.3168  -$  0.0000   $  4.0305  β”‚
β”‚ ESTIMATED END OF MONTH TOTAL   β”‚   $  11.47  β”‚                          β”‚  $  5.2050   $  0.4057   $  0.4733  -$  0.0000   $  6.0840  β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

I think we need a re-verify of the vetting!!!

1 Like

Don’t read, just write? ))

Anyway, @snorkel method is working. Repaired today 11 nodes with unvetted satellites. For some of them i’ve got offline email instead of suspend after ~4-5 hours, so i’ve unblacklisted, restared and blacklisted them again. After 2-3 iterations suspend was received and now they are vetted and almost healthy. Ingress slightly increased )

4 Likes

Just to make sure, have you updated earnings.py to this version?

Nope, i used 14.1.0. The master branch?

Where did 14.2.0 come from?

It’s not yet in master, check this post:

Ah yes - 610 posts in…

July 2025 (Version: 14.2.0)                                             [snapshot: 2025-07-21 20:14:00Z]
REPORTED BY     TYPE      METRIC                PRICE                     DISK  BANDWIDTH        PAYOUT
Node            Ingress   Upload                -not paid-                      577.34 GB
Node            Ingress   Upload Repair         -not paid-                       26.34 GB
Node            Egress    Download              $  2.00 / TB (avg)              136.71 GB       $  0.27
Node            Egress    Download Repair       $  2.00 / TB (avg)              158.81 GB       $  0.32
Node            Egress    Download Audit        $  2.00 / TB (avg)               10.16 MB       $  0.00
Node            Storage   Disk Current Total    -not paid-             3.57 TB
Node            Storage              β”œ Blobs    -not paid-             3.48 TB
Node            Storage              β”” Trash  ┐ -not paid-            93.35 GB
Node+Sat. Calc. Storage   Uncollected Garbage ─ -not paid-            28.44 GB
Node+Sat. Calc. Storage   Total Unpaid Data <β”€β”˜ -not paid-           121.79 GB
Satellite       Storage   Disk Last Report      -not paid-             3.45 TB
Satellite       Storage   Disk Average So Far   -not paid-             3.38 TB
Satellite       Storage   Disk Usage Month      $  1.49 / TBm (avg)    2.31 TBm                 $  3.44
________________________________________________________________________________________________________+
Total                                                                  2.31 TBm 899.21 GB       $  4.03
Estimated total by end of month                                        3.38 TBm   1.34 TB       $  6.08

Payout and held amount by satellite:
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ SATELLITE                      β”‚ HELD AMOUNT β”‚        REPUTATION        β”‚                       PAYOUT THIS MONTH                     β”‚
β”‚              Joined     Month  β”‚      Total  β”‚    Disq    Susp    Down  β”‚    Storage      Egress  Repair/Aud        Held      Payout  β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ ap1.storj.io:7777 (OK)         β”‚             β”‚                          β”‚  $  1.49/TBm $  2.00/TB  $  2.00/TB        0%        100%   β”‚
β”‚              2024-10-27    10  β”‚   $   0.78  β”‚   0.00%   0.00%   0.68%  β”‚  $  0.2125   $  0.0140   $  0.0959  -$  0.0000   $  0.3223  β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ eu1.storj.io:7777 (Vetting result pending > 29144/100 Audits)           β”‚  $  1.49/TBm $  2.00/TB  $  2.00/TB        0%        100%   β”‚
β”‚              2024-10-27    10  β”‚   $   0.72  β”‚   0.00%   0.00%   0.48%  β”‚  $  0.1827   $  0.0074   $  0.1027  -$  0.0000   $  0.2928  β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ saltlake.tardigrade.io:7777 (OK)             β”‚                          β”‚  $  1.49/TBm $  2.00/TB  $  2.00/TB        0%        100%   β”‚
β”‚              2024-10-27    10  β”‚   $   0.00  β”‚   0.00%   0.00%   0.58%  β”‚  $  0.0004   $  0.0001   $  0.0002  -$  0.0000   $  0.0008  β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ us1.storj.io:7777 (OK)         β”‚             β”‚                          β”‚  $  1.49/TBm $  2.00/TB  $  2.00/TB        0%        100%   β”‚
β”‚              2024-10-27    10  β”‚   $   9.97  β”‚   0.00%   0.00%   0.35%  β”‚  $  3.0464   $  0.2519   $  0.1188  -$  0.0000   $  3.4171  β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€ +
β”‚ TOTAL                          β”‚   $  11.47  β”‚                          β”‚  $  3.4420   $  0.2734   $  0.3176  -$  0.0000   $  4.0331  β”‚
β”‚ ESTIMATED END OF MONTH TOTAL   β”‚   $  11.47  β”‚                          β”‚  $  5.2050   $  0.4067   $  0.4724  -$  0.0000   $  6.0841  β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜



So there you have it. We were all gaslighted.

For some reason this was hidden as β€œinappropriate”. Given some of the hate I see posted on these boards, could someone explain what was inappropriate.

I’m stating facts.

The first line was where the announcement for the new version was in the thread - 610 posts in.

The next bit is my result, proving that the validation is borked.

The last statement is a fact - Despite the 14.1.0 tool saying I’m validated, and me and others saying that EU1 satellite was not liking some of my nodes and that there was a issue, we were all told β€œit’s just luck” and β€œit’ll level out”.

Given some of the posters tones in these forums, I’m struggling to understand what β€œinappropriate”.

1 Like