I got the email that the node is unsuspended, but the dashboard still shows the message with supended; maybe it will go away someday.
It should be. However, I think there is a combination of Suspended and Offline. And you got a confirmation, that now your node is unsuspended because being online.. If Iβm right you should receive a confirmation of unsuspended because of a suspension score is now greater than 60%
Apparently, time depends on ingress and with it on the number of audits. Waiting already for 4 hours, checked the correctness of email 15 times, nothing yet ))
P.s: after 5 hours got node offline email and stopped experiment: Suspension 95.25 %, too slow.
Iβm happy it helped! Though, honestly I didnβt expect it to be that fast. This probably means some other parameters of your node were at risk already, yet not communicated to your node correctly. This is actually quite alarming, how many other nodes are considered by the system as healthly, even if they arenβt?
With the focus Storj has had on performance: Iβm sure theyβll get this fixed quickly. If there are many nodes that should be considered for traffic, but arenβt (because of mistakenly still being unvetted)β¦ then customer data isnβt being spread across as many nodes as it could be (and some of them are certainly fast SNOs).
As far as I know there is no vetting for select nodes. So maybe we need a community PR to get this fixedβ¦
Interesting⦠you did the steps I described, right? Not shuting down the entire node, only blacklisting the unvetted one?
I imagine it depends on the number of nodes under the same /24 network that are unvetted and the data gattered so far. I have around 300GB from EU1.
The node is strong, it dosenβt seem to have any problems. The scores are great, the system is Ubuntu server+Docker, Exos 22TB, 32 RAM, N100 cpu. Nothing else is runing on it. And the suspension score already recovered. I think the explanation is what Alexey said: the sat is seeing the node online, but because it dosenβt respond to audits, it is considering that itβs ill.
Still no info from the company when this is getting fixed? I am not experimenting or shutting down my nodes hoping to fix it.
You posted here twice: on a Saturday and Sunday. Are you not receiving the after-hours weekend support you expected?. Just chill
Thanks to snorkelβs experiment we know more about the problem. No need to do it again, I think. If the hypothesis that the bug comes from missing updates to the nodes table is true, Storj engineers will be able to update that table and fix any existing cases with no intervention by node operators. So please just wait a bit more.
Sorry, wasnβt my intention to sound impatient. I just skimmed through messages posted here over the weekend, and seeing experimental user-induced workarounds I was a bit worried, so I decided to ask again.
I am not messing around with production software running 24/7 on a homeserver, thatβs for sure. Company is paying for it, company will fix it, thatβs my opinion.
Itβs not like itβs putting any customer data at risk.
It must respond on audits with the unknown error to get suspension score to drop and being suspended for that.
If it wouldnβt respond, then the satellite will consider it offline instead and the online score will drop.
Two different suspension reasons, they are independent too.
Yup. I Started that thread and nowt has changed. Have 12 nodes, got fobbed off with βitβs just a luck thingβ and yet, STILL, I have the same delta of bandwidthβ¦ Same IP, same drive models, same host. And now, funilly enough, the node thatβs lagging shows unvetted on EU1.
However earnings.py showsβ¦
Payout and held amount by satellite:
ββββββββββββββββββββββββββββββββββ¬ββββββββββββββ¬βββββββββββββββββββββββββββ¬ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β SATELLITE β HELD AMOUNT β REPUTATION β PAYOUT THIS MONTH β
β Joined Month β Total β Disq Susp Down β Storage Egress Repair/Aud Held Payout β
ββββββββββββββββββββββββββββββββββΌββββββββββββββΌβββββββββββββββββββββββββββΌββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
β ap1.storj.io:7777 (OK) β β β $ 1.49/TBm $ 2.00/TB $ 2.00/TB 0% 100% β
β 2024-10-27 10 β $ 0.78 β 0.00% 0.00% 0.68% β $ 0.2125 $ 0.0139 $ 0.0956 -$ 0.0000 $ 0.3220 β
ββββββββββββββββββββββββββββββββββΌββββββββββββββΌβββββββββββββββββββββββββββΌββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
β eu1.storj.io:7777 (OK) β β β $ 1.49/TBm $ 2.00/TB $ 2.00/TB 0% 100% β
β 2024-10-27 10 β $ 0.72 β 0.00% 0.00% 0.48% β $ 0.1827 $ 0.0074 $ 0.1025 -$ 0.0000 $ 0.2926 β
ββββββββββββββββββββββββββββββββββΌββββββββββββββΌβββββββββββββββββββββββββββΌββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
β saltlake.tardigrade.io:7777 (OK) β β $ 1.49/TBm $ 2.00/TB $ 2.00/TB 0% 100% β
β 2024-10-27 10 β $ 0.00 β 0.00% 0.00% 0.58% β $ 0.0004 $ 0.0001 $ 0.0002 -$ 0.0000 $ 0.0008 β
ββββββββββββββββββββββββββββββββββΌββββββββββββββΌβββββββββββββββββββββββββββΌββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
β us1.storj.io:7777 (OK) β β β $ 1.49/TBm $ 2.00/TB $ 2.00/TB 0% 100% β
β 2024-10-27 10 β $ 9.97 β 0.00% 0.00% 0.35% β $ 3.0464 $ 0.2502 $ 0.1185 -$ 0.0000 $ 3.4151 β
ββββββββββββββββββββββββββββββββββΌββββββββββββββΌβββββββββββββββββββββββββββΌββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€ +
β TOTAL β $ 11.47 β β $ 3.4420 $ 0.2716 $ 0.3168 -$ 0.0000 $ 4.0305 β
β ESTIMATED END OF MONTH TOTAL β $ 11.47 β β $ 5.2050 $ 0.4057 $ 0.4733 -$ 0.0000 $ 6.0840 β
ββββββββββββββββββββββββββββββββββ΄ββββββββββββββ΄βββββββββββββββββββββββββββ΄ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
I think we need a re-verify of the vetting!!!
Donβt read, just write? ))
Anyway, @snorkel method is working. Repaired today 11 nodes with unvetted satellites. For some of them iβve got offline email instead of suspend after ~4-5 hours, so iβve unblacklisted, restared and blacklisted them again. After 2-3 iterations suspend was received and now they are vetted and almost healthy. Ingress slightly increased )
Just to make sure, have you updated earnings.py to this version?
Nope, i used 14.1.0. The master branch?
Where did 14.2.0 come from?
Itβs not yet in master, check this post:
Ah yes - 610 posts inβ¦
July 2025 (Version: 14.2.0) [snapshot: 2025-07-21 20:14:00Z]
REPORTED BY TYPE METRIC PRICE DISK BANDWIDTH PAYOUT
Node Ingress Upload -not paid- 577.34 GB
Node Ingress Upload Repair -not paid- 26.34 GB
Node Egress Download $ 2.00 / TB (avg) 136.71 GB $ 0.27
Node Egress Download Repair $ 2.00 / TB (avg) 158.81 GB $ 0.32
Node Egress Download Audit $ 2.00 / TB (avg) 10.16 MB $ 0.00
Node Storage Disk Current Total -not paid- 3.57 TB
Node Storage β Blobs -not paid- 3.48 TB
Node Storage β Trash β -not paid- 93.35 GB
Node+Sat. Calc. Storage Uncollected Garbage β€ -not paid- 28.44 GB
Node+Sat. Calc. Storage Total Unpaid Data <ββ -not paid- 121.79 GB
Satellite Storage Disk Last Report -not paid- 3.45 TB
Satellite Storage Disk Average So Far -not paid- 3.38 TB
Satellite Storage Disk Usage Month $ 1.49 / TBm (avg) 2.31 TBm $ 3.44
________________________________________________________________________________________________________+
Total 2.31 TBm 899.21 GB $ 4.03
Estimated total by end of month 3.38 TBm 1.34 TB $ 6.08
Payout and held amount by satellite:
ββββββββββββββββββββββββββββββββββ¬ββββββββββββββ¬βββββββββββββββββββββββββββ¬ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β SATELLITE β HELD AMOUNT β REPUTATION β PAYOUT THIS MONTH β
β Joined Month β Total β Disq Susp Down β Storage Egress Repair/Aud Held Payout β
ββββββββββββββββββββββββββββββββββΌββββββββββββββΌβββββββββββββββββββββββββββΌββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
β ap1.storj.io:7777 (OK) β β β $ 1.49/TBm $ 2.00/TB $ 2.00/TB 0% 100% β
β 2024-10-27 10 β $ 0.78 β 0.00% 0.00% 0.68% β $ 0.2125 $ 0.0140 $ 0.0959 -$ 0.0000 $ 0.3223 β
ββββββββββββββββββββββββββββββββββΌββββββββββββββΌβββββββββββββββββββββββββββΌββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
β eu1.storj.io:7777 (Vetting result pending > 29144/100 Audits) β $ 1.49/TBm $ 2.00/TB $ 2.00/TB 0% 100% β
β 2024-10-27 10 β $ 0.72 β 0.00% 0.00% 0.48% β $ 0.1827 $ 0.0074 $ 0.1027 -$ 0.0000 $ 0.2928 β
ββββββββββββββββββββββββββββββββββΌββββββββββββββΌβββββββββββββββββββββββββββΌββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
β saltlake.tardigrade.io:7777 (OK) β β $ 1.49/TBm $ 2.00/TB $ 2.00/TB 0% 100% β
β 2024-10-27 10 β $ 0.00 β 0.00% 0.00% 0.58% β $ 0.0004 $ 0.0001 $ 0.0002 -$ 0.0000 $ 0.0008 β
ββββββββββββββββββββββββββββββββββΌββββββββββββββΌβββββββββββββββββββββββββββΌββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
β us1.storj.io:7777 (OK) β β β $ 1.49/TBm $ 2.00/TB $ 2.00/TB 0% 100% β
β 2024-10-27 10 β $ 9.97 β 0.00% 0.00% 0.35% β $ 3.0464 $ 0.2519 $ 0.1188 -$ 0.0000 $ 3.4171 β
ββββββββββββββββββββββββββββββββββΌββββββββββββββΌβββββββββββββββββββββββββββΌββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€ +
β TOTAL β $ 11.47 β β $ 3.4420 $ 0.2734 $ 0.3176 -$ 0.0000 $ 4.0331 β
β ESTIMATED END OF MONTH TOTAL β $ 11.47 β β $ 5.2050 $ 0.4067 $ 0.4724 -$ 0.0000 $ 6.0841 β
ββββββββββββββββββββββββββββββββββ΄ββββββββββββββ΄βββββββββββββββββββββββββββ΄ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
So there you have it. We were all gaslighted.
For some reason this was hidden as βinappropriateβ. Given some of the hate I see posted on these boards, could someone explain what was inappropriate.
Iβm stating facts.
The first line was where the announcement for the new version was in the thread - 610 posts in.
The next bit is my result, proving that the validation is borked.
The last statement is a fact - Despite the 14.1.0 tool saying Iβm validated, and me and others saying that EU1 satellite was not liking some of my nodes and that there was a issue, we were all told βitβs just luckβ and βitβll level outβ.
Given some of the posters tones in these forums, Iβm struggling to understand what βinappropriateβ.