Disqualified on US-1

Node Operators troubleshooting
1

Ok, this node had some troubles with connections, but I hoped I could solve it without getting disqualified. But it seems it happened. But from the dashboard I don’t know why:

banner_disqualification
banner_disqualification1145×149 17.2 KB

dashboard_metrics
dashboard_metrics1209×429 36.2 KB

The last log line for that satellite is

INFO	reputation:service	node scores updated	{"process": "storagenode", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Total Audits": 672401, "Successful Audits": 653796, "Audit Score": 0.9977554834976379, "Online Score": 0.8319589196735543, "Suspension Score": 1, "Audit Score Delta": 0, "Online Score Delta": 0, "Suspension Score Delta": 0}

The remaining logs I have only show connection issues with that satellite, no piece errors and compared to the other satellites it does not look so bad.
So the reason for disqualification is a bit unclear to me.

2

The only explanation I can come up with is that the node got an audit and failed to respond to it in a timely manner (it’s what, 5 minutes?). Online or not doesn’t disqualify you with those percentages, I’ve had nodes that fell well below 70% that were not disqualified.

2 Likes
3

That’s what I have thought.
I have no audit request /failed audit in the logs for that satellite.
This is weird.

4

Any packet loss between your node and US1 (all of them, aren’t they clustered?)?

5

Not as I can see.

But as I said this node had connection troubles.
I have lots of these:

ERROR	contact:service	ping satellite failed

and
at the and many of them even in a row:

2024-04-02T10:00:36Z	INFO	reputation:service	node scores updated	{"process": "storagenode", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Total Audits": 672401, "Successful Audits": 653796, "Audit Score": 0.9977554834976379, "Online Score": 0.8319589196735543, "Suspension Score": 1, "Audit Score Delta": 0, "Online Score Delta": 0, "Suspension Score Delta": 0}
2024-04-02T13:59:26Z	INFO	reputation:service	node scores updated	{"process": "storagenode", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Total Audits": 672401, "Successful Audits": 653796, "Audit Score": 0.9977554834976379, "Online Score": 0.8319589196735543, "Suspension Score": 1, "Audit Score Delta": 0, "Online Score Delta": 0, "Suspension Score Delta": 0}
2024-04-02T17:57:11Z	INFO	reputation:service	node scores updated	{"process": "storagenode", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Total Audits": 672401, "Successful Audits": 653796, "Audit Score": 0.9977554834976379, "Online Score": 0.8319589196735543, "Suspension Score": 1, "Audit Score Delta": 0, "Online Score Delta": 0, "Suspension Score Delta": 0}
2024-04-02T22:00:22Z	INFO	reputation:service	node scores updated	{"process": "storagenode", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Total Audits": 672401, "Successful Audits": 653796, "Audit Score": 0.9977554834976379, "Online Score": 0.8319589196735543, "Suspension Score": 1, "Audit Score Delta": 0, "Online Score Delta": 0, "Suspension Score Delta": 0}
2024-04-03T02:00:04Z	INFO	reputation:service	node scores updated	{"process": "storagenode", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Total Audits": 672401, "Successful Audits": 653796, "Audit Score": 0.9977554834976379, "Online Score": 0.8319589196735543, "Suspension Score": 1, "Audit Score Delta": 0, "Online Score Delta": 0, "Suspension Score Delta": 0}
2024-04-03T05:59:08Z	INFO	reputation:service	node scores updated	{"process": "storagenode", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Total Audits": 672401, "Successful Audits": 653796, "Audit Score": 0.9977554834976379, "Online Score": 0.8319589196735543, "Suspension Score": 1, "Audit Score Delta": 0, "Online Score Delta": 0, "Suspension Score Delta": 0}

But also nothing in between. (No up- or downloads or anything)

6

What does an mtr between that node and us1.storj.io show? Please redact IPs.

7

I think you have to wait another day, than the audit scores will be updated and probably turn out to be below 96% for the us1 satellite. Also happened to me, less than a week ago on the EU-satellite after directory structure of the EU-folder turned out to be damaged. Although it turned out te be repairable with fsck, I found out after I was already disqualified on that specific satellite. Looking backwards in dmesg it apparently happened in less than half a day the errors turned up. But because it was a quite young node, it didn’t have that many audits before the file system corruption struck me. So only three audits were necessary for that disqualification.

8

Any DDOS protection in your router? US1 satellite is the biggest one, so the first one to give terrible if it’s related to the amount of data stored on your node.

9 
										Loss%   Snt   Last   Avg  Best  Wrst StDev
1. 										0.0%    14    0.5   1.1   0.4   4.5   1.2
2. 										0.0%    13    6.1  12.3   5.5  30.4   8.9
3. 										7.7%    13    7.0   7.4   6.1  11.3   1.5
4. 										0.0%    13    7.5   8.0   6.1  14.0   2.7
5. 										0.0%    13    6.8   7.7   6.5  10.0   1.2
6. 										0.0%    13    6.8   7.1   6.7   8.2   0.4
7. 										0.0%    13    6.3   6.4   6.2   7.2   0.3
8. 48.199.150.34.bc.googleusercontent.com  0.0%    13   99.4  99.8  98.9 102.2   1.1

Scores are still the same like in the picture.
Something must have had happened before the time my logs start.

10

What’s hop 3? There is 7.7% loss there. Your ISP?

11

Number 7 is the first hop that leaves the ISPs domain.

12

I doubt that, unless they are directly connected to the storj satellite (the next hop is the satellite). The last point of exit should be either 3 or 4.

13

Here is an mtr from within google cloud (the satellite is on google cloud):

 Host                                                                                                                        Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. (waiting for reply)
 2.                                                                                                                            0.0%    16    0.6   1.4   0.5   6.0   1.5
 3.                                                                                                                            0.0%    16   71.6  51.4   2.9 165.8  46.3
 4.                                                                                                                           66.7%    16   42.2  42.1  41.9  42.2   0.2
 5. (waiting for reply)
 6.                                                                                                                            6.2%    16   56.2  56.5  56.0  58.9   0.7
 7.                                                                                                                            0.0%    16   55.7  55.7  55.6  56.6   0.2
 8.                                                                                                                           93.3%    16   57.0  57.0  57.0  57.0   0.0
 9.                                                                                                                           93.3%    16   56.1  56.1  56.1  56.1   0.0
10. (waiting for reply)
11. (waiting for reply)
12. (waiting for reply)
13. (waiting for reply)
14. (waiting for reply)
15. (waiting for reply)
16. (waiting for reply)
17. 48.199.150.34.bc.googleusercontent.com                                                                                    0.0%    15   55.3  55.4  55.3  56.0   0.2
14

Well, that’s what it is showing. According to WHOIS is the IP at hop 7 owned by my ISP.

@Alexey
Any idea what could be the reason for disqualification? Do you agree that the stats should not be the reason for it?

15

The disqualification usually happens only for these reasons:

  1. Pieces are not available for audit (any reason like read timeout, disk is unreadable/corrupted, permissions are wrong, disk is disconnected, etc.).
  2. Pieces are corrupted.
  3. Node was offline for more than 30 days.

So either pieces are lost (or unavailable) or corrupted.
In the first case you may have some errors in the logs but usually not for timeouts. For the second case there will be no audit errors (to do not give an idea to a malicious user to build a strategy how to bypass audits), all pieces for audits will be successfully downloaded (and this is true), but the satellite doesn’t report the audit result to the node, only in the order with reputation scores.
However, if the node is disqualified before it updated with a new stat, it wouldn’t likely receive any updates from that satellite anymore.
I suspect that your node is in the list of nodes with weirdly corrupted pieces like there: Проблема: Your node has been disqualified, но audit score 96% - #10 by vladro
I could only suggest to try to use this tool to check pieces on the node:

It doesn’t requalify your node, but at least you can make sure that you do not have (or have) corrupted pieces.

1 Like
16
17

I’m not convinced, that this is useful. You already have an information about disqualification. And may request when is it happened.
What is this feature should solve?