Interesting that other satellites show different online scores. It seems that you are having connection problems.
- Do you run antivirus on the same computer that you run the node?
- Is your router acting up because of too many connections?
Are you sure it is disqualified and not suspended?
The disqualification is permanent unlike suspension.
yea, online score is weird. for me US2 is causing issue on two or my 8 nodes. score is going down only on it with no downtime or internet issues.
I just hope that other nodes will still work fine even if US2 stops working. I rarely get anything from it anyway. saltlake and north europe is the same, but at least they dont lower online score.
those 99.X are legit, since I rebooted server few times when I was adding more disks and was making sure everything will come up and designing my deployment.
So, hm… I just noticed today that all of my 8 nodes got disqualified on Saltlake:
This is apparently because my node has been seen as offline for too long:
I should have noticed this earlier, but I don’t check my nodes’ dashboards that often, and because we don’t get e-mail notifications, I missed it 20 days ago:
They all run version v1.70.2, and are seen as 100% online on all other satellites.
Besides, Uptimerobot did not pick up any issue with my nodes which are seen as perfectly online for the past 30 days:
(I’m not sure if it’s related or not, but some of them show a misconfigured QUIC, did not investigate why so far, except all port redirections and docker commands which seem fine)
Any idea what could be going on? 
Maybe related to this?
Unlikely. Online score 0% means that node was offline too long.
But I would ask the team.
Are they all on the same IP and/or ISP?
Are they behind the same firewall/router?
Same IP and ISP, yes.
Also same router.
Basically, they are all installed on the same Raspberry Pi 4B connected to my ISP’s box, here at my place.
Here are the latest logs on one of them (but there are thousands), that seem related to this:
[...]
2023-01-26T05:27:25.466Z ERROR contact:service ping satellite failed {"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "attempts": 12, "error": "ping satellite: check-in network: failed to resolve IP from address: [MY-DDNS-DOMAIN-NAME]:1267, err: lookup [MY-DDNS-DOMAIN-NAME] on [AN-IPv4-ADDRESS]: server misbehaving", "errorVerbose": "ping satellite: check-in network: failed to resolve IP from address: [MY-DDNS-DOMAIN-NAME]:1267, err: lookup [MY-DDNS-DOMAIN-NAME] on [AN-IPv4-ADDRESS]: server misbehaving\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:139\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:100\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:160\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75"}
2023-01-26T06:01:38.858Z ERROR contact:service ping satellite failed {"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "attempts": 1, "error": "ping satellite: check-in network: failed to resolve IP from address: [MY-DDNS-DOMAIN-NAME]:1267, err: lookup [MY-DDNS-DOMAIN-NAME] on [AN-IPv4-ADDRESS]: server misbehaving", "errorVerbose": "ping satellite: check-in network: failed to resolve IP from address: [MY-DDNS-DOMAIN-NAME]:1267, err: lookup [MY-DDNS-DOMAIN-NAME] on [AN-IPv4-ADDRESS]: server misbehaving\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:139\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:100\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:160\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75"}
2023-01-26T06:01:45.170Z ERROR contact:service ping satellite failed {"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "attempts": 2, "error": "ping satellite: check-in network: failed to resolve IP from address: [MY-DDNS-DOMAIN-NAME]:1267, err: lookup [MY-DDNS-DOMAIN-NAME] on [AN-IPv4-ADDRESS]: server misbehaving", "errorVerbose": "ping satellite: check-in network: failed to resolve IP from address: [MY-DDNS-DOMAIN-NAME]:1267, err: lookup [MY-DDNS-DOMAIN-NAME] on [AN-IPv4-ADDRESS]: server misbehaving\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:139\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:100\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:160\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75"}
2023-01-26T06:01:52.532Z ERROR contact:service ping satellite failed {"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "attempts": 3, "error": "ping satellite: check-in network: failed to resolve IP from address: [MY-DDNS-DOMAIN-NAME]:1267, err: lookup [MY-DDNS-DOMAIN-NAME] on [AN-IPv4-ADDRESS]: server misbehaving", "errorVerbose": "ping satellite: check-in network: failed to resolve IP from address: [MY-DDNS-DOMAIN-NAME]:1267, err: lookup [MY-DDNS-DOMAIN-NAME] on [AN-IPv4-ADDRESS]: server misbehaving\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:139\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:100\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:160\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75"}
2023-01-26T06:02:01.911Z ERROR contact:service ping satellite failed {"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "attempts": 4, "error": "ping satellite: check-in network: failed to resolve IP from address: [MY-DDNS-DOMAIN-NAME]:1267, err: lookup [MY-DDNS-DOMAIN-NAME] on [AN-IPv4-ADDRESS]: server misbehaving", "errorVerbose": "ping satellite: check-in network: failed to resolve IP from address: [MY-DDNS-DOMAIN-NAME]:1267, err: lookup [MY-DDNS-DOMAIN-NAME] on [AN-IPv4-ADDRESS]: server misbehaving\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:139\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:100\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:160\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75"}
2023-01-26T06:02:15.222Z ERROR contact:service ping satellite failed {"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "attempts": 5, "error": "ping satellite: check-in network: failed to resolve IP from address: [MY-DDNS-DOMAIN-NAME]:1267, err: lookup [MY-DDNS-DOMAIN-NAME] on [AN-IPv4-ADDRESS]: server misbehaving", "errorVerbose": "ping satellite: check-in network: failed to resolve IP from address: [MY-DDNS-DOMAIN-NAME]:1267, err: lookup [MY-DDNS-DOMAIN-NAME] on [AN-IPv4-ADDRESS]: server misbehaving\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:139\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:100\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:160\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75"}
2023-01-26T06:02:36.580Z ERROR contact:service ping satellite failed {"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "attempts": 6, "error": "ping satellite: check-in network: failed to resolve IP from address: [MY-DDNS-DOMAIN-NAME]:1267, err: lookup [MY-DDNS-DOMAIN-NAME] on [AN-IPv4-ADDRESS]: server misbehaving", "errorVerbose": "ping satellite: check-in network: failed to resolve IP from address: [MY-DDNS-DOMAIN-NAME]:1267, err: lookup [MY-DDNS-DOMAIN-NAME] on [AN-IPv4-ADDRESS]: server misbehaving\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:139\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:100\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:160\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75"}
2023-01-26T06:10:43.469Z ERROR contact:service ping satellite failed {"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "attempts": 10, "error": "ping satellite: check-in network: failed to resolve IP from address: [MY-DDNS-DOMAIN-NAME]:1267, err: lookup [MY-DDNS-DOMAIN-NAME] on [AN-IPv4-ADDRESS]: server misbehaving", "errorVerbose": "ping satellite: check-in network: failed to resolve IP from address: [MY-DDNS-DOMAIN-NAME]:1267, err: lookup [MY-DDNS-DOMAIN-NAME] on [AN-IPv4-ADDRESS]: server misbehaving\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:139\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:100\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:160\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75"}
2023-01-26T06:19:20.824Z ERROR contact:service ping satellite failed {"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "attempts": 11, "error": "ping satellite: check-in network: failed to resolve IP from address: [MY-DDNS-DOMAIN-NAME]:1267, err: lookup [MY-DDNS-DOMAIN-NAME] on [AN-IPv4-ADDRESS]: server misbehaving", "errorVerbose": "ping satellite: check-in network: failed to resolve IP from address: [MY-DDNS-DOMAIN-NAME]:1267, err: lookup [MY-DDNS-DOMAIN-NAME] on [AN-IPv4-ADDRESS]: server misbehaving\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:139\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:100\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:160\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75"}
2023-01-26T06:36:28.139Z ERROR contact:service ping satellite failed {"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "attempts": 12, "error": "ping satellite: check-in network: failed to resolve IP from address: [MY-DDNS-DOMAIN-NAME]:1267, err: lookup [MY-DDNS-DOMAIN-NAME] on [AN-IPv4-ADDRESS]: server misbehaving", "errorVerbose": "ping satellite: check-in network: failed to resolve IP from address: [MY-DDNS-DOMAIN-NAME]:1267, err: lookup [MY-DDNS-DOMAIN-NAME] on [AN-IPv4-ADDRESS]: server misbehaving\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:139\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:100\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:160\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75"}
Note: When I ping [MY-DDNS-DOMAIN-NAME] from my Raspberry Pi, I don’t get the IP address [AN-IPv4-ADDRESS] that appears in the logs above.
This isn’t the first time it was only the saltlake satellite which had an ‘Online’ issue with a node behind a domain name, which needed to be resolved. Could DNS resolution or cache on the satellite be an issue?
Then please wait before deleting its data, I’ll notify the team and we let you know is it satellite-related issue or not.
@Pac in your case I found the record based on the port number. Looks like your DNS name is not very reliable. I tried to resolve it, sometimes it works, sometimes it doesn’t. (Even if I use 1.1.1.1 or 8.8.8.8 dns servers).
Looks to be a problem on this dns provider.
We checked our DNS servers and everything was fine.
I would suggest to use different host names. (no-ip.com and dynv6.com were recommended for me.)
If you changed your hostname, ping me, and I can fix the DQ state for you.
3 Likes
Oh nice!
I’ll PM you with the details.
i had major downtime 2-3 months ago, yes, more than 5h (1 or 2 days)
My guess is the node had no chance to recover, because in the next 30 days after that im sure there was some 5 minut, or few hours downtime, almost everymonth theres some downtime, even 5 minutes, theres always something, because its private, home computers, theres always high risk of few hours downtime in a given month, just like yesterday, the windows 10 updated something and halted internet like he always do, (windows 10 is a ransomewere in fact, i would’t have any node on win10 today knowing that, but 2-3 years ago that was the only windows for storj nodes)
Saying that i haven’t checked anything yet, i woul be considered, because @thebadcat says its still goes down, and his node is online if i understand correctly? but forget it for a moment, and see this: yet that alone, what You are saying, is almost impossible condition to meet. Sure there are months there are no down time. But those are RARE, because again its home computer, sure i have uptime monitoring, but i often see after few minutes the fastest)
So what you are saying is: after some downtime like 5h, or 1 day, the node have 30 days to be 100% online at given satelite in order to start recovering? to start gaining % of downtime back?, its almost impossible i must say, especialy on windows 10, theres very very high risk there will be 5 min, or 30min or few hours downtime in a month. I want to alarm You please, reconsider changes, because look at screens of my nodes, it was very much online, but wasn’t able to recover on saltlake, and other satelites are fine. Give better chanse for online node to recover.
EDIT:
just look at this, this node got saltlake to 0% online, and got us1 at 100%, and this is his uptimerobot:
https://imgur.com/a/kORhjn7
EDIT2: its total downtime is ~48h over period of 3 months, and saltlake went to 0%!
its highly unjust!
No, what Alexei is saying is that it will take 30 days for the outage to clear the dashboard metric.
If your node is online and without error, it will slowly recover. It is alright if your node is down for maintenance so long as it isn’t too long. As you mentioned, everyone’s network or OS needs updates or has power fluctuations.
If your node is down too often, however, it can be disqualified. Just try and keep it online and running properly and you should be fine.
1 Like
its been disqualified on saltlake.
With only 48h down time, (for last 3 months)
I guess 48h was too often.
Your screenshot is showing 90% and is not DQ’d or suspended. Your node would not be disqualified for 48hrs of down time.