I have several nodes running in different locations, and on each node I’m seeing a constant decline in the online score only for the eu1.storj.io satellite since September 25.
I’ve already checked the logs but haven’t found any issues. All other satellites are working fine.
What could be the reason for this, and how can I investigate it further?
1 Like
I had the same on two nodes. A restart of those nodes appears to have fixed for me.
I already try the restart but didn’t change anything
Hello @brizio71,
Welcome back!
Please, use these scripts to figure out when your nodes missed audits:
Then you can check logs of your firewall/router/nodes on these dates.
If you use a DDNS service like ASUS one or DuckDNS, then please switch to NoIP or Cloudflare.
The online score would recover after 30 days online.
I have the same (or very similar) thing too - on all my four nodes (made up from two hosts, one public IP)
I only have uptime robot on one:
From storjnet.info:
API output logged via zabbix:
This is affecting all four nodes similarly, but only for eu1
Nothing in my router or firewall logs, DDNS by freddns.org (although I’ve been on the same IP for more than a year)
I got exactly the same 
restart node didnt do the trick for me….
Same on my nodes, set to direct IP. Only issues on EU1, nodes in central USA.
Tagged the team on this issue. Maybe something gets wrong with the online checkers of EU1.
Hi @Alexey any news from the team ? the problem is still present justo on EU1 satellite and get worst day by day for each node on different locations and different public IP while all other satellites works fine. thanks
Nothing unusual so far.
Could you please try to use IP instead of DDNS?
And you need to check by the scripts above did it help or not, because the online score has an inertia and change would be noticeable only after a week or so.
As soon as I can, I will run the script. In the meantime, I can say that I don’t use DDNS; instead, I map my public IP to a standard DNS record on my domain — the same setup applies to all nodes and the satellites
This issue occurs only on EU1. What is the difference between this satellite and the others?
They are in different geographic locations, uses different channels and magistral providers, etc.
It could be possible, that your DNS provider doesn’t properly respond on requests or there are routing issues somewhere in between the auditors, DNS provider, your node. Too many if’s. So, I would like to suggest to change to IP and check - does it help to increase the amount of responded audits?
It could also help, if you can provide the result of mtr eu1.storj.io from the host with the node.
I’ve checked my nodes config and I already use the IP instead on DNS record on my nodes, then I’ve also tried the DNS resolution and I get:
Non-authoritative answer:
Name: eu1.storj.io
Address: 35.207.121.91
this is the output of the script for one of my node, what I can see ?
{
“id”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”,
“auditHistory”:
}
{
“id”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”,
“auditHistory”: [
{
“windowStart”: “2025-09-23T00:00:00Z”,
“totalCount”: 3289,
“onlineCount”: 3259
},
{
“windowStart”: “2025-09-26T00:00:00Z”,
“totalCount”: 1194,
“onlineCount”: 1188
},
{
“windowStart”: “2025-10-03T12:00:00Z”,
“totalCount”: 2960,
“onlineCount”: 2948
}
]
}
{
“id”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”,
“auditHistory”: [
{
“windowStart”: “2025-09-09T00:00:00Z”,
“totalCount”: 3081,
“onlineCount”: 3077
},
{
“windowStart”: “2025-09-23T00:00:00Z”,
“totalCount”: 6255,
“onlineCount”: 6105
},
{
“windowStart”: “2025-09-26T00:00:00Z”,
“totalCount”: 6052,
“onlineCount”: 6044
},
{
“windowStart”: “2025-10-03T12:00:00Z”,
“totalCount”: 5859,
“onlineCount”: 5803
}
]
}
{
“id”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”,
“auditHistory”: [
{
“windowStart”: “2025-09-23T00:00:00Z”,
“totalCount”: 1856,
“onlineCount”: 1824
},
{
“windowStart”: “2025-09-23T12:00:00Z”,
“totalCount”: 7103,
“onlineCount”: 7093
},
{
“windowStart”: “2025-09-24T00:00:00Z”,
“totalCount”: 16516,
“onlineCount”: 16507
},
{
“windowStart”: “2025-09-24T12:00:00Z”,
“totalCount”: 6369,
“onlineCount”: 6339
},
{
“windowStart”: “2025-09-25T00:00:00Z”,
“totalCount”: 1517,
“onlineCount”: 1508
},
{
“windowStart”: “2025-09-25T12:00:00Z”,
“totalCount”: 585,
“onlineCount”: 580
},
{
“windowStart”: “2025-09-26T00:00:00Z”,
“totalCount”: 729,
“onlineCount”: 726
},
{
“windowStart”: “2025-09-26T12:00:00Z”,
“totalCount”: 908,
“onlineCount”: 900
},
{
“windowStart”: “2025-09-27T00:00:00Z”,
“totalCount”: 1285,
“onlineCount”: 1277
},
{
“windowStart”: “2025-09-27T12:00:00Z”,
“totalCount”: 1934,
“onlineCount”: 1925
},
{
“windowStart”: “2025-09-28T00:00:00Z”,
“totalCount”: 1411,
“onlineCount”: 1406
},
{
“windowStart”: “2025-09-28T12:00:00Z”,
“totalCount”: 959,
“onlineCount”: 944
},
{
“windowStart”: “2025-09-29T00:00:00Z”,
“totalCount”: 1140,
“onlineCount”: 1137
},
{
“windowStart”: “2025-09-30T00:00:00Z”,
“totalCount”: 1606,
“onlineCount”: 1600
},
{
“windowStart”: “2025-09-30T12:00:00Z”,
“totalCount”: 680,
“onlineCount”: 677
},
{
“windowStart”: “2025-10-01T00:00:00Z”,
“totalCount”: 600,
“onlineCount”: 599
},
{
“windowStart”: “2025-10-02T00:00:00Z”,
“totalCount”: 1555,
“onlineCount”: 1527
},
{
“windowStart”: “2025-10-02T12:00:00Z”,
“totalCount”: 857,
“onlineCount”: 840
},
{
“windowStart”: “2025-10-03T00:00:00Z”,
“totalCount”: 20,
“onlineCount”: 19
},
{
“windowStart”: “2025-10-03T12:00:00Z”,
“totalCount”: 2552,
“onlineCount”: 2492
},
{
“windowStart”: “2025-10-04T00:00:00Z”,
“totalCount”: 2130,
“onlineCount”: 2124
},
{
“windowStart”: “2025-10-04T12:00:00Z”,
“totalCount”: 1370,
“onlineCount”: 1356
},
{
“windowStart”: “2025-10-05T00:00:00Z”,
“totalCount”: 1955,
“onlineCount”: 1927
},
{
“windowStart”: “2025-10-05T12:00:00Z”,
“totalCount”: 1299,
“onlineCount”: 1293
},
{
“windowStart”: “2025-10-06T00:00:00Z”,
“totalCount”: 657,
“onlineCount”: 651
},
{
“windowStart”: “2025-10-06T12:00:00Z”,
“totalCount”: 880,
“onlineCount”: 870
}
]
}
Now you can check logs of your router/firewall to see, what was an issue on these dates and times.
If you also have some “smart” or DDoS protection on your router - please disable them.
I already checked my Fortigate logs and didn’t see any dropped or rejected packets. I also checked my firewall rules, and there are no UTM services defined in those rules.
Exactly same thing here: since 25 or 26 September slow decline (around -0.02% each day) on online score for EU1, all other sats remain completely stable.
Location: Madrid
ISP: Movistar
DDNS: Dynu.com
I am monitoring my node with Hetrix and I am not detecting any errors, even from Frankfurt where I suspect EU1 is located.
Thank you for an additional information. Could you please post mtr results? You may send me via DM or submit a support request on https://support.storj.io
The problem continues for me too. Nothing of any significance in the router or firewall logs, DoS protection disabled for some weeks now after it was raised elsewhere on this forum.
Zabbix logged output from the API:
I use Pi-hole for my local DNS, so I queried the log there to see if DNS resolution on the node was a problem:
SELECT domain, type, status, client, reply_type, COUNT(*), ROUND(AVG(reply_time),2), ROUND(MAX(reply_time),2), forward
FROM queries
WHERE
(client = '192.168.16.42' OR client = '192.168.16.106')
AND (DATETIME(timestamp,'unixepoch') > '2025-09-30 11:00:00')
AND domain LIKE '%1.storj.io'
GROUP BY domain, client, status, reply_type
ORDER BY domain, client, status, reply_type;
domain ! type ! status ! client ! reply_type ! COUNT(*) ! ROUND(AVG(reply_time),2) ! ROUND(MAX(reply_time),2) ! forward
ap1.storj.io ! 1 ! 2 ! 192.168.16.106 ! 0 ! 8 ! ! ! 77.109.128.2#53
ap1.storj.io ! 2 ! 2 ! 192.168.16.106 ! 1 ! 166 ! 0.03 ! 0.78 ! 77.109.128.2#53
ap1.storj.io ! 1 ! 2 ! 192.168.16.106 ! 4 ! 201 ! 0.03 ! 0.74 ! 77.109.128.2#53
ap1.storj.io ! 2 ! 3 ! 192.168.16.106 ! 1 ! 230 ! 0.0 ! 0.0 !
ap1.storj.io ! 1 ! 3 ! 192.168.16.106 ! 4 ! 195 ! 0.0 ! 0.0 !
ap1.storj.io ! 1 ! 2 ! 192.168.16.42 ! 0 ! 1 ! ! ! 213.144.129.20#53
ap1.storj.io ! 2 ! 2 ! 192.168.16.42 ! 1 ! 358 ! 0.02 ! 0.29 ! 77.109.128.2#53
ap1.storj.io ! 1 ! 2 ! 192.168.16.42 ! 4 ! 455 ! 0.02 ! 0.29 ! 77.109.128.2#53
ap1.storj.io ! 2 ! 3 ! 192.168.16.42 ! 1 ! 845 ! 0.0 ! 0.0 !
ap1.storj.io ! 1 ! 3 ! 192.168.16.42 ! 4 ! 747 ! 0.0 ! 0.0 !
eu1.storj.io ! 1 ! 2 ! 192.168.16.106 ! 0 ! 8 ! ! ! 77.109.128.2#53
eu1.storj.io ! 2 ! 2 ! 192.168.16.106 ! 1 ! 165 ! 0.03 ! 0.78 ! 77.109.128.2#53
eu1.storj.io ! 1 ! 2 ! 192.168.16.106 ! 4 ! 203 ! 0.03 ! 0.78 ! 77.109.128.2#53
eu1.storj.io ! 2 ! 3 ! 192.168.16.106 ! 1 ! 231 ! 0.0 ! 0.0 !
eu1.storj.io ! 1 ! 3 ! 192.168.16.106 ! 4 ! 193 ! 0.0 ! 0.02 !
eu1.storj.io ! 1 ! 2 ! 192.168.16.42 ! 0 ! 1 ! ! ! 213.144.129.20#53
eu1.storj.io ! 2 ! 2 ! 192.168.16.42 ! 1 ! 354 ! 0.02 ! 0.29 ! 77.109.128.2#53
eu1.storj.io ! 1 ! 2 ! 192.168.16.42 ! 4 ! 455 ! 0.02 ! 0.29 ! 77.109.128.2#53
eu1.storj.io ! 2 ! 3 ! 192.168.16.42 ! 1 ! 845 ! 0.0 ! 0.0 !
eu1.storj.io ! 1 ! 3 ! 192.168.16.42 ! 4 ! 743 ! 0.0 ! 0.0 !
us1.storj.io ! 1 ! 2 ! 192.168.16.106 ! 0 ! 8 ! ! ! 77.109.128.2#53
us1.storj.io ! 2 ! 2 ! 192.168.16.106 ! 1 ! 166 ! 0.03 ! 0.78 ! 77.109.128.2#53
us1.storj.io ! 1 ! 2 ! 192.168.16.106 ! 4 ! 201 ! 0.03 ! 0.78 ! 77.109.128.2#53
us1.storj.io ! 2 ! 3 ! 192.168.16.106 ! 1 ! 233 ! 0.0 ! 0.0 !
us1.storj.io ! 1 ! 3 ! 192.168.16.106 ! 4 ! 198 ! 0.0 ! 0.0 !
us1.storj.io ! 1 ! 2 ! 192.168.16.42 ! 0 ! 1 ! ! ! 213.144.129.20#53
us1.storj.io ! 2 ! 2 ! 192.168.16.42 ! 1 ! 356 ! 0.02 ! 0.29 ! 77.109.128.2#53
us1.storj.io ! 1 ! 2 ! 192.168.16.42 ! 4 ! 455 ! 0.02 ! 0.29 ! 77.109.128.2#53
us1.storj.io ! 2 ! 3 ! 192.168.16.42 ! 1 ! 846 ! 0.0 ! 0.0 !
us1.storj.io ! 1 ! 3 ! 192.168.16.42 ! 4 ! 746 ! 0.0 ! 0.0 !
And here is the output from a short mtr:
Packets Pings
Host Loss% Snt Last Avg Best Wrst StDev
1. router7 0.0% 339 0.8 1.0 0.2 75.4 5.9
2. 109-202-218-1.init7.net 0.0% 339 0.9 1.2 0.6 85.8 4.6
3. r1zrh16.core.init7.net 0.0% 339 1.7 2.5 1.2 96.1 6.2
4. r1zrh2.core.init7.net 5.0% 339 1.8 2.2 1.2 50.5 3.0
5. r1zrh5.core.init7.net 0.0% 339 1.7 3.1 1.3 239.0 13.9
6. 5-180-134-47.init7.net 0.0% 339 7.3 10.8 6.2 196.7 17.0
7. ipv4.de-cix.fra.de.as15169.google.com 25.4% 338 28.8 9.4 6.2 330.8 23.8
8. 91.121.207.35.bc.googleusercontent.com 0.0% 338 7.1 8.8 6.4 189.1 15.2
Hope all that helps
Here is the mtr result to eu1.storj.io:
My traceroute [v0.95]
ubuntu1 (172.17.0.3) -> eu1.storj.io (35.207.121.91) 2025-10-07T17:37:40+0200
Keys: Help Display mode Restart statistics Order of fields quit
Packets Pings
Host Loss% Snt Last Avg Best Wrst StDev
1. 172.17.0.1 0.0% 229 0.1 0.1 0.1 0.7 0.0
2. 192.168.1.1 11.4% 228 0.5 0.6 0.4 2.0 0.3
3. 192.168.144.1 0.0% 228 1.9 2.8 1.1 18.8 2.0
4. 205.red-81-46-64.customer.static. 85.0% 228 1.9 3.0 1.5 15.0 2.3
5. (waiting for reply)
6. (waiting for reply)
7. ae-0-400-grtmadde3.net.telefonica 0.0% 228 3.4 3.5 2.0 23.8 2.1
8. 94.142.123.10 0.0% 228 23.5 24.0 22.2 47.9 2.5
9. 213.140.38.94 0.0% 228 32.5 33.0 31.4 44.2 1.7
10. 91.121.207.35.bc.googleuserconten 0.0% 228 32.1 33.5 32.0 52.7 1.9
I get simlar results when I mtr us1.storj.io:
My traceroute [v0.95]
ubuntu1 (172.17.0.3) -> us1.storj.io (35.212.10.183) 2025-10-07T17:49:19+0200
Keys: Help Display mode Restart statistics Order of fields quit
Packets Pings
Host Loss% Snt Last Avg Best Wrst StDev
1. 172.17.0.1 0.0% 40 0.1 0.1 0.1 0.1 0.0
2. 192.168.1.1 10.0% 40 0.6 0.6 0.4 1.9 0.3
3. 192.168.144.1 0.0% 40 1.3 2.1 1.3 3.1 0.5
4. 205.red-81-46-64.customer.static. 85.0% 40 1.9 2.3 1.9 2.8 0.4
5. (waiting for reply)
6. (waiting for reply)
7. ae-0-400-grtmadde3.net.telefonica 0.0% 39 3.0 3.8 2.2 22.8 3.2
8. 213.140.37.153 0.0% 39 28.1 28.5 26.7 45.3 2.9
9. 94.142.127.10 0.0% 39 108.9 109.0 107.4 113.5 1.2
10. 190.98.141.25 0.0% 39 101.0 102.1 100.8 104.3 0.9
11. 183.10.212.35.bc.googleuserconten 0.0% 39 105.4 105.6 104.1 113.7 2.1
I repeated the test after some minutes and the 10%+ packet loss in my local router was no longer there.