So, the modular satellite uses a new connection pool to take advantage of caching… but it hands back connections that are… (link) ?
Exactly — Gattuso-grade connection pooling: sometimes maybe good, sometimes maybe shit. 
The real problem is that the reputation system treats “sometimes maybe shit” as “node offline”. A reachability check that can be answered from a cache isn’t a reachability check — which is precisely why the audit verifier already uses WithForceDial. The ping-back just needs to do the same.
Is there any hack SNOs can apply? Something like restart nodes frequently? 
Please note: Changing node parameters or using custom cron scripts is entirely at your own risk. I assume no liability whatsoever if these modifications cause your node to fail or break.
Blind restarts won’t help — the stale state lives on the satellite side. But your score only decays once last_contact_success is older than the 4h OnlineWindow, so the game is keeping that clock fresh:
- Lower
--contact.intervalfrom 1h to ~30 min (not below 15 — the satellite rate-limits check-ins to 2 per 10 min). - Restart conditionally: cron-check
lastPingedathttp://<node>:14002/api/snoand restart only if it’s older than ~3h — a restart forces an immediate check-in. - If you control your router: raise conntrack timeouts for established flows — the pooled connections die at your NAT.
External monitors, port checkers, DDNS changes: no effect, wrong path. And relax — suspension only below 60%, the 30-day window heals itself once this is fixed satellite-side.
What the hell is going on here? Chatbots argue wirh chatbots? Nobody can think and post their own thoughts anymore?
If people wanted to read chatbot opinion — chatbots are available to anyone, they could do it on their own.
At least credit the LLM you are using to dump this slop, along with the prompt, because you are clearly not prompting is correctly and cannot be arsed to corect you or argue the nonsense it generated.
Fair point, and happy to disclose: yes, I used an AI assistant (Fable) to help research this — i cloned the repo and I asked it to trace the ping/audit/reputation code paths. I reviewed what it found before posting.
But every claim is a file:line citation you can check yourself: satellite/mud.go:146 (pooled dialer in the modular satellite), common/rpc/dial.go:160 (pool key = node ID only), satellite/orders/service.go:519 (no order limits for overlay-offline nodes → AuditOffline without a dial). If any of those are wrong, point at the line and I’ll happily retract it.
Worth noting: the analysis supports your connection-pool theory from earlier in this thread — it just adds the receipts.
It has already cost me $35 to work with the Fable model here in the forum.
Seems so. But they managed to find an actual workaround 
Now I can open this address, and if you still have the “ping satellite failed” with that address, then it can be added to the first group. Do you have an Uptimerobot or similar external monitoring to check the same on the same time?
for example on
Because it may help developers to take a look on a specific service.
Online check on check-in is performed by a different service than online checkers, so it will be helpful to distinguish them.
Please also provide failure windows around these dates from the audits history using these scripts:
@akester thank you for providing more evidences, that the online check on check-in might be affected too.
Please provide the failure windows around the same dates from the audit history with the script above.
It will fail due to a load balancer configuration, it drops ICMP, so you can exclude this from the equation. You need to check dRPC or at least https/http (so it will use a TCP protocol).
@surfercool I already explained the same - there are two checkers - the online check on check-in (where you receive a “ping satellite failed”) and the online checkers - a separate service, based on audits, but they only check that the node answered or not (the response is processed by the audit checkers, so if the node responded with a known error, the audit score will be affected, if it didn’t respond at all - the online score will be affected, if it’s responded with an unknown error - the suspension score will be affected).
Before that moment I didn’t have an evidence, which provided by @akester and @CubeFan , that the error “ping satellite failed” also comes for the address and port, which can be opened externally. In your similar report the address and port didn’t respond, thus I cannot prove that this online check on check-in is affected too.
Unfortunately your AI agent is wrong - it’s affecting the online score.
And thank you for the provided analyze, it has been shared with the team.
This is related to the fact that ping-back on check-in is performed by the satellite, but audit checks might come from another IP(s), since it’s a separate distributed service.
@surfercool 's AI agent is explained the same.
I provided all feedbacks from this thread.
This is the same node with the log output I shared before:
❯ for item in `curl -sL http://storj.dmz.home.kester.cloud:14003/api/sno | jq '.satellites[].id' -r`; do
curl -s http://storj.dmz.home.kester.cloud:14003/api/sno/satellite/$item | \
jq '{id: .id, auditHistory: [.auditHistory.windows[] | select(.totalCount != .onlineCount)]}'
done
{
"id": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE",
"auditHistory": [
{
"windowStart": "2026-07-17T00:00:00Z",
"totalCount": 13,
"onlineCount": 12
},
{
"windowStart": "2026-07-17T12:00:00Z",
"totalCount": 20,
"onlineCount": 18
},
{
"windowStart": "2026-07-18T00:00:00Z",
"totalCount": 17,
"onlineCount": 15
},
{
"windowStart": "2026-07-18T12:00:00Z",
"totalCount": 9,
"onlineCount": 8
},
{
"windowStart": "2026-07-21T00:00:00Z",
"totalCount": 17,
"onlineCount": 13
},
{
"windowStart": "2026-07-26T00:00:00Z",
"totalCount": 12,
"onlineCount": 10
},
{
"windowStart": "2026-07-28T12:00:00Z",
"totalCount": 9,
"onlineCount": 8
}
]
}
{
"id": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6",
"auditHistory": [
{
"windowStart": "2026-07-01T12:00:00Z",
"totalCount": 245,
"onlineCount": 192
},
{
"windowStart": "2026-07-15T12:00:00Z",
"totalCount": 302,
"onlineCount": 294
},
{
"windowStart": "2026-07-16T00:00:00Z",
"totalCount": 349,
"onlineCount": 330
},
{
"windowStart": "2026-07-16T12:00:00Z",
"totalCount": 143,
"onlineCount": 138
},
{
"windowStart": "2026-07-17T00:00:00Z",
"totalCount": 383,
"onlineCount": 365
},
{
"windowStart": "2026-07-17T12:00:00Z",
"totalCount": 407,
"onlineCount": 392
},
{
"windowStart": "2026-07-18T00:00:00Z",
"totalCount": 399,
"onlineCount": 380
},
{
"windowStart": "2026-07-18T12:00:00Z",
"totalCount": 323,
"onlineCount": 314
},
{
"windowStart": "2026-07-19T00:00:00Z",
"totalCount": 279,
"onlineCount": 268
},
{
"windowStart": "2026-07-19T12:00:00Z",
"totalCount": 304,
"onlineCount": 295
},
{
"windowStart": "2026-07-20T00:00:00Z",
"totalCount": 281,
"onlineCount": 271
},
{
"windowStart": "2026-07-20T12:00:00Z",
"totalCount": 397,
"onlineCount": 378
},
{
"windowStart": "2026-07-21T00:00:00Z",
"totalCount": 129,
"onlineCount": 122
},
{
"windowStart": "2026-07-21T12:00:00Z",
"totalCount": 188,
"onlineCount": 183
},
{
"windowStart": "2026-07-22T00:00:00Z",
"totalCount": 291,
"onlineCount": 281
},
{
"windowStart": "2026-07-22T12:00:00Z",
"totalCount": 339,
"onlineCount": 331
},
{
"windowStart": "2026-07-23T00:00:00Z",
"totalCount": 316,
"onlineCount": 294
},
{
"windowStart": "2026-07-23T12:00:00Z",
"totalCount": 320,
"onlineCount": 307
},
{
"windowStart": "2026-07-24T00:00:00Z",
"totalCount": 319,
"onlineCount": 307
},
{
"windowStart": "2026-07-24T12:00:00Z",
"totalCount": 300,
"onlineCount": 286
},
{
"windowStart": "2026-07-25T00:00:00Z",
"totalCount": 367,
"onlineCount": 346
},
{
"windowStart": "2026-07-25T12:00:00Z",
"totalCount": 336,
"onlineCount": 325
},
{
"windowStart": "2026-07-26T00:00:00Z",
"totalCount": 298,
"onlineCount": 283
},
{
"windowStart": "2026-07-26T12:00:00Z",
"totalCount": 343,
"onlineCount": 330
},
{
"windowStart": "2026-07-27T00:00:00Z",
"totalCount": 329,
"onlineCount": 317
},
{
"windowStart": "2026-07-27T12:00:00Z",
"totalCount": 307,
"onlineCount": 287
},
{
"windowStart": "2026-07-28T00:00:00Z",
"totalCount": 327,
"onlineCount": 314
},
{
"windowStart": "2026-07-28T12:00:00Z",
"totalCount": 311,
"onlineCount": 303
},
{
"windowStart": "2026-07-29T00:00:00Z",
"totalCount": 49,
"onlineCount": 47
}
]
}
{
"id": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S",
"auditHistory": [
{
"windowStart": "2026-07-01T12:00:00Z",
"totalCount": 25,
"onlineCount": 21
},
{
"windowStart": "2026-07-15T12:00:00Z",
"totalCount": 693,
"onlineCount": 686
},
{
"windowStart": "2026-07-16T00:00:00Z",
"totalCount": 348,
"onlineCount": 331
},
{
"windowStart": "2026-07-16T12:00:00Z",
"totalCount": 348,
"onlineCount": 330
},
{
"windowStart": "2026-07-17T00:00:00Z",
"totalCount": 209,
"onlineCount": 200
},
{
"windowStart": "2026-07-17T12:00:00Z",
"totalCount": 256,
"onlineCount": 245
},
{
"windowStart": "2026-07-18T00:00:00Z",
"totalCount": 229,
"onlineCount": 216
},
{
"windowStart": "2026-07-18T12:00:00Z",
"totalCount": 263,
"onlineCount": 252
},
{
"windowStart": "2026-07-19T00:00:00Z",
"totalCount": 216,
"onlineCount": 210
},
{
"windowStart": "2026-07-19T12:00:00Z",
"totalCount": 177,
"onlineCount": 170
},
{
"windowStart": "2026-07-20T00:00:00Z",
"totalCount": 178,
"onlineCount": 173
},
{
"windowStart": "2026-07-20T12:00:00Z",
"totalCount": 26,
"onlineCount": 24
},
{
"windowStart": "2026-07-25T12:00:00Z",
"totalCount": 121,
"onlineCount": 116
},
{
"windowStart": "2026-07-26T00:00:00Z",
"totalCount": 130,
"onlineCount": 123
},
{
"windowStart": "2026-07-26T12:00:00Z",
"totalCount": 128,
"onlineCount": 122
},
{
"windowStart": "2026-07-27T00:00:00Z",
"totalCount": 115,
"onlineCount": 111
},
{
"windowStart": "2026-07-27T12:00:00Z",
"totalCount": 124,
"onlineCount": 117
},
{
"windowStart": "2026-07-28T00:00:00Z",
"totalCount": 109,
"onlineCount": 107
},
{
"windowStart": "2026-07-28T12:00:00Z",
"totalCount": 106,
"onlineCount": 102
}
]
}
{
"id": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs",
"auditHistory": [
{
"windowStart": "2026-07-16T12:00:00Z",
"totalCount": 69,
"onlineCount": 66
},
{
"windowStart": "2026-07-17T00:00:00Z",
"totalCount": 64,
"onlineCount": 61
},
{
"windowStart": "2026-07-17T12:00:00Z",
"totalCount": 49,
"onlineCount": 47
},
{
"windowStart": "2026-07-18T00:00:00Z",
"totalCount": 53,
"onlineCount": 51
},
{
"windowStart": "2026-07-18T12:00:00Z",
"totalCount": 70,
"onlineCount": 63
},
{
"windowStart": "2026-07-19T12:00:00Z",
"totalCount": 66,
"onlineCount": 65
},
{
"windowStart": "2026-07-20T00:00:00Z",
"totalCount": 54,
"onlineCount": 50
},
{
"windowStart": "2026-07-20T12:00:00Z",
"totalCount": 96,
"onlineCount": 91
},
{
"windowStart": "2026-07-21T00:00:00Z",
"totalCount": 62,
"onlineCount": 53
},
{
"windowStart": "2026-07-21T12:00:00Z",
"totalCount": 65,
"onlineCount": 62
},
{
"windowStart": "2026-07-22T00:00:00Z",
"totalCount": 60,
"onlineCount": 57
},
{
"windowStart": "2026-07-22T12:00:00Z",
"totalCount": 59,
"onlineCount": 56
},
{
"windowStart": "2026-07-23T00:00:00Z",
"totalCount": 52,
"onlineCount": 49
},
{
"windowStart": "2026-07-23T12:00:00Z",
"totalCount": 55,
"onlineCount": 53
},
{
"windowStart": "2026-07-24T00:00:00Z",
"totalCount": 66,
"onlineCount": 64
},
{
"windowStart": "2026-07-24T12:00:00Z",
"totalCount": 58,
"onlineCount": 55
},
{
"windowStart": "2026-07-25T00:00:00Z",
"totalCount": 51,
"onlineCount": 48
},
{
"windowStart": "2026-07-25T12:00:00Z",
"totalCount": 65,
"onlineCount": 64
},
{
"windowStart": "2026-07-26T00:00:00Z",
"totalCount": 56,
"onlineCount": 53
},
{
"windowStart": "2026-07-26T12:00:00Z",
"totalCount": 63,
"onlineCount": 61
},
{
"windowStart": "2026-07-27T00:00:00Z",
"totalCount": 53,
"onlineCount": 51
},
{
"windowStart": "2026-07-27T12:00:00Z",
"totalCount": 63,
"onlineCount": 59
},
{
"windowStart": "2026-07-28T00:00:00Z",
"totalCount": 48,
"onlineCount": 46
},
{
"windowStart": "2026-07-28T12:00:00Z",
"totalCount": 48,
"onlineCount": 46
}
]
}
Of note, the failures on July 1 were valid offline failures due to hardware issues on that day. Beyond that, the dates listed should have near 100% uptime.
Thanks Alexey — and fair correction on the online score point, I’ll stand corrected: the check-in doesn’t write the audit history directly, but it does affect the score (at minimum via the 4h online window → no audit order limits → offline bookings; noted that it’s more direct than that).
Re: my address not responding — my dynamic IP rotated (manual reconnect) since those log entries (you can see it change inside my own log: 91.35.24.59 → 91.35.29.55 → 91.35.13.135). My current address is <AKTUELLE-IP-ODER-DYNDNS>:28967, port is open and monitored externally (100% uptime).
Here are my audit history failure windows:
And my ping-back failure timestamps from the log for correlation: Jul 15 17:30Z, Jul 16 08:30Z / 10:30Z / 19:30Z / 20:30Z / 22:30Z, … — all attempts: 1 with the immediate retry succeeding, satellite taking ~135s to report the dial failure (OS connect timeout).
Here are my failure windows (full table below). Summary: starting exactly with the 2026-07-15T12:00Z window, every single 12h window on all four satellites contains offline bookings, at a strikingly uniform rate — SL 4.6%, AP1 4.0%, US1 3.5%, EU1 4.1% (median per window ~4.3% everywhere), around the clock. Before Jul 15 my history is essentially clean.
Correlation with my logs: I only see 1–3 “ping satellite failed” per day (always attempts: 1, the retry seconds later succeeds, satellite takes ~135s to report = OS connect timeout) — but the audit history books me offline ~100–150 times per day. So the separate online-checker service fails to reach me far more often than the check-in ping-back, invisible from the node side, while external monitoring shows 100%.
One honest exception: the Jul 25 00:00Z window (SL 18%, EU1 13%) coincides with my dynamic IP rotating twice that night — stale address until the next check-in, that burst is on my ISP, not you. The steady 4% baseline is not: a constant 1-in-25 loss rate, identical across four satellites with completely different network paths, 24/7, with a sharp onset on Jul 15 — that pattern points at the checker infrastructure, not at thousands of independent nodes.
== 1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE ==
windowStart totalCount onlineCount
2026-07-06T12:00:00Z 310 289
2026-07-15T12:00:00Z 351 346
2026-07-16T00:00:00Z 182 174
2026-07-16T12:00:00Z 240 231
2026-07-17T00:00:00Z 263 249
2026-07-17T12:00:00Z 483 459
2026-07-18T00:00:00Z 319 305
2026-07-18T12:00:00Z 235 227
2026-07-19T00:00:00Z 345 332
2026-07-19T12:00:00Z 288 274
2026-07-20T00:00:00Z 24 22
2026-07-20T12:00:00Z 326 311
2026-07-21T00:00:00Z 412 393
2026-07-21T12:00:00Z 375 359
2026-07-22T00:00:00Z 279 266
2026-07-22T12:00:00Z 268 257
2026-07-23T00:00:00Z 284 274
2026-07-23T12:00:00Z 414 396
2026-07-24T00:00:00Z 226 218
2026-07-24T12:00:00Z 163 159
2026-07-25T00:00:00Z 269 220
2026-07-25T12:00:00Z 292 278
2026-07-26T00:00:00Z 365 352
2026-07-26T12:00:00Z 335 320
2026-07-27T12:00:00Z 226 219
2026-07-28T00:00:00Z 326 315
2026-07-28T12:00:00Z 390 371
2026-07-29T00:00:00Z 31 30
== 121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6 ==
windowStart totalCount onlineCount
2026-06-28T12:00:00Z 100 99
2026-07-15T12:00:00Z 1712 1676
2026-07-16T00:00:00Z 1384 1335
2026-07-16T12:00:00Z 749 717
2026-07-17T00:00:00Z 1686 1614
2026-07-17T12:00:00Z 619 593
2026-07-18T00:00:00Z 179 169
2026-07-18T12:00:00Z 1220 1164
2026-07-19T00:00:00Z 1982 1894
2026-07-19T12:00:00Z 2239 2162
2026-07-20T00:00:00Z 2148 2054
2026-07-20T12:00:00Z 1132 1082
2026-07-21T00:00:00Z 2088 2004
2026-07-21T12:00:00Z 1358 1316
2026-07-22T00:00:00Z 1043 1007
2026-07-22T12:00:00Z 1228 1175
2026-07-23T00:00:00Z 955 905
2026-07-23T12:00:00Z 2143 2055
2026-07-24T00:00:00Z 1320 1266
2026-07-24T12:00:00Z 1087 1042
2026-07-25T00:00:00Z 487 468
2026-07-26T00:00:00Z 46 43
2026-07-26T12:00:00Z 2044 1964
2026-07-27T00:00:00Z 1988 1902
2026-07-27T12:00:00Z 393 368
2026-07-28T00:00:00Z 615 583
2026-07-28T12:00:00Z 411 398
== 12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S ==
windowStart totalCount onlineCount
2026-06-29T12:00:00Z 202 200
2026-07-15T12:00:00Z 3737 3704
2026-07-16T00:00:00Z 1856 1774
2026-07-16T12:00:00Z 667 639
2026-07-17T00:00:00Z 618 589
2026-07-17T12:00:00Z 1209 1165
2026-07-18T00:00:00Z 1249 1202
2026-07-18T12:00:00Z 1273 1226
2026-07-19T00:00:00Z 1209 1159
2026-07-19T12:00:00Z 880 834
2026-07-20T00:00:00Z 909 870
2026-07-20T12:00:00Z 100 95
2026-07-25T00:00:00Z 33 31
2026-07-25T12:00:00Z 645 613
2026-07-26T00:00:00Z 568 539
2026-07-26T12:00:00Z 608 584
2026-07-27T00:00:00Z 614 586
2026-07-27T12:00:00Z 589 572
2026-07-28T00:00:00Z 628 610
2026-07-28T12:00:00Z 633 599
2026-07-29T00:00:00Z 20 19
== 12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs ==
windowStart totalCount onlineCount
2026-06-28T12:00:00Z 1336 1330
2026-07-15T12:00:00Z 2854 2851
2026-07-16T00:00:00Z 512 492
2026-07-16T12:00:00Z 584 566
2026-07-17T00:00:00Z 1064 1022
2026-07-17T12:00:00Z 1141 1088
2026-07-18T00:00:00Z 1090 1044
2026-07-18T12:00:00Z 1123 1066
2026-07-19T00:00:00Z 1100 1056
2026-07-19T12:00:00Z 485 461
2026-07-20T00:00:00Z 699 662
2026-07-20T12:00:00Z 605 578
2026-07-21T00:00:00Z 472 450
2026-07-21T12:00:00Z 503 481
2026-07-22T00:00:00Z 533 511
2026-07-22T12:00:00Z 1080 1036
2026-07-23T00:00:00Z 451 432
2026-07-23T12:00:00Z 433 420
2026-07-24T00:00:00Z 620 593
2026-07-24T12:00:00Z 294 282
2026-07-25T00:00:00Z 1152 999
2026-07-25T12:00:00Z 1183 1136
2026-07-26T00:00:00Z 1086 1055
2026-07-26T12:00:00Z 484 464
2026-07-27T00:00:00Z 1070 1032
2026-07-27T12:00:00Z 301 290
2026-07-28T00:00:00Z 606 579
2026-07-28T12:00:00Z 358 337
Ok, thank you. This confirms that the online checkers have an issue to contact nodes, and when there is a true offline event (when IP is rotated), it affects the online score even more.
I think that the reason concluded by your AI agent can be the root cause.
I wasn’t aware of this issue, I don’t monitor my nodes or the forum like I used too, and I missed this thread. It seems that I too have this problem. I don’t use DDNS services, I pay for static IP, my 2 ISPs are reliable, my getway routers are mine, not ISP’s, nothing changed on my side. So Storj’s sats may be the culprit.
I don’t have time to investigate or the budget for AI, but what baffles me reading through this, is why do you use caches when the checks are suppose to check the current situation, not some cached passed state?
Here is the output of the script for node1:
Node1
cubefan@storj-chia01:~$ for item in `curl -sL http://192.168.188.92:14001/api/sno | jq '.satellites[].id' -r`; do
curl -s http://192.168.188.92:14001/api/sno/satellite/$item | \
jq '{id: .id, auditHistory: [.auditHistory.windows[] | select(.totalCount != .onlineCount)]}'
done
{
"id": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE",
"auditHistory": [
{
"windowStart": "2026-07-15T12:00:00Z",
"totalCount": 146,
"onlineCount": 144
},
{
"windowStart": "2026-07-16T00:00:00Z",
"totalCount": 134,
"onlineCount": 127
},
{
"windowStart": "2026-07-16T12:00:00Z",
"totalCount": 61,
"onlineCount": 60
},
{
"windowStart": "2026-07-17T00:00:00Z",
"totalCount": 87,
"onlineCount": 80
},
{
"windowStart": "2026-07-17T12:00:00Z",
"totalCount": 192,
"onlineCount": 184
},
{
"windowStart": "2026-07-18T00:00:00Z",
"totalCount": 84,
"onlineCount": 80
},
{
"windowStart": "2026-07-18T12:00:00Z",
"totalCount": 112,
"onlineCount": 108
},
{
"windowStart": "2026-07-19T00:00:00Z",
"totalCount": 135,
"onlineCount": 128
},
{
"windowStart": "2026-07-19T12:00:00Z",
"totalCount": 90,
"onlineCount": 89
},
{
"windowStart": "2026-07-20T00:00:00Z",
"totalCount": 77,
"onlineCount": 72
},
{
"windowStart": "2026-07-20T12:00:00Z",
"totalCount": 126,
"onlineCount": 120
},
{
"windowStart": "2026-07-21T00:00:00Z",
"totalCount": 127,
"onlineCount": 124
},
{
"windowStart": "2026-07-21T12:00:00Z",
"totalCount": 116,
"onlineCount": 112
},
{
"windowStart": "2026-07-22T00:00:00Z",
"totalCount": 82,
"onlineCount": 80
},
{
"windowStart": "2026-07-22T12:00:00Z",
"totalCount": 81,
"onlineCount": 77
},
{
"windowStart": "2026-07-23T00:00:00Z",
"totalCount": 105,
"onlineCount": 99
},
{
"windowStart": "2026-07-23T12:00:00Z",
"totalCount": 127,
"onlineCount": 119
},
{
"windowStart": "2026-07-24T00:00:00Z",
"totalCount": 116,
"onlineCount": 113
},
{
"windowStart": "2026-07-24T12:00:00Z",
"totalCount": 95,
"onlineCount": 90
},
{
"windowStart": "2026-07-25T00:00:00Z",
"totalCount": 91,
"onlineCount": 89
},
{
"windowStart": "2026-07-25T12:00:00Z",
"totalCount": 110,
"onlineCount": 101
},
{
"windowStart": "2026-07-26T00:00:00Z",
"totalCount": 118,
"onlineCount": 115
},
{
"windowStart": "2026-07-26T12:00:00Z",
"totalCount": 103,
"onlineCount": 98
},
{
"windowStart": "2026-07-27T12:00:00Z",
"totalCount": 94,
"onlineCount": 89
},
{
"windowStart": "2026-07-28T00:00:00Z",
"totalCount": 112,
"onlineCount": 107
},
{
"windowStart": "2026-07-28T12:00:00Z",
"totalCount": 120,
"onlineCount": 116
},
{
"windowStart": "2026-07-29T00:00:00Z",
"totalCount": 16,
"onlineCount": 15
}
]
}
{
"id": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6",
"auditHistory": [
{
"windowStart": "2026-07-15T12:00:00Z",
"totalCount": 1500,
"onlineCount": 1459
},
{
"windowStart": "2026-07-16T00:00:00Z",
"totalCount": 712,
"onlineCount": 676
},
{
"windowStart": "2026-07-16T12:00:00Z",
"totalCount": 351,
"onlineCount": 336
},
{
"windowStart": "2026-07-17T00:00:00Z",
"totalCount": 1056,
"onlineCount": 1022
},
{
"windowStart": "2026-07-17T12:00:00Z",
"totalCount": 1199,
"onlineCount": 1140
},
{
"windowStart": "2026-07-18T00:00:00Z",
"totalCount": 465,
"onlineCount": 446
},
{
"windowStart": "2026-07-18T12:00:00Z",
"totalCount": 952,
"onlineCount": 909
},
{
"windowStart": "2026-07-19T00:00:00Z",
"totalCount": 677,
"onlineCount": 628
},
{
"windowStart": "2026-07-19T12:00:00Z",
"totalCount": 1580,
"onlineCount": 1502
},
{
"windowStart": "2026-07-20T00:00:00Z",
"totalCount": 1439,
"onlineCount": 1371
},
{
"windowStart": "2026-07-20T12:00:00Z",
"totalCount": 707,
"onlineCount": 682
},
{
"windowStart": "2026-07-21T00:00:00Z",
"totalCount": 777,
"onlineCount": 741
},
{
"windowStart": "2026-07-21T12:00:00Z",
"totalCount": 600,
"onlineCount": 579
},
{
"windowStart": "2026-07-22T00:00:00Z",
"totalCount": 656,
"onlineCount": 633
},
{
"windowStart": "2026-07-22T12:00:00Z",
"totalCount": 1397,
"onlineCount": 1350
},
{
"windowStart": "2026-07-23T00:00:00Z",
"totalCount": 842,
"onlineCount": 808
},
{
"windowStart": "2026-07-23T12:00:00Z",
"totalCount": 1438,
"onlineCount": 1383
},
{
"windowStart": "2026-07-24T00:00:00Z",
"totalCount": 1439,
"onlineCount": 1370
},
{
"windowStart": "2026-07-24T12:00:00Z",
"totalCount": 689,
"onlineCount": 663
},
{
"windowStart": "2026-07-25T00:00:00Z",
"totalCount": 1401,
"onlineCount": 1352
},
{
"windowStart": "2026-07-25T12:00:00Z",
"totalCount": 1438,
"onlineCount": 1366
},
{
"windowStart": "2026-07-26T00:00:00Z",
"totalCount": 1248,
"onlineCount": 1186
},
{
"windowStart": "2026-07-26T12:00:00Z",
"totalCount": 1538,
"onlineCount": 1459
},
{
"windowStart": "2026-07-27T00:00:00Z",
"totalCount": 1447,
"onlineCount": 1371
},
{
"windowStart": "2026-07-27T12:00:00Z",
"totalCount": 905,
"onlineCount": 862
},
{
"windowStart": "2026-07-28T00:00:00Z",
"totalCount": 1088,
"onlineCount": 1036
},
{
"windowStart": "2026-07-28T12:00:00Z",
"totalCount": 1241,
"onlineCount": 1188
},
{
"windowStart": "2026-07-29T00:00:00Z",
"totalCount": 522,
"onlineCount": 503
}
]
}
{
"id": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S",
"auditHistory": [
{
"windowStart": "2026-07-15T12:00:00Z",
"totalCount": 3567,
"onlineCount": 3542
},
{
"windowStart": "2026-07-16T00:00:00Z",
"totalCount": 1161,
"onlineCount": 1114
},
{
"windowStart": "2026-07-16T12:00:00Z",
"totalCount": 709,
"onlineCount": 672
},
{
"windowStart": "2026-07-17T00:00:00Z",
"totalCount": 393,
"onlineCount": 377
},
{
"windowStart": "2026-07-17T12:00:00Z",
"totalCount": 701,
"onlineCount": 666
},
{
"windowStart": "2026-07-18T00:00:00Z",
"totalCount": 1284,
"onlineCount": 1225
},
{
"windowStart": "2026-07-18T12:00:00Z",
"totalCount": 624,
"onlineCount": 596
},
{
"windowStart": "2026-07-19T00:00:00Z",
"totalCount": 721,
"onlineCount": 687
},
{
"windowStart": "2026-07-19T12:00:00Z",
"totalCount": 871,
"onlineCount": 830
},
{
"windowStart": "2026-07-20T00:00:00Z",
"totalCount": 908,
"onlineCount": 868
},
{
"windowStart": "2026-07-20T12:00:00Z",
"totalCount": 114,
"onlineCount": 111
},
{
"windowStart": "2026-07-25T12:00:00Z",
"totalCount": 289,
"onlineCount": 278
},
{
"windowStart": "2026-07-26T00:00:00Z",
"totalCount": 296,
"onlineCount": 280
},
{
"windowStart": "2026-07-26T12:00:00Z",
"totalCount": 657,
"onlineCount": 636
},
{
"windowStart": "2026-07-27T00:00:00Z",
"totalCount": 600,
"onlineCount": 575
},
{
"windowStart": "2026-07-27T12:00:00Z",
"totalCount": 636,
"onlineCount": 606
},
{
"windowStart": "2026-07-28T00:00:00Z",
"totalCount": 646,
"onlineCount": 615
},
{
"windowStart": "2026-07-28T12:00:00Z",
"totalCount": 635,
"onlineCount": 603
},
{
"windowStart": "2026-07-29T00:00:00Z",
"totalCount": 187,
"onlineCount": 179
}
]
}
{
"id": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs",
"auditHistory": [
{
"windowStart": "2026-07-15T12:00:00Z",
"totalCount": 1930,
"onlineCount": 1927
},
{
"windowStart": "2026-07-16T00:00:00Z",
"totalCount": 370,
"onlineCount": 353
},
{
"windowStart": "2026-07-16T12:00:00Z",
"totalCount": 971,
"onlineCount": 933
},
{
"windowStart": "2026-07-17T00:00:00Z",
"totalCount": 355,
"onlineCount": 344
},
{
"windowStart": "2026-07-17T12:00:00Z",
"totalCount": 692,
"onlineCount": 673
},
{
"windowStart": "2026-07-18T00:00:00Z",
"totalCount": 772,
"onlineCount": 742
},
{
"windowStart": "2026-07-18T12:00:00Z",
"totalCount": 752,
"onlineCount": 725
},
{
"windowStart": "2026-07-19T00:00:00Z",
"totalCount": 527,
"onlineCount": 490
},
{
"windowStart": "2026-07-19T12:00:00Z",
"totalCount": 346,
"onlineCount": 331
},
{
"windowStart": "2026-07-20T00:00:00Z",
"totalCount": 564,
"onlineCount": 541
},
{
"windowStart": "2026-07-20T12:00:00Z",
"totalCount": 1021,
"onlineCount": 975
},
{
"windowStart": "2026-07-21T00:00:00Z",
"totalCount": 720,
"onlineCount": 681
},
{
"windowStart": "2026-07-21T12:00:00Z",
"totalCount": 752,
"onlineCount": 720
},
{
"windowStart": "2026-07-22T00:00:00Z",
"totalCount": 733,
"onlineCount": 700
},
{
"windowStart": "2026-07-22T12:00:00Z",
"totalCount": 708,
"onlineCount": 686
},
{
"windowStart": "2026-07-23T00:00:00Z",
"totalCount": 62,
"onlineCount": 58
},
{
"windowStart": "2026-07-23T12:00:00Z",
"totalCount": 721,
"onlineCount": 682
},
{
"windowStart": "2026-07-24T00:00:00Z",
"totalCount": 735,
"onlineCount": 704
},
{
"windowStart": "2026-07-24T12:00:00Z",
"totalCount": 731,
"onlineCount": 704
},
{
"windowStart": "2026-07-25T00:00:00Z",
"totalCount": 772,
"onlineCount": 737
},
{
"windowStart": "2026-07-25T12:00:00Z",
"totalCount": 772,
"onlineCount": 746
},
{
"windowStart": "2026-07-26T00:00:00Z",
"totalCount": 745,
"onlineCount": 730
},
{
"windowStart": "2026-07-26T12:00:00Z",
"totalCount": 771,
"onlineCount": 737
},
{
"windowStart": "2026-07-27T00:00:00Z",
"totalCount": 239,
"onlineCount": 225
},
{
"windowStart": "2026-07-27T12:00:00Z",
"totalCount": 194,
"onlineCount": 187
},
{
"windowStart": "2026-07-28T00:00:00Z",
"totalCount": 663,
"onlineCount": 639
},
{
"windowStart": "2026-07-28T12:00:00Z",
"totalCount": 806,
"onlineCount": 777
},
{
"windowStart": "2026-07-29T00:00:00Z",
"totalCount": 255,
"onlineCount": 246
}
]
}
Unfortunately, I hadn’t adjusted the external monitoring for node1. However, I got the same results for Node 7.
Node7
cubefan@storj-chia01:~$ for item in `curl -sL http://192.168.188.92:14007/api/sno | jq '.satellites[].id' -r`; do curl -s http://192.168.188.92:14007/api/sno/satellite/$item | jq '{id: .id, auditHistory: [.auditHistory.windows[] | select(.totalCount != .onlineCount)]}'; done
{
"id": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE",
"auditHistory": [
{
"windowStart": "2026-07-15T12:00:00Z",
"totalCount": 148,
"onlineCount": 145
},
{
"windowStart": "2026-07-16T00:00:00Z",
"totalCount": 170,
"onlineCount": 161
},
{
"windowStart": "2026-07-16T12:00:00Z",
"totalCount": 105,
"onlineCount": 103
},
{
"windowStart": "2026-07-17T00:00:00Z",
"totalCount": 123,
"onlineCount": 119
},
{
"windowStart": "2026-07-17T12:00:00Z",
"totalCount": 239,
"onlineCount": 229
},
{
"windowStart": "2026-07-18T00:00:00Z",
"totalCount": 126,
"onlineCount": 122
},
{
"windowStart": "2026-07-18T12:00:00Z",
"totalCount": 176,
"onlineCount": 159
},
{
"windowStart": "2026-07-19T00:00:00Z",
"totalCount": 147,
"onlineCount": 135
},
{
"windowStart": "2026-07-19T12:00:00Z",
"totalCount": 113,
"onlineCount": 108
},
{
"windowStart": "2026-07-20T00:00:00Z",
"totalCount": 110,
"onlineCount": 105
},
{
"windowStart": "2026-07-20T12:00:00Z",
"totalCount": 135,
"onlineCount": 128
},
{
"windowStart": "2026-07-21T00:00:00Z",
"totalCount": 166,
"onlineCount": 161
},
{
"windowStart": "2026-07-21T12:00:00Z",
"totalCount": 146,
"onlineCount": 140
},
{
"windowStart": "2026-07-22T00:00:00Z",
"totalCount": 115,
"onlineCount": 106
},
{
"windowStart": "2026-07-22T12:00:00Z",
"totalCount": 106,
"onlineCount": 102
},
{
"windowStart": "2026-07-23T00:00:00Z",
"totalCount": 125,
"onlineCount": 117
},
{
"windowStart": "2026-07-23T12:00:00Z",
"totalCount": 161,
"onlineCount": 155
},
{
"windowStart": "2026-07-24T00:00:00Z",
"totalCount": 158,
"onlineCount": 147
},
{
"windowStart": "2026-07-24T12:00:00Z",
"totalCount": 100,
"onlineCount": 96
},
{
"windowStart": "2026-07-25T00:00:00Z",
"totalCount": 129,
"onlineCount": 126
},
{
"windowStart": "2026-07-25T12:00:00Z",
"totalCount": 145,
"onlineCount": 138
},
{
"windowStart": "2026-07-26T00:00:00Z",
"totalCount": 161,
"onlineCount": 150
},
{
"windowStart": "2026-07-26T12:00:00Z",
"totalCount": 148,
"onlineCount": 146
},
{
"windowStart": "2026-07-27T00:00:00Z",
"totalCount": 105,
"onlineCount": 101
},
{
"windowStart": "2026-07-27T12:00:00Z",
"totalCount": 83,
"onlineCount": 82
},
{
"windowStart": "2026-07-28T00:00:00Z",
"totalCount": 167,
"onlineCount": 162
},
{
"windowStart": "2026-07-28T12:00:00Z",
"totalCount": 172,
"onlineCount": 163
},
{
"windowStart": "2026-07-29T00:00:00Z",
"totalCount": 24,
"onlineCount": 23
}
]
}
{
"id": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6",
"auditHistory": [
{
"windowStart": "2026-07-15T12:00:00Z",
"totalCount": 1402,
"onlineCount": 1376
},
{
"windowStart": "2026-07-16T00:00:00Z",
"totalCount": 1512,
"onlineCount": 1446
},
{
"windowStart": "2026-07-16T12:00:00Z",
"totalCount": 655,
"onlineCount": 631
},
{
"windowStart": "2026-07-17T00:00:00Z",
"totalCount": 86,
"onlineCount": 82
},
{
"windowStart": "2026-07-17T12:00:00Z",
"totalCount": 1541,
"onlineCount": 1472
},
{
"windowStart": "2026-07-18T00:00:00Z",
"totalCount": 1423,
"onlineCount": 1370
},
{
"windowStart": "2026-07-18T12:00:00Z",
"totalCount": 869,
"onlineCount": 828
},
{
"windowStart": "2026-07-19T00:00:00Z",
"totalCount": 665,
"onlineCount": 639
},
{
"windowStart": "2026-07-19T12:00:00Z",
"totalCount": 965,
"onlineCount": 919
},
{
"windowStart": "2026-07-20T00:00:00Z",
"totalCount": 648,
"onlineCount": 617
},
{
"windowStart": "2026-07-20T12:00:00Z",
"totalCount": 1219,
"onlineCount": 1165
},
{
"windowStart": "2026-07-21T00:00:00Z",
"totalCount": 574,
"onlineCount": 549
},
{
"windowStart": "2026-07-21T12:00:00Z",
"totalCount": 373,
"onlineCount": 353
},
{
"windowStart": "2026-07-22T00:00:00Z",
"totalCount": 647,
"onlineCount": 624
},
{
"windowStart": "2026-07-22T12:00:00Z",
"totalCount": 1357,
"onlineCount": 1298
},
{
"windowStart": "2026-07-23T00:00:00Z",
"totalCount": 928,
"onlineCount": 897
},
{
"windowStart": "2026-07-23T12:00:00Z",
"totalCount": 770,
"onlineCount": 732
},
{
"windowStart": "2026-07-24T00:00:00Z",
"totalCount": 1113,
"onlineCount": 1070
},
{
"windowStart": "2026-07-24T12:00:00Z",
"totalCount": 1376,
"onlineCount": 1314
},
{
"windowStart": "2026-07-25T00:00:00Z",
"totalCount": 757,
"onlineCount": 733
},
{
"windowStart": "2026-07-25T12:00:00Z",
"totalCount": 1479,
"onlineCount": 1430
},
{
"windowStart": "2026-07-26T00:00:00Z",
"totalCount": 570,
"onlineCount": 549
},
{
"windowStart": "2026-07-26T12:00:00Z",
"totalCount": 1471,
"onlineCount": 1410
},
{
"windowStart": "2026-07-27T00:00:00Z",
"totalCount": 658,
"onlineCount": 627
},
{
"windowStart": "2026-07-27T12:00:00Z",
"totalCount": 611,
"onlineCount": 585
},
{
"windowStart": "2026-07-28T00:00:00Z",
"totalCount": 840,
"onlineCount": 798
},
{
"windowStart": "2026-07-28T12:00:00Z",
"totalCount": 1401,
"onlineCount": 1338
},
{
"windowStart": "2026-07-29T00:00:00Z",
"totalCount": 504,
"onlineCount": 482
}
]
}
{
"id": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S",
"auditHistory": [
{
"windowStart": "2026-07-15T12:00:00Z",
"totalCount": 2264,
"onlineCount": 2230
},
{
"windowStart": "2026-07-16T00:00:00Z",
"totalCount": 1128,
"onlineCount": 1076
},
{
"windowStart": "2026-07-16T12:00:00Z",
"totalCount": 1023,
"onlineCount": 973
},
{
"windowStart": "2026-07-17T00:00:00Z",
"totalCount": 609,
"onlineCount": 586
},
{
"windowStart": "2026-07-17T12:00:00Z",
"totalCount": 561,
"onlineCount": 536
},
{
"windowStart": "2026-07-18T00:00:00Z",
"totalCount": 750,
"onlineCount": 725
},
{
"windowStart": "2026-07-18T12:00:00Z",
"totalCount": 791,
"onlineCount": 751
},
{
"windowStart": "2026-07-19T00:00:00Z",
"totalCount": 364,
"onlineCount": 335
},
{
"windowStart": "2026-07-19T12:00:00Z",
"totalCount": 612,
"onlineCount": 585
},
{
"windowStart": "2026-07-20T00:00:00Z",
"totalCount": 597,
"onlineCount": 574
},
{
"windowStart": "2026-07-20T12:00:00Z",
"totalCount": 47,
"onlineCount": 46
},
{
"windowStart": "2026-07-25T00:00:00Z",
"totalCount": 14,
"onlineCount": 13
},
{
"windowStart": "2026-07-25T12:00:00Z",
"totalCount": 381,
"onlineCount": 359
},
{
"windowStart": "2026-07-26T00:00:00Z",
"totalCount": 405,
"onlineCount": 393
},
{
"windowStart": "2026-07-26T12:00:00Z",
"totalCount": 404,
"onlineCount": 384
},
{
"windowStart": "2026-07-27T00:00:00Z",
"totalCount": 373,
"onlineCount": 361
},
{
"windowStart": "2026-07-27T12:00:00Z",
"totalCount": 402,
"onlineCount": 384
},
{
"windowStart": "2026-07-28T00:00:00Z",
"totalCount": 387,
"onlineCount": 368
},
{
"windowStart": "2026-07-28T12:00:00Z",
"totalCount": 363,
"onlineCount": 348
},
{
"windowStart": "2026-07-29T00:00:00Z",
"totalCount": 129,
"onlineCount": 126
}
]
}
{
"id": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs",
"auditHistory": [
{
"windowStart": "2026-07-15T12:00:00Z",
"totalCount": 1643,
"onlineCount": 1642
},
{
"windowStart": "2026-07-16T00:00:00Z",
"totalCount": 311,
"onlineCount": 292
},
{
"windowStart": "2026-07-16T12:00:00Z",
"totalCount": 787,
"onlineCount": 763
},
{
"windowStart": "2026-07-17T00:00:00Z",
"totalCount": 685,
"onlineCount": 652
},
{
"windowStart": "2026-07-17T12:00:00Z",
"totalCount": 682,
"onlineCount": 660
},
{
"windowStart": "2026-07-18T00:00:00Z",
"totalCount": 653,
"onlineCount": 633
},
{
"windowStart": "2026-07-18T12:00:00Z",
"totalCount": 626,
"onlineCount": 605
},
{
"windowStart": "2026-07-19T00:00:00Z",
"totalCount": 666,
"onlineCount": 635
},
{
"windowStart": "2026-07-19T12:00:00Z",
"totalCount": 603,
"onlineCount": 584
},
{
"windowStart": "2026-07-20T00:00:00Z",
"totalCount": 373,
"onlineCount": 363
},
{
"windowStart": "2026-07-20T12:00:00Z",
"totalCount": 793,
"onlineCount": 765
},
{
"windowStart": "2026-07-21T00:00:00Z",
"totalCount": 663,
"onlineCount": 633
},
{
"windowStart": "2026-07-21T12:00:00Z",
"totalCount": 687,
"onlineCount": 669
},
{
"windowStart": "2026-07-22T00:00:00Z",
"totalCount": 661,
"onlineCount": 641
},
{
"windowStart": "2026-07-22T12:00:00Z",
"totalCount": 654,
"onlineCount": 614
},
{
"windowStart": "2026-07-23T00:00:00Z",
"totalCount": 657,
"onlineCount": 626
},
{
"windowStart": "2026-07-23T12:00:00Z",
"totalCount": 736,
"onlineCount": 699
},
{
"windowStart": "2026-07-24T00:00:00Z",
"totalCount": 631,
"onlineCount": 610
},
{
"windowStart": "2026-07-24T12:00:00Z",
"totalCount": 666,
"onlineCount": 631
},
{
"windowStart": "2026-07-25T00:00:00Z",
"totalCount": 601,
"onlineCount": 571
},
{
"windowStart": "2026-07-25T12:00:00Z",
"totalCount": 323,
"onlineCount": 315
},
{
"windowStart": "2026-07-26T00:00:00Z",
"totalCount": 656,
"onlineCount": 630
},
{
"windowStart": "2026-07-26T12:00:00Z",
"totalCount": 620,
"onlineCount": 597
},
{
"windowStart": "2026-07-27T00:00:00Z",
"totalCount": 659,
"onlineCount": 629
},
{
"windowStart": "2026-07-27T12:00:00Z",
"totalCount": 646,
"onlineCount": 624
},
{
"windowStart": "2026-07-28T00:00:00Z",
"totalCount": 660,
"onlineCount": 634
},
{
"windowStart": "2026-07-28T12:00:00Z",
"totalCount": 663,
"onlineCount": 638
},
{
"windowStart": "2026-07-29T00:00:00Z",
"totalCount": 238,
"onlineCount": 234
}
]
}
Ping error Node7
cubefan@storj-chia01:~$ cat /mnt/storj/ssd/node007_2024.07/logs/node.log.25 | grep ping
cubefan@storj-chia01:~$ cat /mnt/storj/ssd/node007_2024.07/logs/node.log.24 | grep ping
cubefan@storj-chia01:~$ cat /mnt/storj/ssd/node007_2024.07/logs/node.log.23 | grep ping
cubefan@storj-chia01:~$ cat /mnt/storj/ssd/node007_2024.07/logs/node.log.22 | grep ping
cubefan@storj-chia01:~$ cat /mnt/storj/ssd/node007_2024.07/logs/node.log.21 | grep ping
cubefan@storj-chia01:~$ cat /mnt/storj/ssd/node007_2024.07/logs/node.log.20 | grep ping
cubefan@storj-chia01:~$ cat /mnt/storj/ssd/node007_2024.07/logs/node.log.19 | grep ping
cubefan@storj-chia01:~$ cat /mnt/storj/ssd/node007_2024.07/logs/node.log.18 | grep ping
cubefan@storj-chia01:~$ cat /mnt/storj/ssd/node007_2024.07/logs/node.log.17 | grep ping
cubefan@storj-chia01:~$ cat /mnt/storj/ssd/node007_2024.07/logs/node.log.16 | grep ping
cubefan@storj-chia01:~$ cat /mnt/storj/ssd/node007_2024.07/logs/node.log.15 | grep ping
cubefan@storj-chia01:~$ cat /mnt/storj/ssd/node007_2024.07/logs/node.log.14 | grep ping
cubefan@storj-chia01:~$ cat /mnt/storj/ssd/node007_2024.07/logs/node.log.13 | grep ping
2026-07-16T06:10:24+02:00 ERROR contact:service ping satellite failed {"process": "storagenode", "satellite_id": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "attempts": 1, "error": "ping satellite: failed to ping storage node, your node indicated error code: 0, rpc: tcp connector failed: rpc: dial tcp 87.106.118.84:28967: connect: connection timed out", "errorVerbose": "ping satellite: failed to ping storage node, your node indicated error code: 0, rpc: tcp connector failed: rpc: dial tcp 87.106.118.84:28967: connect: connection timed out\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:232\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:169\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:89\n\tstorj.io/common/sync2.(*Cycle).Run:163\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:93"}
2026-07-16T08:10:25+02:00 ERROR contact:service ping satellite failed {"process": "storagenode", "satellite_id": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "attempts": 1, "error": "ping satellite: failed to ping storage node, your node indicated error code: 0, rpc: tcp connector failed: rpc: dial tcp 87.106.118.84:28967: connect: connection timed out", "errorVerbose": "ping satellite: failed to ping storage node, your node indicated error code: 0, rpc: tcp connector failed: rpc: dial tcp 87.106.118.84:28967: connect: connection timed out\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:232\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:169\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:89\n\tstorj.io/common/sync2.(*Cycle).Run:163\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:93"}
2026-07-16T09:10:25+02:00 ERROR contact:service ping satellite failed {"process": "storagenode", "satellite_id": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "attempts": 1, "error": "ping satellite: failed to ping storage node, your node indicated error code: 0, rpc: tcp connector failed: rpc: dial tcp 87.106.118.84:28967: connect: connection timed out", "errorVerbose": "ping satellite: failed to ping storage node, your node indicated error code: 0, rpc: tcp connector failed: rpc: dial tcp 87.106.118.84:28967: connect: connection timed out\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:232\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:169\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:89\n\tstorj.io/common/sync2.(*Cycle).Run:163\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:93"}
2026-07-16T16:10:24+02:00 ERROR contact:service ping satellite failed {"process": "storagenode", "satellite_id": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "attempts": 1, "error": "ping satellite: failed to ping storage node, your node indicated error code: 0, rpc: tcp connector failed: rpc: dial tcp 87.106.118.84:28967: connect: connection timed out", "errorVerbose": "ping satellite: failed to ping storage node, your node indicated error code: 0, rpc: tcp connector failed: rpc: dial tcp 87.106.118.84:28967: connect: connection timed out\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:232\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:169\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:89\n\tstorj.io/common/sync2.(*Cycle).Run:163\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:93"}
2026-07-16T17:10:24+02:00 ERROR contact:service ping satellite failed {"process": "storagenode", "satellite_id": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "attempts": 1, "error": "ping satellite: failed to ping storage node, your node indicated error code: 0, rpc: tcp connector failed: rpc: dial tcp 87.106.118.84:28967: connect: connection timed out", "errorVerbose": "ping satellite: failed to ping storage node, your node indicated error code: 0, rpc: tcp connector failed: rpc: dial tcp 87.106.118.84:28967: connect: connection timed out\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:232\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:169\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:89\n\tstorj.io/common/sync2.(*Cycle).Run:163\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:93"}
cubefan@storj-chia01:~$ cat /mnt/storj/ssd/node007_2024.07/logs/node.log.12 | grep ping
2026-07-17T00:10:25+02:00 ERROR contact:service ping satellite failed {"process": "storagenode", "satellite_id": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "attempts": 1, "error": "ping satellite: failed to ping storage node, your node indicated error code: 0, rpc: tcp connector failed: rpc: dial tcp 87.106.118.84:28967: connect: connection timed out", "errorVerbose": "ping satellite: failed to ping storage node, your node indicated error code: 0, rpc: tcp connector failed: rpc: dial tcp 87.106.118.84:28967: connect: connection timed out\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:232\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:169\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:89\n\tstorj.io/common/sync2.(*Cycle).Run:163\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:93"}
2026-07-17T06:10:25+02:00 ERROR contact:service ping satellite failed {"process": "storagenode", "satellite_id": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "attempts": 1, "error": "ping satellite: failed to ping storage node, your node indicated error code: 0, rpc: tcp connector failed: rpc: dial tcp 87.106.118.84:28967: connect: connection timed out", "errorVerbose": "ping satellite: failed to ping storage node, your node indicated error code: 0, rpc: tcp connector failed: rpc: dial tcp 87.106.118.84:28967: connect: connection timed out\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:232\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:169\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:89\n\tstorj.io/common/sync2.(*Cycle).Run:163\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:93"}
2026-07-17T20:10:24+02:00 ERROR contact:service ping satellite failed {"process": "storagenode", "satellite_id": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "attempts": 1, "error": "ping satellite: failed to ping storage node, your node indicated error code: 0, rpc: tcp connector failed: rpc: dial tcp 87.106.118.84:28967: connect: connection timed out", "errorVerbose": "ping satellite: failed to ping storage node, your node indicated error code: 0, rpc: tcp connector failed: rpc: dial tcp 87.106.118.84:28967: connect: connection timed out\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:232\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:169\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:89\n\tstorj.io/common/sync2.(*Cycle).Run:163\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:93"}
2026-07-17T20:12:40+02:00 ERROR contact:service ping satellite failed {"process": "storagenode", "satellite_id": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "attempts": 2, "error": "ping satellite: failed to ping storage node, your node indicated error code: 0, rpc: tcp connector failed: rpc: dial tcp 87.106.118.84:28967: connect: connection timed out", "errorVerbose": "ping satellite: failed to ping storage node, your node indicated error code: 0, rpc: tcp connector failed: rpc: dial tcp 87.106.118.84:28967: connect: connection timed out\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:232\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:169\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:89\n\tstorj.io/common/sync2.(*Cycle).Run:163\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:93"}
cubefan@storj-chia01:~$ cat /mnt/storj/ssd/node007_2024.07/logs/node.log.11 | grep ping
2026-07-18T00:10:24+02:00 ERROR contact:service ping satellite failed {"process": "storagenode", "satellite_id": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "attempts": 1, "error": "ping satellite: failed to ping storage node, your node indicated error code: 0, rpc: tcp connector failed: rpc: dial tcp 87.106.118.84:28967: connect: connection timed out", "errorVerbose": "ping satellite: failed to ping storage node, your node indicated error code: 0, rpc: tcp connector failed: rpc: dial tcp 87.106.118.84:28967: connect: connection timed out\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:232\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:169\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:89\n\tstorj.io/common/sync2.(*Cycle).Run:163\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:93"}
2026-07-18T08:10:25+02:00 ERROR contact:service ping satellite failed {"process": "storagenode", "satellite_id": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "attempts": 1, "error": "ping satellite: failed to ping storage node, your node indicated error code: 0, rpc: tcp connector failed: rpc: dial tcp 87.106.118.84:28967: connect: connection timed out", "errorVerbose": "ping satellite: failed to ping storage node, your node indicated error code: 0, rpc: tcp connector failed: rpc: dial tcp 87.106.118.84:28967: connect: connection timed out\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:232\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:169\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:89\n\tstorj.io/common/sync2.(*Cycle).Run:163\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:93"}
2026-07-18T13:10:25+02:00 ERROR contact:service ping satellite failed {"process": "storagenode", "satellite_id": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "attempts": 1, "error": "ping satellite: failed to ping storage node, your node indicated error code: 0, rpc: tcp connector failed: rpc: dial tcp 87.106.118.84:28967: connect: connection timed out", "errorVerbose": "ping satellite: failed to ping storage node, your node indicated error code: 0, rpc: tcp connector failed: rpc: dial tcp 87.106.118.84:28967: connect: connection timed out\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:232\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:169\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:89\n\tstorj.io/common/sync2.(*Cycle).Run:163\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:93"}
2026-07-18T15:36:37+02:00 ERROR contact:service ping satellite failed {"process": "storagenode", "satellite_id": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "attempts": 1, "error": "ping satellite: failed to ping storage node, your node indicated error code: 0, rpc: tcp connector failed: rpc: dial tcp 87.106.118.84:28967: connect: connection timed out", "errorVerbose": "ping satellite: failed to ping storage node, your node indicated error code: 0, rpc: tcp connector failed: rpc: dial tcp 87.106.118.84:28967: connect: connection timed out\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:232\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:169\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:89\n\tstorj.io/common/sync2.(*Cycle).Run:163\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:93"}
2026-07-18T17:36:37+02:00 ERROR contact:service ping satellite failed {"process": "storagenode", "satellite_id": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "attempts": 1, "error": "ping satellite: failed to ping storage node, your node indicated error code: 0, rpc: tcp connector failed: rpc: dial tcp 87.106.118.84:28967: connect: connection timed out", "errorVerbose": "ping satellite: failed to ping storage node, your node indicated error code: 0, rpc: tcp connector failed: rpc: dial tcp 87.106.118.84:28967: connect: connection timed out\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:232\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:169\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:89\n\tstorj.io/common/sync2.(*Cycle).Run:163\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:93"}
Thanks all to help collect enough statistic. All shared with the team.
It supposed to use not a cached state, but cached IPs, like we use it for the node selection. This cache is updated on every check-in. It might be a problem for nodes which changes IP more often than once a hour, but it’s a self corrected state, even if the node is lucky enough to update its IP right after the check-in. Earlier that cache I believe also updated on audits, which might happen much more often than once a hour. But now, if these audits now doesn’t update the cache, it may try connecting the previous IP and rendered as offline until the next check-in.
However, here you showed, that the node might have a static IP, but still rendered as offline, even if it’s actually online. This makes me think, that the analyze above is correct, and the code even didn’t try to actually connect the node due to an internal timeout.
For me it sounds like a bug. Will wait for the team’s response.
We all expierience this since a week or 2 ago. There is already a thread regarding the situation. It seems there is a bug on the satellite side.
No problem, thank you for taking the time to reply.
My bet is a network issue, not a bug on satellite side. As satellites are running on different providers, the routes are different. There is a very active investigation with the hosting provider about routes to specific ASNs, but it also depends on upstream support.
We will reset all online scores which were declined due to false positive checks.
But do have a solution soon? Is it safe now to do some maintenance and short downtimes? I’m scared of downtimes because when they rerun the online-score won’t increase.