One satelite shows 90% online, ther others 100%

[boelle]

is this normal? node id 1gmDCyzjP6sCz6X6fZwVXrHndFztkS1GWFqE1jiVZTVmp4QRSf

[Alexey]

No, it’s not normal. This is mean that this satellite was unable to contact your node.
You may check when it was happened:

My uptime should be 100% on all satellites i have not gotten any uptime robot notifications of downtime in months

Would you mind to provide dashboard from the uptimerobot? And also, please, provide result of the command: PowerShell foreach ($item in ((Invoke-WebRequest http://localhost:14002/api/sno).Content | ConvertFrom-Json).satellites.id) { ((Invoke-WebRequest http://localhost:14002/api/sno/satellite/$item).Content | ConvertFrom-Json).auditHistory.windows | Where-Object{$_.totalCount -ne $_.onlineCount} | ForEach-Object{Write-Host $item; $_ | Format-Table -AutoSize} } bash for item in `…

Then check logs of your firewall/router on those dates to figure out why is it happened.

You may also find some explanations in your node’s logs as well, search for ping satellite failed except rate.

[boelle]

hmm… no output

root@bo-omv:~# ./check-online-score.sh
root@bo-omv:~#

script:

for item in `curl -sL http://localhost:14002/api/sno | jq '.satellites[].id' -r`; do
    curl -s http://192.168.0.163:14002/api/sno/satellite/$item | \
    jq '{id: .id, auditHistory: [.auditHistory.windows[] | select(.totalCount != .onlineCount)]}'
done

@Alexey

You may also find some explanations in your node’s logs as well, search for ping satellite failed except rate.

are there a oneline command that can do that? will try and locate the log

[boelle]

i think

docker logs storagenode | grep "ping satellite failed except rate"

EDIT… no that did not work

root@bo-omv:~# docker logs storagenode | grep 'ping satellite failed except rate'
-bash:  grep: kommando ikke fundet
root@bo-omv:~#

but grep is installed…

root@bo-omv:~# grep
Brug: grep [FLAG]... MØNSTRE [FIL]...
Prøv "grep --help" for mere information.
root@bo-omv:~#

[Alexey]

@boelle

hmm… no output

@boelle

`curl -sL http://localhost:14002/api/sno

should be

`curl -sL http://192.168.0.163:14002/api/sno

@boelle

docker logs storagenode | grep "ping satellite failed except rate"

it should be

docker logs storagenode 2>&1 | grep "ping satellite failed" | grep -v "rate"

[boelle]

@Alexey

should be

`curl -sL http://192.168.0.163:14002/api/sno

upps…

root@bo-omv:~# ./check-online-score.sh
jq: error (at <stdin>:1): Cannot iterate over null (null)
{
  "id": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6",
  "auditHistory": []
}
{
  "id": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S",
  "auditHistory": [
    {
      "windowStart": "2024-09-10T00:00:00Z",
      "totalCount": 5,
      "onlineCount": 4
    }
  ]
}
{
  "id": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs",
  "auditHistory": []
}
root@bo-omv:~#

@Alexey

docker logs storagenode 2>&1 | grep “ping satellite failed” | grep -v “rate”

root@bo-omv:~# docker logs storagenode 2>&1 | grep "ping satellite failed" | grep -v "rate"
root@bo-omv:~#

looks very strange

[Alexey]

Why is so? The docker logs keeps only 5 logs with 20MiB each, also they got deleted with the container recreating.
You can see the date when the node missed the audit check. Since it is fairly new, it is expected to be sensitive to downtime, see

How is the online score calculated? - Storj Docs

[boelle]

@Alexey

You can see the date when the node missed the audit check

i assume that is if the logs had not been deleted… would be nice if the logs was not deleted at restart but only after 30 days…

but oh well i will just ignore that only one satelite says 90% and then hope it goes up again on its own

[Alexey]

@boelle

i assume that is if the logs had not been deleted… would be nice if the logs was not deleted at restart but only after 30 days…

Easy

How do I redirect my logs to a file? - Storj Docs

But likely you would need a logrotate configured too.

@boelle

but oh well i will just ignore that only one satelite says 90% and then hope it goes up again on its own

yes. It needs 30 days online to fully recover. Each downtime requires another 30 days online to recover.

[boelle]

question came up in my head… are there not a way so “ask” us1.storj.io:7777 about this too ?

EDIT just when it failed so i could check against my router logs

[Alexey]

Yes, restart the node to force check-in. However, the restart itself would produce a downtime, so, well

Just keep it online, that’s all what’s needed. The node performs an automatically check-in every hour by default.

[boelle]

so there are no way to directly ask the satelite when a given node missed the check?

could be nice as i could then look in my router first… or maybe it was when i restarted the node myself

Q to the above… can a node restart itself?

[Alexey]

@boelle

Q to the above… can a node restart itself?

yes. but only for FATAL or/and Unrecoverable errors, and if the restart policy for the container is set to always or unless-stopped.

See also

How do I check my node when I'm away from my machine? - Storj Docs

[boelle]

@boelle

so there are no way to directly ask the satelite when a given node missed the check?

you missed that part…

[Alexey]

nope. I usually answer only on questions which can be answered and trying to do not answer on duplicate questions from the same requester.
I would try to expand my previous response.
There is no way to do so except restart the node to force check-in, however, this will only updates reputation on your node and would try to contact it and report the result of the attempt (here you would see a ping satellite failed response, if the satellite was unable to contact your node and the reason), and maybe some other useful stat, like the disk usage confirmed on the satellites, and so on.

There is no way to force the satellite to audit your node again. Just keep it online.

[boelle]

no problemo

but it could be a nice to have feature… ie to ask when a node failed the check so diagnosis of network problems was possible

but yes, i dont want to force a check, that could lead to cheating to get better statistics

[Alexey]

@boelle

but it could be a nice to have feature… ie to ask when a node failed the check so diagnosis of network problems was possible

I do not think so. It’s a spending developers time for nothing. You may just bring it online and check, that the online score would grow.
I see no point to implement the force check, which will increase load on the satellites without any benefit for the customers.

You always may restart the node and make sure that’s not throwing any errors like ping satellite failed.
Or use your mobile phone and the mobile internet and try to open your public IP with port, you will get a response, if it’s externally available. Also UptimeRobot or similar would do the same for you over the time with a nice graphs.

[boelle]

@Alexey

implement the force check,

i did not say force check

[Julio]

Can you not just check your e-mail offline notification date/time stamps to corelate to your router logs?

Cheap DDNS service - like duckdns or something?

[boelle]

@Julio

Can you not just check your e-mail offline notification date/time stamps to corelate to your router logs?

if i got them yes

@Julio

Cheap DDNS service - like duckdns or something?

no i use static up so no need for DDNS