Is there a way to query online status from command line

Node Operators FAQ
1

Is there a way to query online status from command line, like executing a command that returns status in shell?
(Dashboard keeps running and not returning to shell.)
Thank you in advance

2

Yes, it’s. See https://documentation.storj.io/setup/cli/dashboard#cli-storage-node-dashboard

3

The dashboard keeps running and not returning to shell.
Is there a way to return the status in the shell without human intervention?

4

Why do you want to return to shell without human intervention ?

5

To automate things. Why do you ask?

6

I have never come across such a request. Most requests are how they can access the dashboard. You can exit by Ctrl+C but you want it automated which got me curious :slight_smile:

7

Seems you want to call an API instead
See

8

So this is the real solution:

image
image1121Ă—84 4.2 KB 

(echo "OFFLINE";bash -c 'docker exec -it storagenode /app/dashboard.sh | head -9 | grep "Status" | cut -c8-') | tail -1
2 Likes
10

I think there are better approaches: Search results for 'tags:monitoring+offline order:latest' - Storj Community Forum (official)

You may also request your external address with port of your node with curl (but not from the LAN), it would return the status in json.

See also

12

Yeap,

curl http://82.192.164.154:28967 | jq -r '.AllHealthy'

Or just check the HTTP status code. If it’s 200, then everything should be ok.

1 Like
14

But you get the response, so the node is online. It would become “AllHealthy”, when your online score would be 100%.

This is because it uses the time window of the last 30 days, see

17

Your node is somewhere in the middle of the month, where the previous days only have an offline events. You need to keep it online for the next 30 days to fully recover. Each downtime requires another 30 days to recover.

However, I would also recommend to check your logs for errors like ping satellite failed and exclude all rate errors.

20

It should be

sudo docker logs storagenode 2>&1 | grep "ping satellite failed" | grep -v "rate"

You may grab the output from the CLI dashboard too, as in the solution in this thread,

however, since you use sudo for docker commands, then you need to add sudo before the docker command inside this line:

(echo "OFFLINE";bash -c 'sudo docker exec -it storagenode /app/dashboard.sh | head -9 | grep "Status" | cut -c8-') | tail -1

But if you want to use it in some script, then you shouldn’t add sudo there, the wrapper script should be run with sudo instead.

22

The gui dashboard is comparing last satellite ping with current time to check online state. In bash I am doing the same like so:

      LASTPINGED=$(curl -s http://127.0.0.1:14002/api/sno/ | jq -r '.lastPinged')
      TIMEDIFF=$(($(date -d "$(date)" +%s)-$(date -d "$LASTPINGED" +%s)))
      if [ $TIMEDIFF -lt 120 ]
      then
         echo "node online"
      else
         echo "node offline"
      fi
1 Like
24

I am using ubuntu amd64, other systems might need a different syntax. However the way to do it is to ask the api endpoint for lastPinged timestamp and compare with current time.

26

There are other options to parse a json string for sure, jq was just the first I found… :wink:

28 
curl -s http://127.0.0.1:14002/api/sno/ | grep -Po '"lastPinged": *\K"[^"]*"'
1 Like
31

I decided to use 120 seconds, the dashboard uses a higher number. I guess your result is in seconds but +3600 because timestamps have 1 hour diff (time zone).

edit:
It seems to be 120 minutes in the original code: storj/web/storagenode/src/app/store/modules/node.ts at 9a3c5d2f068dbefc526b9c16b2ffe8df5a6d0dc4 · storj/storj · GitHub

33

This needs to be addressed first. There no point in reimplementing jq with grep, while your system drive is literally choking to death.

I’m however strongly against checking the node status from within the network. There is also no need to reimplement existing monitoring solutions, that are already capable of parsing json and notifying you when something is not right.

example

image
image1290Ă—1564 165 KB

That’s another thing. Reviewing status every morning? Why give yourself another chore just to miss node getting offline by 24 hours? Get notified when problem occurs immediately.

1 Like
35

It could be simple outdated kernel images or temporary APT packages.
You may try simple things like

sudo apt autoremove

Then search for a previously installed images which are not needed anymore and remove them too.

You could also just forget to extend the default image to the full size of SD, see Prepare.