Easy monitoring for Storagenodes

[Alexey]

@aurelb87

Can I consider that if I have “AllHealthy”: true, it means that my node is OK?

Yes! And even if not all healthy - too, because it’s online. “AllHealthy” means that your node’s reputation is in a good state too.

[aurelb87]

OK that’s clear for me. Why the value of “statuses” is “null” ?

Also, may I use the environment variable on my docker container (STORJ_HEALTHCHECK_DETAILS) to have more details ?

[elek]

@aurelb87

Also, may I use the environment variable on my docker container (STORJ_HEALTHCHECK_DETAILS) to have more details ?

I am using this flag.

It is disabled by default only because it exposes the satellites IDS what you use to the public, but because everybody uses the same satellites, I don’t see any risk… Just somebody complained…

It will show per-satellite stat. Which can be useful to see if the problem is real.

Example:

"Statuses": [
    {
      "OnlineScore": 1,
      "SatelliteID": "1nVXgrduQWu1d5HGXgaTTMbKmMXy5U3Fr7drMXDbrcf8w1jQ5y",
      "DisqualifiedAt": null,
      "SuspendedAt": null
    },
    {
      "OnlineScore": 1,
      "SatelliteID": "12ZQbQ8WWFEfKNE9dP78B1frhJ8PmyYmr8occLEf1mQ1ovgVWy",
      "DisqualifiedAt": null,
      "SuspendedAt": null
    },

[aurelb87]

Hello @elek,

Thanks for your explanation.
I know understand.

[pangolin]

@aurelb87

Could you please tell me more about HTTP calls or other solution ?

If you want to go deeper into monitoring nodes, you could also use the debug API. Enable this in config file and enjoy a bunch of information.

# address to listen on for debug endpoints
# debug.addr: :6000

[Alexey]

I would add a link to the post with some examples for usage of the debug port:

Guide to debug my storage node, uplink, s3 gateway, satellite

We have integrated a package called monkit in all storage nodes. It is very usefull for debuging. By default it will open a random port. In order to connect to the debug port you will need to: Add debug.addr: ":5999" to the config file. Now the port is fixed and will not change on every restart. Add -p 127.0.0.1:5999:5999 to the docker run command. Ready for action! You can request the following informations: curl localhost:5999/mon/funcs Find out how long a function like GetExpired need…

See also

Setting Up a Dashboard for Commercial Storage Nodes - Storj Docs

[CubeFan]

I’ve set this up for myself and it’s really great to get even more data. What I would like to know is whether it is possible to display the statistics for all nodes, as is the case now, but also for each individual node if you have several? That would be a great extension

[Alexey]

Do you mean something like that?

Prometheus Storj-Exporter

I’m mostly using Prometheus to monitor stuff and was in need for storj exporter to monitor things like audit counts and uptimes per satellite, data as reported by storagenode etc… So I made one: [image] Checkout the README to setup Storj-Exporter Update exporter to latest version: Use watchtower or update manually using the following commands: docker pull anclrii/storj-exporter:latest ; \ docker stop storj-exporter ; \ docker rm storj-export…

You may also check other solutions for monitoring: Search results for 'tags:monitoring #sno-category:sno-start' - Storj Community Forum (official)

[CubeFan]

No, that’s not what I mean, I already use it.

I mean that you could see the upper part for each node individually or at least label the values in the lower part so that you know which node it comes from. Something like GET downloads node 01, … or uploads node 01, …

Can’t this be done somehow with individual jobs or labels in prometheus?

[Alexey]

I guess yes. At least I saw many dashboards here, on the forum, for multiple nodes.
I think you can search them and ask authors to share.
One of the examples:

How to monitor all nodes in your LAN using prometheus + grafana [linux using docker]

1. Setting up prometheus For storing all statistics of the nodes, we use prometheus. So choose one PC/Server where you want to run prometheus. It doesn’t need to be a powerful PC and it doesn’t need much space, you can decide how much space prometheus is allowed to use or how long you want to store your statistics. However, make sure you choose a PC (or a directory) that doesn’t write to an SD card as that would shorten its lifespan significantly. On that chosen PC we first create a prometheus.…

[Tech Preview] Email alerts with Grafana and Prometheus

I would like to setup email alerts for my storage nodes. It turns out the storage node already has a Prometheus endpoint. It is running on the debug endpoint /metrics. If you haven’t set the debug endpoint please visit Guide to debug my storage node, uplink, s3 gateway, satellite I have already a Prometheus instance running and Grafana inclusive email alerts. Here are my configs: docker-compose.yaml version: "3.3" services: prometheus: restart: unless-stopped user: 993:991 ports…

See other prometheus posts or limit a search to: Search results for 'tags:multiple-nodes+prometheus' - Storj Community Forum (official)