How to monitor all nodes in your LAN using prometheus + grafana [linux using docker]

TheMightyGreek · January 3, 2021, 3:48pm

damn that’s a stupid mistake thanks !

EDIT: sorry for spamming the forum this much for such a stupid mistake and losing your time @kevink …

kevink · January 3, 2021, 4:05pm

no problem. happy if you get it working.

dragonhogan · January 3, 2021, 4:33pm

@kevink really appreciate all of your help on this. So here’s where I’ve gotten to thus far:

got prometheus installed/configured/running on a standalone machine that isn’t currently running a Storj node. Here’s the “targets” page:

image816×513 69.4 KB
got storj-exporter installed on all three machines that are running nodes
got grafana installed on standalone machine

image901×307 90.3 KB

Now when I go to the grafana page (ip:3000) I get the grafana login page.

I couldn’t find a “create” account link on it, so I went over to grafana.com and created an account, and verified my e-mail address…although when I go back to ip:3000, I keep getting the “invalid username or password” error…but if I go back to grafana.com, I can log in without a problem…what am I missing here?

maxsch · January 3, 2021, 4:44pm

Try enter admin for username and password.
There is default login: https://grafana.com/docs/grafana/latest/getting-started/getting-started/

dragonhogan · January 3, 2021, 4:44pm

ah…just did another google search and figured out that after installing grafana…the default username/password is admin/admin…THEN it asks you to change the password…so I’ll keep plugging along…

kevink · January 3, 2021, 4:45pm

yes sorry should have thought about that. I’ll add it to the how-to. glad you got this far! Now you’re almost finished

dragonhogan · January 3, 2021, 4:56pm

I just went through the import dashboard steps, and I’m getting an error about “datasource name” not found. Is this normal when first running and I just need to wait? or did I do something wrong?

kevink · January 3, 2021, 5:05pm

ah… added the missing steps of adding the datasource.

dragonhogan · January 3, 2021, 5:12pm

and now I’m getting “http error bad gateway” message when adding that configuration to grafana:

maxsch · January 3, 2021, 5:13pm

Uh yeah, I am on Synology with multiple nodes and exporters, without STORJ_API_PORT of each node it did not work at all…

kevink · January 3, 2021, 5:14pm

oh yeah I forgot… localhost won’t work with docker… you have to use your ip instead of localhost

kevink · January 3, 2021, 5:15pm

What do you mean? STORJ_API_PORT does never have to be used with docker on the same host.

dragonhogan · January 3, 2021, 5:18pm

NICE!!!

I have data

Thank you sooooo much for your help on this. After going through all of that, I would have never been able to figure all of that out on my own with what I was able to find.

kevink · January 3, 2021, 5:35pm

Glad we got it finally working! Thanks for helping me find what was missing in the how-to

dragonhogan · January 3, 2021, 8:20pm

okay, new question, in my Node Summary Table, as you can see in the previous post, the DQ and SU for all 3 nodes is “No Data” and then Held values for Node 1 and 2 are both “no Data”. any idea what would cause those fields to not be populated? Those fields have not populated with any data still…

Skyblockpro1 · January 3, 2021, 8:24pm

Hey guys I know this might sound weird but I run the node on a windows system and cannot seam to understand how to get it working. Any suggestions. Because it seams all of this is just for docker.

Thanks any help much appreciated

kevink · January 3, 2021, 8:40pm

Honestly I have no idea. I’ve never seen it unpopulated. Can you check if your databases are ok? Maybe we can continue this in the exporter thread (just to keep things better organized):

kevink · January 3, 2021, 8:40pm

Sorry this is indeed all for docker and linux systems. I have no idea how to get it working on windows. You could however run the exporter on your linux machine and have it pointed to the dashboard port on the windows machines.
Although… the exporter is just a python script, so you could just run it directly in windows. You might need to set up some autostart for it though.
No idea about prometheus and grafana on windows though.

dragonhogan · January 3, 2021, 8:45pm

so I just found that you had posted a while back, a link to your grafana dashboard

gist.github.com

https://gist.github.com/kevinkk525/7dea9f744ad6fff009b0195bae9a3125

storj_exporter_dashboard.json

{
  "annotations": {
    "list": [
      {
        "builtIn": 1,
        "datasource": "-- Grafana --",
        "enable": true,
        "hide": true,
        "iconColor": "rgba(0, 211, 255, 1)",
        "name": "Annotations & Alerts",

This file has been truncated. show original

and I just added that one, and it’s showing suspension values…

so not sure why that other one isn’t…I’m not real worried about it, just thought I’d ask. the Storj-web-dashboard has great values for DQ and suspension, so I can’t image that anything is wrong with my databases.

kevink · January 3, 2021, 8:51pm

I just checked the screenshot of your last dashboard and it’s not the same as the one I get when I import the json file from the repository:

Can you please remove that dashboard again and add it from the repository?

(by the way: the suspension score from my dashboard is not the same as the suspension mentioned in the new dashboard. in the new dashboard that’s just the suspension status.)