Prometheus Storj-Exporter

Node Operators getting started
44

It looks like your query inspector response prints the complete output of storj-exporter.

Did you by any chance add storj-exporter as a prometheus datasource in grafana?
The setup should be storj-exporter => prometheus-server => grafana , do you have the prometheus server?

Currently setup instructions are missing indeed. Setup is fairly straight-forward and works out of the box for the most part, but it does require some experience with prometheus. I will probably be adding some instructions, maybe docker compose to make it easier.

45

Ok i got it now.

How can i change the Labels? here my prometheus.yml:

My Home-Node here

- targets: ['storj-exporter:9651']
  labels:
    instance: 'Home'

# Second Node External
- targets: ['<ip-of-external-server>:9651']
  labels:
    instance: 'External-Node-1'

- targets: [<ip-of-anotner-external-server>':4142']
  labels:
    instance: 'External-Node-2'

- targets: [':<ip-of-another-different-external-server>4103']
  labels:
    instance: 'External-Node-3'

If i use this i dont get any data on the lower part of the grafana-dashboard. The problem is there are querys like “storj_diskSpace_used{instance=~”$node."}" but $node. does not have any data.

Bildschirmfoto%20von%202019-10-30%2013-55-43
Bildschirmfoto%20von%202019-10-30%2013-55-431598×713 110 KB

If i change this to the name of the instance it will work but i dont like to do this on this way.

46

You might need to change the variable $node in the options. I run 3 nodes on one host and had to change it to “/.*/”

Interesting that I am missing some network performance graphs on the right side:

grafik
grafik1533×847 163 KB

1 Like
47

That’s what I was doing wrong too. Can someone post their full .yml I’ve never used prometheus before and am having trouble getting it working.

48

NVM, got it working. This is pretty damn cool. Thanks!

49

hi, when i add the grafana template, i become this error:

Failed create dashboard model

undefined is not iterable (cannot read property Symbol(Symbol.iterator))

51

Hi, i’m trying my very first steps with prometheus.

So far done:

  1. Installation of Docker container with Storj-Exporter, curl 127.0.0.1:9651 works
  2. Installation of Prometheus from Debian repository, is running on port 9090
  3. Download Storj-Exporter-dashboard

Q1: were to put the .json file for dashboard? (/etc/prometheus/…?)
Q2: what lines needed to add in /etc/prometheus/prometheus.yml?
Q3: missing further steps?

2 Likes
52

These are the lines I added at the bottom of /etc/prometheus/prometheus.yml

job_name: storj
static_configs:
- targets: ['localhost:9651']

And the .json file for the dashboard, you import that in Grafana through your browser.

1 Like
53

Hello,

This is a wonderful project, I really appreciate your efforts!
However, I am running in to trouble while setting it up. Here’s the deal:

I have a total of three nodes, 2 on 1 machine and 1 on another machine. I am still trying to set up the nodes running on the same machine (they run on different ports). Everything “appears” to be fine, however the data shown in Grafana is absolutely nonsensical.

To lay things out:
Node 1 - storagenode - runs on ports (external:internal) 14002:14002. It has 1.6TB allocated and is full.
Node 2 - storagenode3tb - runs on 14001:14002, respectively. It’s relatively new and has a few GB used with a total of 2.4TB allocated.

Here are excerpts from the Dockerfiles for each node’s Prometheus docker image:
storagenode:
ENV STORJ_HOST_ADDRESS=storagenode STORJ_API_PORT=14002 STORJ_EXPORTER_PORT=9651
It runs on external port 9650.

storagenode3tb:
ENV STORJ_HOST_ADDRESS=storagenode3tb STORJ_API_PORT=14002 STORJ_EXPORTER_PORT=9651
This one runs on external port 9650.

At first glance, everything seems fine, for example here is the disk usage reported by each Prometheus instance:
storagenode:

# TYPE storj_diskSpace_used gauge
storj_diskSpace_used 1.600012061184e+012

storagenode3tb:

# TYPE storj_diskSpace_used gauge
storj_diskSpace_used 6.422592256e+09

The Prometheus server has the following appended to prometheus.yml:

  - job_name: storagenode
    static_configs:
    - targets: ['192.168.13.73:9651', '192.168.13.73:9650']

and the data sources are scraped and evaluated every 60s.

Now, getting to Grafana and its nonsense. Here you will see 5 screen snips - 2 per node, refreshing the webpage immediately after taking the first screenshot for each. Last one is a view of “All” nodes.
The data doesn’t make sense! :frowning:

[Album] Imgur

Any help would be much appreciated!

54

@Krawi hope this worked for you. I will try to find some time to update the readme with detailed installation steps and maybe create all in one solution to deploy prometheus/grafana/storj-exporter in docker.

55

This does look weird, did you manage to resolve?
There are 3 components at play in this dashboard setup:

Once the above is all set the storj-exporter dashboard should be showing graphs. Can you confirm you have all set as above, specifically the data source in grafana?

I know about at least 2 issues which could break the graphs still:

  • One is with running multiple storj nodes on the same IP in which case setting the Node variable to /.*/ will help. I’ll update the dashboard to the same in the repo. Looking at the screenshots it doesn’t seem to be the issue.
  • Another is with new nodes that were started just recently. Storagenode is not exposing some api entries for such nodes until they get some values other then 0. I updated exporter to treat these missing entries as 0 but I don’t have new nodes to test this on and not sure if I fixed all relevant entries.
2 Likes
56

Is it possible to set this up on a Windows machine using the GUI version of Storj?
Asking because I reinstalled Windows and got stuck on the exporter. There is no docker container to pull data out from.

57

No, for now I have given up on it because I can’t dedicate the time.
It is very strange because every single page refresh the data is completely different.

For the record, I have indeed set the variable to /.*.
I will run it later today when my second node is a bit more mature, but right now I am troubleshooting an issue with my main node.

58

Yes, i got it finally working, thanks.

I have a suggestion for Storj-exporter: TLS-Support to secure the export to Prometheus.
Together with IP whitelisting the host which runs Prometheus by the firewall on the Storj node, this should protect the data if sent over internet.

59

It should be possible as long as gui version also exposes the api. Can you check if anything is listening on port 14002 on windows or if it can be enabled? Then you should be able to run the python script/docker to pull data from that port.

60

Hello, i have now another issue.

Setup a second node on a different host / IP, changed the prometheus.yml:

  - job_name: storj
    static_configs:
      - targets: ['domain.tld:9651']
        labels:
          instance: 'Cloud'

      - targets: ['storj-01.domain.tld:9651']
        labels:
          instance: 'Node1'

The Summary dashboard looks OK:
Summary

But the two nodes seems to be merged:
Node%20Select

Consequently the Details dashboard is showing only this ‘All’ entry:

All%20Details
All%20Details711×351 87.1 KB

Anybody with an idea what’s need to changed in my setup?

1 Like
61

@Krawi try this or reimport the dashboard from the repo, I updated it yesterday for the same.

1 Like
62

Updated dashboard and additionaly a new one, feels like Xmas today! :grinning:

63

I have just started to test storj-exporter and am new to Grafana and Prometheus. I have 2 nodes on my server (one that I started more or less a day ago). So far I only see one node on the dashboard which corresponds to my oldest one. How long am I supposed to wait until it appears on the dashboard?

Many thanks!

64

Did you set up a 2nd exporter for the new node?