How to monitor all nodes in your LAN using prometheus + grafana [linux using docker]

lyoth · January 4, 2023, 12:23am

Use

sum(storj_payout_currentMonth{type=~“currentMonthExpectations”,instance=~“$node.*”}) / 100

Walter1 · January 9, 2023, 2:23pm

Hello together,

I’m trying to use a different port than 14002 but it doesn’t seem to work.

docker run -d --restart=unless-stopped --name=STORJ-1-Exporter --link=STORJ-1 -p 9651:9651 -e STORJ_HOST_ADDRESS=111.111.111.22 -e STORJ_API_PORT:14003 anclrii/storj-exporter:latest

It still wants 14002 instead of 14003 even with STORJ_API_PORT. Does anyone know what to do?

Thanks and kind regards,

Edit: I got it. You need to assign the API PORT also with = instead of :.

Tomaae · January 10, 2023, 2:03am

When you use -e, first value is port on host, second is port on container.
If you are connecting to container directly, it will still be original port. on host it will be redirected port.

Alexey · January 10, 2023, 4:56am

should be -e STORJ_API_PORT=14003
And make sure that it’s published in your storagenode container with -p 14003:14002 or -p 111.111.111.22:14003:14002

that’s right

marckt · February 13, 2023, 5:11am

Something went wrong with the new version 1.72.2

Traceback (most recent call last):
  File "./storj-exporter.py", line 236, in <module>
    REGISTRY.register(StorjCollector())
  File "/usr/local/lib/python3.7/site-packages/prometheus_client/registry.py", line 26, in register
    names = self._get_names(collector)
  File "/usr/local/lib/python3.7/site-packages/prometheus_client/registry.py", line 66, in _get_names
    for metric in desc_func():
  File "./storj-exporter.py", line 192, in collect
    yield from self.add_sat_metrics()
  File "./storj-exporter.py", line 130, in add_sat_metrics
    yield from self.add_extended_sat_metrics(sat)    
  File "./storj-exporter.py", line 153, in add_extended_sat_metrics
    data            = sat['sat_data'].get('audits', None)
AttributeError: 'NoneType' object has no attribute 'get'
Traceback (most recent call last):
  File "./storj-exporter.py", line 236, in <module>
    REGISTRY.register(StorjCollector())
  File "/usr/local/lib/python3.7/site-packages/prometheus_client/registry.py", line 26, in register
    names = self._get_names(collector)
  File "/usr/local/lib/python3.7/site-packages/prometheus_client/registry.py", line 66, in _get_names
    for metric in desc_func():
  File "./storj-exporter.py", line 192, in collect
    yield from self.add_sat_metrics()
  File "./storj-exporter.py", line 130, in add_sat_metrics
    yield from self.add_extended_sat_metrics(sat)    
  File "./storj-exporter.py", line 153, in add_extended_sat_metrics
    data            = sat['sat_data'].get('audits', None)
AttributeError: 'NoneType' object has no attribute 'get'
Traceback (most recent call last):
  File "./storj-exporter.py", line 236, in <module>
    REGISTRY.register(StorjCollector())
  File "/usr/local/lib/python3.7/site-packages/prometheus_client/registry.py", line 26, in register
    names = self._get_names(collector)
  File "/usr/local/lib/python3.7/site-packages/prometheus_client/registry.py", line 66, in _get_names
    for metric in desc_func():
  File "./storj-exporter.py", line 192, in collect
    yield from self.add_sat_metrics()
  File "./storj-exporter.py", line 130, in add_sat_metrics
    yield from self.add_extended_sat_metrics(sat)    
  File "./storj-exporter.py", line 153, in add_extended_sat_metrics
    data            = sat['sat_data'].get('audits', None)
AttributeError: 'NoneType' object has no attribute 'get'
Traceback (most recent call last):
  File "./storj-exporter.py", line 236, in <module>
    REGISTRY.register(StorjCollector())
  File "/usr/local/lib/python3.7/site-packages/prometheus_client/registry.py", line 26, in register
    names = self._get_names(collector)
  File "/usr/local/lib/python3.7/site-packages/prometheus_client/registry.py", line 66, in _get_names
    for metric in desc_func():
  File "./storj-exporter.py", line 192, in collect
    yield from self.add_sat_metrics()
  File "./storj-exporter.py", line 130, in add_sat_metrics
    yield from self.add_extended_sat_metrics(sat)    
  File "./storj-exporter.py", line 153, in add_extended_sat_metrics
    data            = sat['sat_data'].get('audits', None)
AttributeError: 'NoneType' object has no attribute 'get'
Traceback (most recent call last):
  File "./storj-exporter.py", line 236, in <module>
    REGISTRY.register(StorjCollector())
  File "/usr/local/lib/python3.7/site-packages/prometheus_client/registry.py", line 26, in register
    names = self._get_names(collector)
  File "/usr/local/lib/python3.7/site-packages/prometheus_client/registry.py", line 66, in _get_names
    for metric in desc_func():
  File "./storj-exporter.py", line 192, in collect
    yield from self.add_sat_metrics()
  File "./storj-exporter.py", line 130, in add_sat_metrics
    yield from self.add_extended_sat_metrics(sat)    
  File "./storj-exporter.py", line 153, in add_extended_sat_metrics
    data            = sat['sat_data'].get('audits', None)
AttributeError: 'NoneType' object has no attribute 'get'
Traceback (most recent call last):
  File "./storj-exporter.py", line 236, in <module>
    REGISTRY.register(StorjCollector())
  File "/usr/local/lib/python3.7/site-packages/prometheus_client/registry.py", line 26, in register
    names = self._get_names(collector)
  File "/usr/local/lib/python3.7/site-packages/prometheus_client/registry.py", line 66, in _get_names
    for metric in desc_func():
  File "./storj-exporter.py", line 192, in collect
    yield from self.add_sat_metrics()
  File "./storj-exporter.py", line 130, in add_sat_metrics
    yield from self.add_extended_sat_metrics(sat)    
  File "./storj-exporter.py", line 153, in add_extended_sat_metrics
    data            = sat['sat_data'].get('audits', None)
AttributeError: 'NoneType' object has no attribute 'get'
Traceback (most recent call last):
  File "./storj-exporter.py", line 236, in <module>
    REGISTRY.register(StorjCollector())
  File "/usr/local/lib/python3.7/site-packages/prometheus_client/registry.py", line 26, in register
    names = self._get_names(collector)
  File "/usr/local/lib/python3.7/site-packages/prometheus_client/registry.py", line 66, in _get_names
    for metric in desc_func():
  File "./storj-exporter.py", line 192, in collect
    yield from self.add_sat_metrics()
  File "./storj-exporter.py", line 130, in add_sat_metrics
    yield from self.add_extended_sat_metrics(sat)    
  File "./storj-exporter.py", line 153, in add_extended_sat_metrics
    data            = sat['sat_data'].get('audits', None)
AttributeError: 'NoneType' object has no attribute 'get'

striker43 · February 13, 2023, 7:58am

I have exactly the same issue, but strangely not on all of my nodes…

Krystof · February 13, 2023, 11:08am

I have the same issue on 3 exporter on synology
remove storj-exporter
restart storagenode
create storj-exporter with connect to IP and port directly

work to me

Dunc4n1d4h0 · February 13, 2023, 3:28pm

Strange, I have node set with -p 14003:14002, and I don’t need STORJ_API_PORT at all.
It seems that --link is for that.

I just used:

docker run -d --link=storagenode1 --name=storj-exporter1 -p 9651:9651 -e STORJ_HOST_ADDRESS=storagenode1 anclrii/storj-exporter:latest
docker run -d --link=storagenode2 --name=storj-exporter2 -p 9652:9651 -e STORJ_HOST_ADDRESS=storagenode2 anclrii/storj-exporter:latest

Alexey · February 14, 2023, 2:49am

because the exporter will access nodes via internal containers network (you specifically linked them with --link option), all your nodes listening on 14002, unless you changed this port with console.address: parameter in your config.yaml or with argument --console.address after the image name in your docker run command.
I think the exporter will access the default 14002 port unless changed with environment variable STORJ_API_PORT.
However, if you configure exporter on a different network (I.e. would not use --link), it will require to use an external port (which you mapped on your host), but may use the LAN address of your host, not the external (because otherwise you will be forced to expose your node’s dashboard to the internet without any protection).

Walter1 · March 27, 2023, 2:34pm

Hello together,

currently I’m trying the following command:

docker run -d -p 9090:9090 --restart unless-stopped --user root:root --name prometheus
-v /mnt/apd/prometheus/config/prometheus.yml:/etc/prometheus/prometheus.yml
-v /mnt/apd/prometheus/data:/prometheus
prom/prometheus --storage.tsdb.retention.time=360d --storage.tsdb.retention.size=30GB
–config.file=/etc/prometheus/prometheus.yml --storage.tsdb.path=/prometheus

And Prometheus is running, but it is not fetching the config file in /mnt/apd/prometheus/config/prometheus.yml.

Does anyone see the mistake here?

Vadim · March 27, 2023, 2:42pm

if you want it work on local network may be you need specify pc ip here on what interface it will run, other it wil run on localhost and will be avalable only in this pc

Walter1 · March 27, 2023, 2:49pm

?

It’s fine and working as intend. Hat to expand the scrape pools in the upper left corner, then the entries appear. In the original it is bindet with /etc but it is working with /config as well.

Alexey · March 28, 2023, 3:07am

How did you get it? Just to clarify - it will not re-read config without restart I believe.

Walter1 · March 28, 2023, 8:05am

It’s fine for now. It was fetching it but I didnt look properly in the upper left corner to unselect the first node. When you do this you can see all the nodes.

Walter1 · April 4, 2023, 4:08pm

Hello together,

when gathering the data for the dashboard I receive many interrupts:

Are there best settings for the scraping? I want to avoid scraping in seconds, great would be every minute without drawing on performance.

"# Sample config for Prometheus.

global:
scrape_interval: 60s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
evaluation_interval: 60s # Evaluate rules every 15 seconds. The default is every 1 minute.
"# scrape_timeout is set to the global default (10s).

"# Attach these labels to any time series or alerts when communicating with
"# external systems (federation, remote storage, Alertmanager).
external_labels:
monitor: ‘example’

"# Alertmanager configuration
alerting:
alertmanagers:

static_configs:
- targets: [‘localhost:9093’]

"# Load rules once and periodically evaluate them according to the global ‘evaluation_interval’.
rule_files:
"# - “first_rules.yml”
"# - “second_rules.yml”

"# A scrape configuration containing exactly one endpoint to scrape:
"# Here it’s Prometheus itself.
scrape_configs:
"# The job name is added as a label job=<job_name> to any timeseries scraped from this config.

job_name: STORJ-1

"# Override the global default and scrape targets from this job every 5 seconds.
scrape_interval: 60s
scrape_timeout: 60s

Walter1 · September 26, 2023, 3:26pm

Hello together,

currently I am facing the following problem:

When I add a new node I get the following error messages:

Can you advice what to do and how to resolve this?

arrogantrabbit · September 26, 2023, 6:14pm

You’ve cut of the exception and the top of stack… Please post entire exception message.

Copy text and place it between

```
and
```

Dr.Ko · January 5, 2024, 3:49pm

sudo docker run -d -p 9090:9090 --restart unless-stopped --user 1000:1000 --name prometheus \
	-v /volume1/prometheus/config/prometheus.yml:/etc/prometheus/prometheus.yml \
	-v /volume1/prometheus:/prometheus \
	prom/prometheus --storage.tsdb.retention.time=360d --storage.tsdb.retention.size=30GB \
	--config.file=/etc/prometheus/prometheus.yml --storage.tsdb.path=/prometheus

I don’t know why prometheus isn’t working. Does anyone know what I wrote wrong? The path where the yml file was created is /volume1/prometheus/config.
I am a Synology user.

Alexey · January 6, 2024, 3:26am

Perhaps user 1000:1000 doesn’t have full permissions on content of the path /volume1/prometheus?

Please show result of the command:

ls -n /volume1/prometheus

Dr.Ko · January 6, 2024, 5:12am

sdfdsf

I am attaching a screenshot!

I created the config and prometheus folders myself.
(mkdir /volume1/prometheus…)

1704524267794

It comes out like this when I connect to root