# Prometheus Storj-Exporter **URL:** \ **Category:** getting started\ **Tags:** monitoring\ **Created:** [September 26, 2019, 10:39am UTC](https://forum.storj.io/t/prometheus-storj-exporter/1939 "2019-09-26T10:39:28Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![greener](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/greener/32/62_2.png) [@greener](https://forum.storj.io/u/greener)\ **Post date:** [September 26, 2019, 10:39am UTC](https://forum.storj.io/t/prometheus-storj-exporter/1939/1 "2019-09-26T10:39:28Z")
I’m mostly using Prometheus to monitor stuff and was in need for storj exporter to monitor things like audit counts and uptimes per satellite, data as reported by storagenode etc.. So I made one: > **[GitHub - anclrii/Storj-Exporter: Prometheus exporter for monitoring Storj storage...](https://github.com/anclrii/Storj-Exporter)** > > Prometheus exporter for monitoring Storj storage nodes > **[GitHub - anclrii/Storj-Exporter-dashboard: Storj-exporter Grafana dashboard](https://github.com/anclrii/Storj-Exporter-dashboard)** > > Storj-exporter Grafana dashboard [![](https://raw.githubusercontent.com/anclrii/Storj-Exporter-dashboard/master/storj-exporter-boom-table.png) ](https://raw.githubusercontent.com/anclrii/Storj-Exporter-dashboard/master/storj-exporter-boom-table.png) ### Checkout the [README](https://github.com/anclrii/Storj-Exporter) to setup Storj-Exporter ### Update exporter to latest version: Use `watchtower` or update manually using the following commands: ```auto docker pull anclrii/storj-exporter:latest ; \ docker stop storj-exporter ; \ docker rm storj-exporter ; \ docker run -d --restart=unless-stopped --link=storagenode --name=storj-exporter -p 9651:9651 anclrii/storj-exporter:latest # or your custom options ``` * * * ## Installing full monitoring stack Monitoring stack requires the following components: - [Storj-Exporter](https://github.com/anclrii/Storj-Exporter) - collects storagenode metrics - [Prometheus](https://prometheus.io/) server - scrapes and stores metrics from Storj-Exporter - [Grafana](https://grafana.com/) - pulls data from Prometheus and visualises it - Storj-Exporter-dashboard - a template dashboard for Storj-Exporter metrics Here’s how metrics flow through the stack: * * * **Storagenode =\> Storj-Exporter =\> Prometheus server =\> Grafana (Storj-Exporter-dashboard)** * * * ## Installation guides: - [Full monitoring stack with docker-compose](https://github.com/anclrii/Storj-Exporter-dashboard/tree/master/quick_start) - [How to monitor all nodes in your LAN using prometheus + grafana [linux using docker]](https://forum.storj.io/t/how-to-monitor-all-nodes-in-your-lan-using-prometheus-grafana-linux-using-docker/10998) by @kevink - [Guide in German](https://addictedtocode.de/storj/das-monitoring-von-storj-storagenodes/) by @unrealSpeedy - [Ansible role storj-beastmode](https://github.com/toconspiracy/storj-beastmode/tree/stable) by @Arkina ([forum thread](https://forum.storj.io/t/ansible-deployment/681/10)) - [Windows Prometheus Exporter Windows Step By Step Guide](https://forum.storj.io/t/windows-prometheus-exporter-windows-step-by-step-guide/14539) ## Official installation docs for Prometheus/Grafana: - Prometheus - [Installation | Prometheus](https://prometheus.io/docs/prometheus/latest/installation/) - Grafana - [Install Grafana | Grafana documentation](https://grafana.com/docs/grafana/latest/installation/) ## F.A.Q: - Why is Summary tables (boom table) showing `No data` for DQ SU columns and sorting doesn’t match screenshots - There are bugs in `v1.4.0` of the plugin. Installing `yesoreyeram-boomtable-panel plugin 1.3.0` resolves this. ---
**Author:** ![kevink](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/kevink/32/6077_2.png) [@kevink](https://forum.storj.io/u/kevink)\ **Post date:** [September 26, 2019, 12:12pm UTC](https://forum.storj.io/t/prometheus-storj-exporter/1939/2 "2019-09-26T12:12:37Z")
Works great so far. Thanks. Do you have multiple nodes to see how those could be visualized together? ---
**Author:** ![greener](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/greener/32/62_2.png) [@greener](https://forum.storj.io/u/greener)\ **Post date:** [September 26, 2019, 12:32pm UTC](https://forum.storj.io/t/prometheus-storj-exporter/1939/3 "2019-09-26T12:32:17Z")
I use metrics queries like `storj_sat_audit_successCount{instance=~"$node.*"}` and if panel options are set as below it will generate a panel per node. ![repeat](https://lc-storj.ams3.digitaloceanspaces.com/original/1X/159cae8aac518d9fe35d9109976f0f63743811b4.jpeg) You could probably show per satellite graphs for multiple nodes on one graph as well using the query above so that it would plot 8 graphs for 2 nodes. I am planning to support multiple nodes on the dashboard. ---
**Author:** ![greener](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/greener/32/62_2.png) [@greener](https://forum.storj.io/u/greener)\ **Post date:** [October 6, 2019, 11:23am UTC](https://forum.storj.io/t/prometheus-storj-exporter/1939/4 "2019-10-06T11:23:46Z")
# v0.2.0 update I added some missing metrics available in SN api and refactored some repeating code. Also aggregated most metrics by adding labels to metric group instead of having separate metric for each item. #### I noticed 100+ downloads on docker hub 😲 so I’m going to keep both old metrics and new aggregated metrics until at least v1.0.0 release as this would break existing dashboards. Please switch to new metrics names/labels until then. For example: ```auto storj_sat_audit_successCount{satellite="121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6"} 66.0 storj_sat_audit_totalCount{satellite="121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6"} 66.0 storj_sat_audit_beta{satellite="121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6"} 0.0 storj_sat_audit_alpha{satellite="121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6"} 19.0 storj_sat_audit_score{satellite="121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6"} 1.0 ``` becomes: ```auto storj_sat_audit{satellite="121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6",type="successCount"} 66.0 storj_sat_audit{satellite="121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6",type="totalCount"} 66.0 storj_sat_audit{satellite="121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6",type="beta"} 0.0 storj_sat_audit{satellite="121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6",type="alpha"} 19.0 storj_sat_audit{satellite="121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6",type="score"} 1.0 ``` Here are all the new metric names, I expect these will remain unchanged for as long as api exposes them: ```auto ## aggregated storj_total_diskspace storj_total_bandwidth storj_sat_summary storj_sat_audit storj_sat_uptime ## added in 0.2.0 storj_sat_month_egress storj_sat_month_ingress storj_sat_day_egress storj_sat_day_ingress storj_sat_month_storage storj_sat_day_storage ``` - `storj_sat_month*` - metrics expose a sum of all daily records for current month per `type` (`'repair','audit','usage'` etc) - `storj_sat_day*` - metrics expose a values from the last daily record (current day) per `type` (`'repair','audit','usage'` etc) ### To update to latest version run the following: ```auto docker pull anclrii/storj-exporter:latest ; \ docker stop storj-exporter ; \ docker rm storj-exporter ; \ docker run -d --restart=unless-stopped --link=storagenode --name=storj-exporter -p 9651:9651 anclrii/storj-exporter:latest # or your custom options ``` I’m still drafting the Grafana board, stay tuned … ---
**Author:** ![Cmdrd](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/cmdrd/32/1128_2.png) [@Cmdrd](https://forum.storj.io/u/Cmdrd)\ **Post date:** [October 8, 2019, 3:57am UTC](https://forum.storj.io/t/prometheus-storj-exporter/1939/5 "2019-10-08T03:57:06Z")
Working as expected, thanks a lot for putting this together! Hopefully Storj provides good documentation for the Storage Node API going forward to utilize for this. I tried enabling the debug port as per the town hall Q/A suggestion but the metrics being exposed are not in this type of format so were not really usable for ingestion into Prometheus, but this solves that problem. I have a systemd service built for the standalone script and just making sure that it works properly then probably submit a PR if you want it. Pretty simple but saves some digging and testing for anyone wanting to use the standalone script. ---
**Author:** ![greener](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/greener/32/62_2.png) [@greener](https://forum.storj.io/u/greener)\ **Post date:** [October 8, 2019, 10:17pm UTC](https://forum.storj.io/t/prometheus-storj-exporter/1939/6 "2019-10-08T22:17:06Z")
I’m glad it works for you! Systemd service sounds great, please do raise a PR. ---
**Author:** ![greener](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/greener/32/62_2.png) [@greener](https://forum.storj.io/u/greener)\ **Post date:** [October 8, 2019, 10:27pm UTC](https://forum.storj.io/t/prometheus-storj-exporter/1939/7 "2019-10-08T22:27:39Z")
I just published Grafana Storj Exporter dashboard that I use with this exporter. It took a while to figure out the units and formulas for api numbers and some bits might still be off but most items seem accurate. Do not rely on Earnings tile much as I only got 1 payout so far 😉 will confirm on next round. > **[anclrii/Storj-Exporter-dashboard](https://github.com/anclrii/Storj-Exporter-dashboard)** > > Storj-exporter Grafana dashboard. Contribute to anclrii/Storj-Exporter-dashboard development by creating an account on GitHub. Storj-exporter Grafana dashboard to visualise [Storj-Exporter](https://github.com/anclrii/Storj-Exporter) metrics for multiple Storj storage nodes. ## combined dashboard ![combined dashboard](https://github.com/anclrii/Storj-Exporter-Dashboard/raw/master/combined%20dashboard.png) ## details dashboard ![details dashboard](https://github.com/anclrii/Storj-Exporter-Dashboard/raw/master/details%20dashboard.png) ---
**Author:** ![kevink](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/kevink/32/6077_2.png) [@kevink](https://forum.storj.io/u/kevink)\ **Post date:** [October 9, 2019, 6:06am UTC](https://forum.storj.io/t/prometheus-storj-exporter/1939/8 "2019-10-09T06:06:08Z")
This is quite awesome! I pull stats every 10 minutes and don’t get a smooth graph that way. Any way to fix this or do I really need to pull stats every minute? ![grafik](https://lc-storj.ams3.digitaloceanspaces.com/original/1X/77fc1fb5a39209baef63dea7a74011431763c4dd.png) ---
**Author:** ![kevink](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/kevink/32/6077_2.png) [@kevink](https://forum.storj.io/u/kevink)\ **Post date:** [October 9, 2019, 6:09am UTC](https://forum.storj.io/t/prometheus-storj-exporter/1939/9 "2019-10-09T06:09:29Z")
Also there’s a slight problem if you run multiple nodes on the same host (same ip, different ports) in the details view as those get combined to one node: ![grafik](https://storj-s3.bcdn.literatehosting.net/original/1X/b2aa6ce8e1c7cac4d3b4bebd227ec27b5420fdc2.png) ---
**Author:** ![greener](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/greener/32/62_2.png) [@greener](https://forum.storj.io/u/greener)\ **Post date:** [October 9, 2019, 6:29am UTC](https://forum.storj.io/t/prometheus-storj-exporter/1939/10 "2019-10-09T06:29:48Z")
You would need to update rate intervals from [5m] to at least [10m]. I’ll try to move this to variables so can be selected for all graphs. ---
**Author:** ![kevink](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/kevink/32/6077_2.png) [@kevink](https://forum.storj.io/u/kevink)\ **Post date:** [October 9, 2019, 6:48am UTC](https://forum.storj.io/t/prometheus-storj-exporter/1939/11 "2019-10-09T06:48:44Z")
Thanks. Interestingly this only changed some graphs and even that not perfectly (10m scrape interval, 60m rate interval in graph): ![grafik](https://lc-storj.ams3.digitaloceanspaces.com/original/1X/88ebec5b01037145c4eaa736c1c6c8170a295b71.png) Looks like I should change my scrape interval to 5 minutes and check again after a while. ---
**Author:** ![greener](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/greener/32/62_2.png) [@greener](https://forum.storj.io/u/greener)\ **Post date:** [October 9, 2019, 6:53am UTC](https://forum.storj.io/t/prometheus-storj-exporter/1939/12 "2019-10-09T06:53:58Z")
Hmm… Did you update queries for each graph? ---
**Author:** ![kevink](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/kevink/32/6077_2.png) [@kevink](https://forum.storj.io/u/kevink)\ **Post date:** [October 9, 2019, 6:54am UTC](https://forum.storj.io/t/prometheus-storj-exporter/1939/13 "2019-10-09T06:54:25Z")
Yes I replaced every occurence of [5m] with [60m] Edit: With scrape interval 5 minutes it works correctly. With 10 minutes not even the graph in prometheus directly was smooth so maybe it has something to do with the exporter? ![grafik](https://storj-s3.bcdn.literatehosting.net/original/1X/7518d5ad70005d72ba09359f004288920e382cc3.png) ---
**Author:** ![greener](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/greener/32/62_2.png) [@greener](https://forum.storj.io/u/greener)\ **Post date:** [October 9, 2019, 8:08pm UTC](https://forum.storj.io/t/prometheus-storj-exporter/1939/14 "2019-10-09T20:08:03Z")
@kevink Here’s what I found - [https://github.com/prometheus/prometheus/blob/81d284f806ef828e8f323088e00871fedd5a77c2/docs/querying/basics.md#staleness](https://github.com/prometheus/prometheus/blob/81d284f806ef828e8f323088e00871fedd5a77c2/docs/querying/basics.md#staleness) It seems prometheus treats metrics \> 5m old as stale by default. For prometheus v2+ you can override this interval with `--query.lookback-delta=5m` commandline option. That should help with the graphs but will also delay the graph showing the node stopped reporting. In general they recommend to keep scrape interval to \< 2m with default lookback-delta=5m which probably means you want to set `lookback-delta=25m` with 10m scrapes… I didn’t test this though and not sure if this is a good idea. Also check this post [https://stackoverflow.com/questions/56882734/grafana-dashboard-not-showing-data-when-zoomed-out](https://stackoverflow.com/questions/56882734/grafana-dashboard-not-showing-data-when-zoomed-out) ---
**Author:** ![greener](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/greener/32/62_2.png) [@greener](https://forum.storj.io/u/greener)\ **Post date:** [October 9, 2019, 8:19pm UTC](https://forum.storj.io/t/prometheus-storj-exporter/1939/15 "2019-10-09T20:19:09Z")
For your other problem try to change `$node` variable `regex` to `/.*/` instead of `/([^:]+):.*/` This will render ports in $node variables and should fix your issue. The reason I strip the port is because I’m using other node-exporter and cadvisor with this same dashboard and each exporter has a different port for the same host. ---
**Author:** ![kevink](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/kevink/32/6077_2.png) [@kevink](https://forum.storj.io/u/kevink)\ **Post date:** [October 9, 2019, 8:25pm UTC](https://forum.storj.io/t/prometheus-storj-exporter/1939/16 "2019-10-09T20:25:18Z")
Oh that is interesting, thanks a lot, I didn’t think about that. Scraping that often will make the database quite big if I want to keep my whole stats for a year… Have to check the links very carefully and think about how I want to set up my system… ---
**Author:** ![kevink](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/kevink/32/6077_2.png) [@kevink](https://forum.storj.io/u/kevink)\ **Post date:** [October 9, 2019, 8:25pm UTC](https://forum.storj.io/t/prometheus-storj-exporter/1939/17 "2019-10-09T20:25:35Z")
> For your other problem try to change `$node` variable `regex` to `/.*/` instead of `/([^:]+):.*/` > This will render ports in $node variables and should fix your issue. Thanks, that worked perfectly. ---
**Author:** ![greener](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/greener/32/62_2.png) [@greener](https://forum.storj.io/u/greener)\ **Post date:** [October 9, 2019, 9:54pm UTC](https://forum.storj.io/t/prometheus-storj-exporter/1939/18 "2019-10-09T21:54:20Z")
Updated the grafana dashboard to match units to the recent update: > **[anclrii/Storj-Exporter-dashboard](https://github.com/anclrii/Storj-Exporter-dashboard)** > > Storj-exporter Grafana dashboard. Contribute to anclrii/Storj-Exporter-dashboard development by creating an account on GitHub. ---
**Author:** ![greener](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/greener/32/62_2.png) [@greener](https://forum.storj.io/u/greener)\ **Post date:** [October 13, 2019, 4:05pm UTC](https://forum.storj.io/t/prometheus-storj-exporter/1939/19 "2019-10-13T16:05:46Z")
# v0.2.3 update - remove duplicate labels - [fix failing on missing sat api data](https://github.com/anclrii/Storj-Exporter/commit/28a107decdc1789e0338ce8d8cc41caa908a2dd7) > **[anclrii/Storj-Exporter](https://github.com/anclrii/Storj-Exporter)** > > Prometheus exporter for monitoring Storj storage nodes - anclrii/Storj-Exporter ### To update to latest version run the following: ```auto docker pull anclrii/storj-exporter:latest ; \ docker stop storj-exporter ; \ docker rm storj-exporter ; \ docker run -d --restart=unless-stopped --link=storagenode --name=storj-exporter -p 9651:9651 anclrii/storj-exporter:latest # or your custom options ``` Watchtower might also pickup the update if configured Last time I forgot to tag new version to latest so the update might not have worked. Should work this time. # Grafana dashboard update - show score \* 100 as percent for uptime and audits > **[anclrii/Storj-Exporter-dashboard](https://github.com/anclrii/Storj-Exporter-dashboard)** > > Storj-exporter Grafana dashboard. Contribute to anclrii/Storj-Exporter-dashboard development by creating an account on GitHub. ---
**Author:** ![kevink](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/kevink/32/6077_2.png) [@kevink](https://forum.storj.io/u/kevink)\ **Post date:** [October 13, 2019, 5:33pm UTC](https://forum.storj.io/t/prometheus-storj-exporter/1939/20 "2019-10-13T17:33:28Z")
Thanks a lot for the update! I won’t be able to confirm the fix since that satelite since has data for my node. [Next page](https://forum.storj.io/t/prometheus-storj-exporter/1939.md?page=2)