Prometheus Storj-Exporter

@kevink thanks will update the dashboard, can you point me to where onlineScore change was discussed please? Has anything else changed about it? Totally missed it. I added a metric for onlineScore api value but I was sure the old one still works and both are the same thing.

This is what I found:

There is definitely more but not about the API itself: Blueprint: tracking downtime with audits

1 Like

That helps thanks. What do you think would be reasonable thresholds for this and maybe audit as well? Currently I got 99.8 - orange | 99.5 - red text colour thresholds set for these cells.

That’s merged to master here for both summary and sat tables https://github.com/anclrii/Storj-Exporter-dashboard/blob/53dee2e8ac653d898f840b2745e8ad26ab86e3f7/Storj-Exporter-Boom-Table.json

I’m not quite sure either. I used red below 97%.
Currently you can have a downtime of up to 288h per months (which will change at some point) so that would be 60%. But making everything below 60% red seems really low.

But 99.5% is definitely too high. That was a very old requirement that won’t be enforced again so I wouldn’t go that high for the red threshold.

Hi,
What is the Metric code thingy for Grafana to get the storj uptime percentage? Mine still shows as 0%

It should work with latest version of exporter and dashboard. Have you updated both and still having issues?

How do I check my current version of the exporter?

I have seemed to fix the issue, for some reason my storj-exporter image wasn’t updating (I have to build it manually), but a docker prune fixed it.

Just curious if there’s any reason to run my own grafana instead of using the free cloud based one?

I don’t need more than one user, and I am not aware of more than 5 dashboards for storj?

Thanks and hope you are well.

edit - Also why dont we add the shared dashboards on there?

I have both but keep using the cloud one since I tried it. So far so good, as long as limitations work for you I don’t see any problems with it.

Not sure what you mean, add where?

My question would rather be: Why would you use a cloud grafana if you are deploying your prometheus locally anyway?

In the cloud grafana there’s a ‘shared dashboards’ where storj could have a dashboard that we all use so that we dont have to individually update all the time.’

This isn’t relevant to me as I have machines on various networks/locations so no matter where I put the grafana, it will never be local to all nodes.

Ok I see what you mean, will look into it, good idea.

2 Likes

I wish we could add this into the same docker i’m running for each storage node. Having tons of extra exporter dockers is annoying. I guess modifying storj’s stuf though would mean the updater wouldn’t work right.

Getting this setup today though and testing out that free grafana to view it all at.

1 Like

you could build your own image from both sources… probs not rcommended though :smiley:

Ideally it would be great if storj binary had the functionality to expose prometheus metrics. This is quite a common practice when building modern apps and when I started with storj I was expecting it to be there. Asking around I was pointed to a message that mentions prometheus metrics are exposed in storj debug mode but I couldn’t find it there either… so I made my own and sharing it in case it’s useful for others. I considered contributing to storj codebase directly but I figured it would take a while to negotiate and I have no experience with golang, my programming skill is very basic and python was a quick solution for me.

Since 1.0.1 release it seems to be fairly stable and fail-safe and maintenance for any upcoming changes in api should be easy enough to support it further. I was going to start looking at contributing to storj at this point but with current traffic rates I’m quickly loosing interest. I’d probably be out by now if I didn’t have the exporter to support (my earnings don’t cover my running costs currently :man_shrugging:). Though I’m still happy to support the existing exporter implementation for as long as people use it.

I’m not sure if there’s a better alternative to running it in a second container, I’d stay away from messing with the official storagenode images. There’s an option to run exporter as a systemd service if that’s more convenient, but docker works great for me so far. I’m not a fan of aggregating multiple exporters into one exporter container or tracking multiple storagenodes with one exporter. Happy to discuss other alternatives on how to improve this side of things.

1 Like

Thanks a lot man you rock! If we ever cross paths, the beer is on me.

How can I find Docker container hostname or FQDN to fill - STORJ_HOST_ADDRESS ? Thank you.