Prometheus Storj-Exporter

[kevink]

I’m working on a combined dashboard, it’ll be available soon (1-2 weeks maybe).

As for the exporter, a combination is not possible. Not because of the port but because they work completely different.
greeners exporter transforms the API into prometheus format using a python prometheus library.
My exporter is basically just a pattern analyzer that I didn’t write myself, it’s a go library that I configured to find all information in the logs.

So they work completely different and use different libraries. Furthermore, it’s not good practice to mix software for different purposes. His uses the API, mine uses logfiles.
If it were easily possible, I’d have no problem combining both exporters but there isn’t really. If there is a python library to parse the logfiles in a similar way (using grok patterns) then it would be easier and could be considered, even though it’s a combination of 2 different workloads.

[Arkina]

Thx for the Responses. Indeed both exporters have different sources but the same output in the end, which would be Prometheus.

@kevink

it’s not good practice to mix software for different purposes. His uses the API, mine uses logfiles.

Therefore i see them as software with the same purpose, but i guess that more like a philosophical discussion ^^

I’ll had a short look at it and it really is not easy to merge, so two containers do not seem to be a bad solution for now.

[greener]

Also if I understand this post correctly

Grafana Metrics

Grafana is awesome. It would allow the storage node community to setup a grafana dashboard inclusive email alerts. The email alerts could get fired within 5 minutes after a problem was detected instead of hours later when the satellite might notice it. Currently, there is already a grafana dashboard but it requires running a log scraper process. It would be nice if the storage node could send these metrics directly. The storage node already contains a metric endpoint. So we just need to make su…

this suggests that potentially upload/download counts may be exposed in the api which would make it easy to integrate for me. Better yet storj binary might expose prometheus metrics directly making my exporter obsolete.

[kevink]

@greener

Better yet storj binary might expose prometheus metrics directly making my exporter obsolete.

yeah I guess if that ever happens it won’t be within this year Probably too many more important things to do, especially because there already is a good community solution

@greener

this suggests that potentially upload/download counts may be exposed in the api which would make it easy to integrate for me

which would be great too… Counting the log lines works well but if you ever restart the exporter, it messes with your count for a little while as it misses some log lines.

[greener]

@kevink

I’m working on a combined dashboard, it’ll be available soon (1-2 weeks maybe).

Not sure if you want to publish it under own repo, but you’re welcome to use mine so it’s all in one place. I just added you as collaborator but no pressure

[kevink]

oh thanks!
Guess it could also be useful in case there’s a bug in the exporter or an api changes if that’s alright of course.

About the dashboard: I guess I might make a PR first since I’m kind of changing some fundamental things like showing ingress as positive and egress as negative, because that’s how netdata does it (net in as positive, net out as negative) but your dashboard is the other way around. So not sure how you like those changes Also I kind of don’t like the looks of some of the new graphs so overall the dashboard would be quite different from your new one (even though it serves as the basis lol).

Edit: oh wait I forgot that the dashboard and exporter repositories are separated

[greener]

I’ll add you to the exporter one as well just in case

As to the dashboard, I think you can add it alongside the existing one so that both are available. Those changes sound good to me and we can backport them to existing dashboard for consistency. I don’t like some of the graphs either lol

[kevink]

ok Sounds good, thx.

[greener]

@greener

I’ve updated repo build action to build multi-architecture docker images

That’s 1.0.8 published with latest tag for multiple architectures/platforms that include Raspberry Pi.
Now docker pull anclrii/storj-exporter:latest should work on rpi as well and no need to build the image.

[waistcoat]

How are the plans for the api change going? I’ve updated one of my nodes to 1.21.1 as it’s live now on the update servers.

On casual inspection only the Node Summary Table seems to be affected on the dashboard.

	Net Out	Net In	Disk Total	Disk Used	Trash	Payout	Held	Uptime	Audit	Version	UTD	DQ	SU
node01	280.22 GB	43.14 GB	1.900 TB	1.896 TB	3.52 GB	$7.65	$1.91	99.98%	100.00%	1.20.2
node02	14.19 GB	216.45 GB	550.000 GB	251.907 GB	12.98 GB	$0.54	$0.27	undefined	undefined	1.21.1

[kevink]

You could try building the exporter or using the dev image.
But changes will be merged once greener can confirm the api changes I guess, which means once the docker image is updated.

[kevink]

On a side note: I published a new dashboard:

Log-Exporter for Prometheus with Grafana Dashboard

Time for an update: You need to update the log-exporter container! because with the last update storjlabs added a “size” attribute to all upload messages and I only added it to the upload_successful… So you are missing information at the moment. New Dashboard I created a new dashboard that also needs the storj-exporter from @greener Prometheus Storj-Exporter and can be found in his dashboard repository as well as mine. You can download it from here: https://raw.githubusercontent.com/kevinkk…

[waistcoat]

@kevink

You could try building the exporter or using the dev image.
But changes will be merged once greener can confirm the api changes I guess, which means once the docker image is updated.

Had seen the PR but hadn’t realised it was from the dev branch. Yes I’m happy to test if that’s helpful. I wondered since Storj Labs don’t seem to version their API (which is annoying), whether we should check the version manually first and then do the appropriate query. This would allow us to decouple the timing of the changes and just do the right thing. i.e. >= 1.21.x do new query, less do old one. Just an idea.
I take it no dashboard changes will be required.
Because I’m using TrueNAS and FreeBSD jails, I don’t use docker for my StorageNodes and probably won’t be running a log exporter anytime soon.
I do use docker on a completely different linux host running exporter(s), prometheus and grafana.

[kevink]

Yeah API versioning would be great. It’s just a bit more work to change the exporter code now to support API versioning but it’s doable. PRs appreciated

Would be helpful if you could confirm that the changes work as expected and nothing else changed.
Dashboard doesn’t need changes, the metrics in prometheus stay the same.

BTW: you can run the log-exporter without docker too, I just haven’t documented that. You’d need the grok-exporter from the original repository and use my configuration file, then it’ll work too. Before creating a docker container, that was the way I tested it. For most people it’s just easier to use docker.

[waistcoat]

Build the dev branch and deployed it on the exporter pointed at the 1.21.1 storagenode. I’m still getting undefined in the Node summary table in the grafana dashboard, however probing the ports directly looks OK.

storagenode on old api with old exporter
# TYPE storj_sat_audit gauge storj_sat_audit{satellite="1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE",type="totalCount",url="saltlake.tardigrade.io:7777"} 3752.0 storj_sat_audit{satellite="1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE",type="successCount",url="saltlake.tardigrade.io:7777"} 3752.0 storj_sat_audit{satellite="1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE",type="alpha",url="saltlake.tardigrade.io:7777"} 20.0 storj_sat_audit{satellite="1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE",type="beta",url="saltlake.tardigrade.io:7777"} 0.0 storj_sat_audit{satellite="1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE",type="unknownAlpha",url="saltlake.tardigrade.io:7777"} 20.0 storj_sat_audit{satellite="1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE",type="unknownBeta",url="saltlake.tardigrade.io:7777"} 0.0 storj_sat_audit{satellite="1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE",type="score",url="saltlake.tardigrade.io:7777"} 1.0 storj_sat_audit{satellite="1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE",type="unknownScore",url="saltlake.tardigrade.io:7777"} 1.0

storagenode on new api with dev exporter
# TYPE storj_sat_audit gauge storj_sat_audit{satellite="1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE",type="auditScore",url="saltlake.tardigrade.io:7777"} 1.0 storj_sat_audit{satellite="1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE",type="suspensionScore",url="saltlake.tardigrade.io:7777"} 1.0 storj_sat_audit{satellite="1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE",type="onlineScore",url="saltlake.tardigrade.io:7777"} 1.0 storj_sat_audit{satellite="1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE",type="satelliteName",url="saltlake.tardigrade.io:7777"} 0.0

[kevink]

Thanks, that looks good.

If everything looks good in the dashboard, it might be time for @greener to merge the PR.

[greener]

@waistcoat

decouple the timing of the changes and just do the right thing

This would be great indeed but there’s not enough material to work with there. There is no api versioning but also it’s not even clear what the changes are going to be :), reading A couple changes to SNO API - #22 by kevink

It’s not clear if total audit count is removed for example. If you can share satellite api response for new version like curl -s 'http://localhost:14001/api/sno/satellite/12tRQrMTWUWwzwGh18i7Fqs67kmdhH9t6aToeiwbo5mfS2rUmo' | jq that would help.

I just rebased the dev branch to bring in the multi-arch build and it built new images for all platforms. Can test it with docker pull anclrii/storj-exporter:1.0.9-beta.1. On my dashboard this version removed Vetting and Audit columns on satellite summary table, as well as undefined on Audit for node summary table.

[waistcoat]

@kevink

Thanks, that looks good.

If everything looks good in the dashboard, it might be time for @greener to merge the PR.

As I said it doesn’t look good in my dashboard.
I’m using: Storj-Exporter-Boom-Table.json, and it seems to be hard-coded to the values output by the exporter.
It appears to be using storj_sat_audit{type=\"successCount\" and storj_sat_audit{type=\"score\" to output the Uptime% and Audit% counts, with the exporter changes these no longer exist. I think it would need to be storj_sat_audit{type=\"onlineScore\" and storj_sat_audit{type=\"auditScore\".
This is going to become a bit of a pain if we have to keep everything in sync and more changes are made to the API.

[kevink]

@waistcoat

This is going to become a bit of a pain if we have to keep everything in sync and more changes are made to the API.

yeah it will be… same for my log exporter. in the last update they added a parameter to the upload logs… but at least they are already trying by giving us some advance notice so we can prepare. Not sure we can do anything else at the moment to improve the situation.

@waistcoat

It appears to be using storj_sat_audit{type=\"successCount\" and storj_sat_audit{type=\"score\" to output the Uptime% and Audit% counts, with the exporter changes these no longer exist. I think it would need to be storj_sat_audit{type=\"onlineScore\" and storj_sat_audit{type=\"auditScore\" .

oh you’re totally right about that… Guess there is a need for an updated dashboard.

[CryptoPumpkin]

Posting a big thanks for this awesome work of yours!
I just set it up on my Unraid server nodes. Had some trouble with network and with enviroment variable (STORJ_HOST_ADDRESS) not binding inside the container, but got them changed and finally all seems to work well.
Thanks! This is way better than the storj webUI.