I’m using the Prometheus/Grafana Storj-Exporter and there is one thing I would like to do and can’t get it to work:
To the node summary table I want to add a columns that displays the node age (could be months or date, does not matter).
Determining the age would be when the node joined its first satellite.
We do have this date as “nodeJoinedAt” in the storj_sat_nodeJoinedAt_info metric and I am able to get ‘some’ date into an additional columne, but comparing it with the json from the API call, the date is not correct.
What gave me some result was to use topk(1, storj_sat_nodeJoinedAt_info{nodeJoinedAt!="", instance=~"$node.*"}) by (instance) as PromQL together with {{instance}} | nodeJoinedAt | {{nodeJoinedAt}} as legend. Similar to the ‘version’ column. So this adds me a column and displays a date, but the date is not the correct one.
It would be very helpful to have the node age in the table, so hopefully someone with better knowledge could tell me how to achieve what I want?
I am not very familiar with Storj-Exporter, but based on what I see in the source code it should work. (Except that I don’t understand where the date is converted).
Can you please share more details, how is the date “not correct”?
What I would do to debug this problem:
Check the real expected date using the same REST API which is used by the exporter. This can be done with curl localhost:14002/api/sno/satellite/12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB on your node.
Checking the exporter output with curl localhost:9651.
IMHO the second should contain something in epoch not in human readable format.
Thanks for trying to help.
I am just at the beginning of trying to understand the Prometheus/Grafana thing and it seems to be complex.
Regarding wrong date: When I open the nodes API for a specific satellite, I see joinedAt: 2021-06-26T07:21:58.757083Z.
When I use my code as PromQL I get returned as date for this node: 2019-10-22T07:51:54.414574Z which is completely false as this node is not that old. So for some reason that I don’t understand and can’t handle, I don’t get the correct joinedAt dates per node but accidental ones from other nodes.
I think what I want can be done, as all the data is there, I just don’t know how to put it together.
Well, my setup is a bit amended.
I am running one exporter that pulls the required data from the nodes.
That is one static_config with multiple targets indeed.
What is not clear how the instance= tag is filled.
It can be done by the exporter, but I don’t see this in the code
It can be done by the prometheus per source, but as far as I understood, you use one source with multiple targets.
My best guess so far is that data from multiple source uses the same tags. But it’s hard to prove without knowing the configuration exactly. I would suggest checking the raw output with curl commands above.
I can’t see how the required logic can be implemented in Grafana to query only this specific value.
I believe it has to be done in the Exporter, maybe even by creating a new metric that queries only for the joined date and passes it to Grafana.
Prometheus can only store strings in labels but I don’t think prometheus functions can be applied to labels so it’s currently not possible to compute “oldest” value out of multiple sat labels. I did not bother converting this to something like epoch date because I did not expect it will be used at all. It is doable though and once this is exported as a metric value it should be possible to just use min function to get the oldest value.
What I did for now is:
There is an API endpoint in sno/satellites : earliestJoinedAt that seems to hold exactly what I wanted. I created a new metric in Exporter and use this in Grafana. It seems to work.
But you are right it seems to be a string only. I have not yet found out how to make it a time value or convert it into the number of months. This would be required to sort the table by that value.