Prometheus Storj-Exporter

Krawi · August 14, 2021, 11:51am

It gives the whole thing a terrible performance.
In a worst case scenario your node could be blacklisted by the DNS server. (DDOS attack!)

idfxken · August 14, 2021, 11:56am

Wasn’t planning on doing that, to be clear. But more in the lines of something like this:

#!/bin/bash
HOSTNAME=dynamichost.domain.com
LOGFILE=$HOME/ufw.log
Current_IP=$(host $HOSTNAME | head -n1 | cut -f4 -d ' ')

if [ ! -f $LOGFILE ]; then
    /usr/sbin/ufw allow from $Current_IP to any port 22 proto tcp
    echo $Current_IP > $LOGFILE
else

    Old_IP=$(cat $LOGFILE)
    if [ "$Current_IP" = "$Old_IP" ] ; then
        echo IP address has not changed
    else
        /usr/sbin/ufw delete allow from $Old_IP to any port 9651 proto tcp
        /usr/sbin/ufw allow from $Current_IP to any port 9651 proto tcp
        echo $Current_IP > $LOGFILE
        echo iptables have been updated
    fi
fi

with crontab
Thank you for the firewall rules inspiration!

kevink · August 14, 2021, 12:16pm

don’t expose the port when running it with docker and use an ssh tunnel to get into the server?

Krawi · August 14, 2021, 1:57pm

Using something like this?:
https://github.com/discordianfish/prometheus-tunnel

For the persistent SSH tunnel:
https://medium.com/@gary4est/autossh-port-forwarding-23088d948787

On the Prometheus server you need the typical SSH setup with root disabled, no plaintext passwords, Private/Public Keys for authentication, Fail2Ban installed, if possible changed port.

Fail2Ban with increasing ban times:
https://visei.com/2020/05/incremental-banning-with-fail2ban/

Nice idea for a (long?) weekend

idfxken · August 14, 2021, 6:28pm

Thats quite a lot of moving parts. (more stuff that can break)
Also don’t like public sshd open on the prom server (defeats vpn)
I care (a bit) less about the nodes

For now I’ll go with the ufw + crontab script, my train of thoughts:

Only ‘specific ip’ can access 9651
Hostiles have no intel on 9651/ it seems closed (iptables drops the packets)
Hostiles have to know the correct ip + spoof it to gain very little info/attack surface

For me this seems reasonably secure, fast and simple to deploy.

PS:
Fail2ban, forced public key auth(exempt specific ip/subnet), rootlogin disabled.
Spread the holy word!

greener · June 13, 2022, 9:39am

I released version 2.1.0 of the exporter recently where some old metrics were deprecated in favour of labels for the same data.

Storj-Exporter-dashboard/Storj-Exporter-Boom-Table.json at 527f9946e625dad9ba0864e90b5a05a2f296145b · anclrii/Storj-Exporter-dashboard · GitHub is updated accordingly, please import the latest version of the dashboard.

Some issues were reported with v2 and other dashboards here 2.0.0 - No Data · Issue #64 · anclrii/Storj-Exporter · GitHub so those may need updating too as they may still be using the deprecated metrics.

greener · June 24, 2022, 2:49pm

Version 2.1.2
Adding currentMonthExpectations metric

Also added a column for payout expectations in grafana dashboard GitHub - anclrii/Storj-Exporter-dashboard: Storj-exporter Grafana dashboard

Chris21788 · July 5, 2022, 6:14pm

So I had the below written up, but found that the estimate query just didn’t refresh with all of the data, so I’m going to leave this here if anyone needs the query.

Took a look at your new Expectations addition to the dashboard and I know it’s a Storj API endpoint. I currently use the
following:((sum(storj_payout_currentMonth{type=~"payout|held",instance=~"$node.*"})/100)/((day_of_month()-1)+(hour()/24)))*days_in_month()

This will essentially take the current payout so far this month, divide it by the current progress into the month (as a decimal - to get the average daily payment), then multiply it by the number of days within the month.

Mine starts off inaccurate, but gets more accurate as the month goes on…something I think that’s ok.

It’s only $0.25 off, so I’m ok with that, but I’ll see who’s more accurate as we near the end of the month.

Thanks again @greener for the updates.

LrrrAc · July 5, 2022, 6:58pm

I graphed the difference between your formula (I used something very similar before this latest metric addition, thanks greener for adding that!) and the new metric, and after adding seconds to get it a bit more granular I got this:

Pretty close either way.

niu12503 · November 11, 2022, 3:03pm

ti’s work good ,thx ,thx

agente · November 14, 2022, 1:00pm

I did a fresh install and I have the signature problem with boom table plugin. In .env I have GF_USERS_ALLOW_SIGN_UP=false but seems not working

ps: docker installation

maxsch · November 14, 2022, 5:55pm

Pls try following
GF_PLUGINS_ALLOW_LOADING_UNSIGNED_PLUGINS=yesoreyeram-boomtable-panel
GF_INSTALL_PLUGINS=yesoreyeram-boomtable-panel 1.3.0,grafana-clock-panel,grafana-simple-json-datasource

agente · November 14, 2022, 9:24pm

I need to delete some file or just docker compose down and up?

agente · November 18, 2022, 10:57am

I reinstall everything changing .env with
GF_PLUGINS_ALLOW_LOADING_UNSIGNED_PLUGINS=yesoreyeram-boomtable-panel
GF_INSTALL_PLUGINS=yesoreyeram-boomtable-panel 1.3.0,grafana-clock-panel,grafana-simple-json-datasource

I have the same problem. Anyone have solved the problem with new grafana?

lyoth · November 18, 2022, 9:14pm

you need to run this command to install it

grafana-cli plugins install yesoreyeram-boomtable-panel

agente · November 18, 2022, 10:23pm

I have signature problem. Plugin installed but not enabled for missing signature

Tomaae · December 6, 2022, 2:18am

I made a compose deployment which includes everything in simple to mass deploy format.

storj-node with log to file
preconfigured zksync
logrotate for storj-node
storj exporter
storj log exporter
storj updater
running on shared dedicated internal network

storj-deployment

I used it to deploy 8 nodes + grafana.
All steps are compiled together on github for simplicity and fast deployment.

Feel free to use it or modify it in any way you like.

kaloyan · December 6, 2022, 10:00am

@Tomaae This is fantastic! I have been using GitHub - anclrii/Storj-Exporter-dashboard: Storj-exporter Grafana dashboard, and I will be looking to upgrade my SNO setup to your solution.

Have you considered adding the Prometheus Node Exporter into the mix to monitor OS-level metrics like CPU usage?

I am pleased to see how the SNO community self-organized to create this multi-node monitoring solution based on Prometheus and Grafana. Is there anything that we from Storj Labs can help with? For example:

Are there any new API endpoints that we should add to the storage node?
Will it help for visibility if we add a setup guide to the official documentation at Storj Node Operator - Storj Node Operator Docs? Any volunteers to write it up?
Will it help if we add a Github repo under Storj - Third Party · GitHub to facilitate collaboration among the community?

Anything else we can help with?

Tomaae · December 6, 2022, 10:31am

Its the same setup, just bit modified so everything is put together.

I have not really considered OS level metrics, since my underplaying OS has them all already.
But adding them should not be a problem and could help others, I will have a look at that.

I would have lot to say regarding official docs, but dont want to derail this thread. So feel free to contact me in different way if you are interested in my feedback. Not sure if there are DM’s here.

Tomaae · December 6, 2022, 12:34pm

Implemented. I will add screenshot in few hours after there are some data