Prometheus Storj-Exporter

[kevink]

try the name of the storagnode container.

[Marvin-Divo]

@Doom4535

#!/usr/bin/env bash
docker run -d --restart always --stop-timeout 300 \
    -p 28967:28967 \
    -p 14001:14002 \
    -p 127.0.0.1:6001:5999 \
    -e WALLET="<address_removed>" \
    -e EMAIL="<email_removed>" \
    -e ADDRESS="<url_removed>:28967" \
    -e STORAGE="1.6TB" \
    --mount type=bind,source="/srv/dev-disk-by-label-DataDrive01/StorjNode01/Identity",destination=/app/identity \
    --mount type=bind,source="/srv/dev-disk-by-label-DataDrive01/StorjNode01/Data",destination=/app/config \
    --name StorjNode-01 storjlabs/storagenode:latest \
    --debug.addr=":5999" \
    #--log.level error

Hello. What --debug.addr switch do? Can’t find any information about it. Thanks.

[Doom4535]

@Marvin-Divo, the debug flag sets the debug port to a specific value; it is my understanding that it otherwise selects a random port (not sure why it isn’t just disabled if it uses a random one). This port provides additional status information; unfortunately, you are right and it does seem to be missing official documentation, for more details see this post:

Guide to debug my storage node, uplink, s3 gateway, satellite

We have integrated a package called monkit in all storage nodes. It is very usefull for debuging. By default it will open a random port. In order to connect to the debug port you will need to: Add debug.addr: ":5999" to the config file. Now the port is fixed and will not change on every restart. Add -p 127.0.0.1:5999:5999 to the docker run command. Ready for action! You can request the following informations: curl localhost:5999/mon/funcs Find out how long a function like GetExpired need…

[dragonhogan]

@greener good afternoon mate, was recommended to you by kevinK on something odd with what I’m seeing on my "Storj-Exporter-Boom-Table.json.

We briefly discussed in this thread: How to monitor all nodes in your LAN using prometheus + grafana [linux using docker] - #45 by dragonhogan

Where the grafana dashboard, specifically the “node summary table” looks like this:

when it should look like this:

kevinK and I discussed the method of importing the .json text directly from the github repo into grafana web UI, and we couldn’t figure out why we’re getting such different results.

Let me know your thoughts.

[greener]

Hey @dragonhogan - I just replied in that thread too, try to install yesoreyeram-boomtable-panel plugin version 1.3.0
as there’s bugs in the later versions with newer Grafana

Boom Table 1.4.0 does not work with Grafana 7.1.5 (Windows)

[Bug] column order changed to alphabetically

Hope this helps

[greener]

I’m trying to pin a link to @kevink 's guide (How to monitor all nodes in your LAN using prometheus + grafana [linux using docker]) in my first message here and it looks like editing is locked for me now. I can edit my later posts but not the first one. Can anyone help me with unlocking it?

@kevink if you can give Storj-Exporter-dashboard/quick_start at dev · anclrii/Storj-Exporter-dashboard · GitHub a try and let me know if any issues. I don’t use docker-compose and not sure how much sense this makes Also there may be some things there that you can reference in your guide, it uses grafana provision feature to configure datasource and dashboard automatically.

[kevink]

I think you need @Alexey to make it a Wiki. (gonna need that in my thread too at some point…)

Thanks, I’ll take a look at it tomorrow!

[kevink]

I took a look at your quick_start and I like it. Certainly faster to get grafana and prometheus set up correctly. The section about setting up grafana and the dasboard is quite big in my how-to and this would cut it down to a few lines.

Just a few thoughts:

• Is it possible to remove the preconfigured password so the user can just log in with admin,admin and change it afterwards himself? Storing a password in plaintext in the .env is not really good
• many projects use an .env.example that you have to copy as .env so you can make changes in your repo to the .env.example if that is every going to be neccessary (just a thought)
• I would run both containers as user 1000:1000 instead of root
• Considering some answers in my thread, I’d now actually go for a predefined path in all settings, like /var/prometheus and /var/grafana. That way nobody can do something wrong, forget something or similar and can just follow the commands 1:1. If a user is advanced enough, he can change the paths of course.
• I guess the default prometheus parameters are good enough, in my guide I also mentioned retention times and sizes to keep the prometheus database within a chosen range
• Maybe you should add a note about not running prometheus on the sdcard on rpis? (in which case you would need a bind mount instead)
• In the prometheus.yml I’d add the option to label the nodes properly like:

    static_configs:
      - targets: ["storj-exporter6:9651"]
        labels:
          instance: "node6"

(btw: Didn’t realize I could just pack all exporters into one job… guess that is a lot better than defining 6 separate jobs)

[greener]

All good points, there’s more to discuss, I want to keep it as simple as possible and just be an example so people can extend it as they need.

1. Is it possible to remove the preconfigured password so the user can just log in with admin,admin and change it afterwards himself? Storing a password in plaintext in the .env is not really good

I agree the password should be changed and not stored in .env, although I’m not sure what the usual workflow is with docker-compose. Would you permanently save and reuse it to update confs or just use it once? What happens if you update password in UI and not in .env, would it revert the default password? In my case I use ansible to configure these things and use ansible-vault to store such variables encrypted. I guess it’s up to the user to manage this and I can add a note in the security considerations section, also set default to admin:admin.

2. many projects use an .env.example that you have to copy as .env so you can make changes in your repo to the .env.example if that is every going to be neccessary (just a thought)

Yep sounds good

3. I would run both containers as user 1000:1000 instead of root

Yep, need to look into it. I was reading Run Grafana Docker image | Grafana documentation seems just need to set user 472 and confirm group directory permissions.

4. Considering some answers in my thread, I’d now actually go for a predefined path in all settings, like /var/prometheus and /var/grafana. That way nobody can do something wrong, forget something or similar and can just follow the commands 1:1. If a user is advanced enough, he can change the paths of course.

I’m not sure. This is probably the most sensitive part where it would be good if everyone understands what they are doing because there’s a risk of filling up the filesystem 100% and halting the whole system with painful downtime if set up on the same machine as running storagenode and /var is on a root volume. A good default option is to use volumes instead of bind-mounts and docker-compose will create/store them in /var/lib/docker/volumes/ (Manage data in Docker | Docker Docs), but I specifically made this paths mandatory to edit because of risks.

5. I guess the default prometheus parameters are good enough, in my guide I also mentioned retention times and sizes to keep the prometheus database within a chosen range

It would be good to add retention defaults set to at least 35 days I agree.

6. Maybe you should add a note about not running prometheus on the sdcard on rpis? (in which case you would need a bind mount instead)

Sure. I’m not using rpis, would it kill IO on sd?

7. In the prometheus.yml I’d add the option to label the nodes properly like:

We can, but it’s a custom thing really. I use meaningful hostnames in targets and so I don’t label instance at all.

I’ll be adding some of these changes in for sure, thanks for your feedback and let me know your thoughts on the above.

(btw: Didn’t realize I could just pack all exporters into one job… guess that is a lot better than defining 6 separate jobs)

Yeah, it’s quite flexible - you can do multiple groups of targets with shared labels for each like:

    static_configs:
      - targets:
        - storj-exporter1:9651
        - storj-exporter2:9652
        labels:
          environment: "dev"
      - targets:
        - storj-exporter3:9651
        - storj-exporter4:9652
        labels:
          environment: "prod"
      ...

and you can have one target per group for unique labels per target:

    static_configs:
      - targets:
        - storj-exporter1:9651
        labels:
          instance: "node1"
      - targets:
        - storj-exporter2:9651
        labels:
          instance: "node2"
      ...

also dynamic relabel is really useful for larger inventories as well as autodiscovery Configuration | Prometheus

[kevink]

@greener

Would you permanently save and reuse it to update confs or just use it once?

If you want to update the grafana container, the easiest way is to use docker-compose pull and then docker-compose up -d again so I’d definitely keep the whole directory.
So if not setting a password in docker-compose.yml doesn’t create problems with the remaining auto-setup, I’d actually prefer that as it doesn’t result in some password laying around in plaintext (or look into more advanced stuff like ansible).

@greener

Yep, need to look into it. I was reading Run Grafana Docker image | Grafana documentation seems just need to set user 472 and confirm group directory permissions.

Isn’t that just for migration? For a new installation you can choose whatever id you want. Personally I just prefer 1000:1000 so I can access everything with my main user account. Might not be good in all situations of course…

@greener

I’m not sure. This is probably the most sensitive part where it would be good if everyone understands what they are doing because there’s a risk of filling up the filesystem 100% and halting the whole system with painful downtime if set up on the same machine as running storagenode and /var is on a root volume

Yeah it’s a good point. And it’s easier to understand in your guide which path you have to change.

@greener

Sure. I’m not using rpis, would it kill IO on sd?

Depends on your SD card. If you just set up a system that does absolutely nothing except running a storagenode and you use a small card, then it might kill the card soon, especially if you have more nodes or add netdata stats too.

@greener

We can, but it’s a custom thing really. I use meaningful hostnames in targets and so I don’t label instance at all.

ok, well I have multiple nodes on the same machine and I don’t want them showing up as hostname:port in the dashboard, I prefer setting node names.

@greener

Yeah, it’s quite flexible - you can do multiple groups of targets with shared labels for each like:

That looks good, I should try that

[Pac]

Would it be possible to run Prometheus on the RPi 4B running nodes, and make it use only a certain amount of RAM like 1GB for instance (hence no writes on the SD card) if one wanted to have statistics on a short history only?
Just asking. The best would be to dedicate a bit of storage somewhere, on a CMR drive obviously…

One way would be a RAMdisk I guess. But data would be lost when the machine reboots… I feel like it’s not a good idea but, I’m just wondering

[kevink]

@Pac

and make it use only a certain amount of RAM like 1GB for instance (hence no writes on the SD card)

@Pac

One way would be a RAMdisk I guess

I don’t see why not… Would start over every time you reboot but other than that…

@Pac

The best would be to dedicate a bit of storage somewhere, on a CMR drive obviously…

If you run your node on a CMR, prometheus would be fine on that drive too.

[Puffball]

This is probably a dumb question, but how do I change the order of the columns in the Boom Tables? They seem to be organised by the column name in alphabetical order. For example I want to move the Suspension column so that it is next to the Disqualification column.

[greener]

Try to install yesoreyeram-boomtable-panel plugin version 1.3.0 as there’s bugs in the later versions. I’m not sure if sorting is also a bug but it changed to alphabetical in 1.4.0 and that’s not too convenient. In 1.3.0 it is sorted in order of metric queries.

[SirRichardTheFirst]

what sourcery this¿ prometheus really looks amazing… a video on that would be great !

[greener]

Just updated master and latest image to include handling SIGTERM. At least in my case docker took long time to stop exporter container without this. I was running this update on my node for a while and not seen any issues so far.

Also looking at A couple changes to SNO API I made some changes in dev https://github.com/anclrii/Storj-Exporter/compare/dev, not sure if I understand this correctly though. If anyone was checking the same - let me know if this is not what changed or raise a PR.
Either way I think I will have to wait for this changes to be released so I have something to test against. This will also break parts of grafana dashboard related to audit but will be a quick fix. Will likely loose some info on the dashboard as well, like audit Vetting column.

[greener]

I’ve updated repo build action to build multi-architecture docker images for each new version. This should allow fetching arm images from docker-hub for Raspberry Pi and arm-based hardware. I don’t have anywhere to test it though.

Can somebody try anclrii/storj-exporter:1.0.6-beta.3 image on a Raspberry Pi and feedback the results please? You don’t need to specify architecture anywhere as docker will pull the one that matches the system.

[kevink]

Awesome! I’ll probably steal your work for my exporter too

I tested it on my RPI (which doesn’t run a node though…) and it seems to work just fine.

[Arkina]

@kevink, @greener Just as an idea, maybe you want to integrate each others exporter into one ? So there is no need to have 2 exporters running per node. Maybe same can be done with the Dashboards?

AFAIK you can give additional permissions on github per repository for other github users, so you can work on one repo, without awaiting PRs and so on.

I already tough about todo it myself but then there would be an additional repo which wouldn’t make sense at all ^^ If it helps i would help on this task, i plan to use both exporters anyway.

[greener]

Would be great to do this and was thinking the same. Unfortunately I don’t see an easy way of doing this. @kevink log exporter is a go binary exporter and has to listen on a separate port even if I add it to the same docker image. No way to integrate it into my python code.

I was thinking maybe adding a collector that would implement similar log parsing functionality in python and expose the same metrics, but this will take a while for me to do. If anybody wants to propose a PR that would be great.

Another thing to consider is log path. It would be great to somehow feed logs from container directly.