/metrics not showing all metrics on new storjnodes under docker

Node Operators troubleshooting
1

Hello,

I’m adding new Storj storage nodes on my NAS after installing new HDDs. The nodes are running fine and show up normally in the Storj dashboard, but I’m not seeing the usual Prometheus metrics at /metrics for the new nodes (they’re less than a week old).

For all nodes (using macvlan network style - each node got its own IP), I set STORJ_DEBUG_ADDR: “node-IP:9501” in my Docker Compose, and I can access it through the http://node-IP:9501/metrics.

However, the output only shows a minimal set of metrics (mainly function and function_times). My older nodes (same Docker-based setup) expose the full metrics output as expected.

Is this limited /metrics output normal for new nodes, or could it indicate a config/version issue? Any suggestions on what to check?

Thank you!

2

Answering my own question (in my opinion, this looks like an updater/config conflict — I saw similar mentions in older posts):

I compared config.yaml from an older node (metrics OK) and a newer node (metrics missing). On the new node, the debug.addr line was set/uncommented, while on the old node it was commented out.

Fix:

  • I commented out debug.addr in the new node’s config.yaml
  • I kept STORJ_DEBUG_ADDR set in Docker Compose (STORJ_DEBUG_ADDR: "x.x.x.x:9501")

After that, the full metrics started showing up normally on /metrics.

Hope this helps anyone facing the same issue.

2 Likes
3

This is a little bit weird, because the env variable should override what’s set in the config.yaml file.

4

Hi Alexey,

Yes, I agree — it’s strange, especially since I didn’t manually change config.yaml during the setup of the new nodes.

For reference, I used the Docker Compose YAML below to create the new nodes. Please let me know if I am doing something wrong there.

The SETUP line is removed after the first run.

services:
  storjnodeX:
    image: storjlabs/storagenode:latest
    container_name: storjnode0X
    restart: unless-stopped
    stop_grace_period: 300s
    environment:
      WALLET: "0x000000000000000000000000000000000000000"
      EMAIL: "storjX@mydomain.com"
      ADDRESS: "storjnodeX.mydomain.com:28967"
      STORAGE: "XXTB"
      STORJ_DEBUG_ADDR: "x.x.x.X:9501"
      SETUP: "true"
    volumes:
      - "/mnt/NODE0X-ID:/app/identity"
      - "/mnt/NODE0X-CONFIG:/app/config"
      - "/mnt/NODE0X-DATA:/app/config/storage"
    networks:
      storj-network:
        ipv4_address: "x.x.x.X"

networks:
  storj-network:
    external: true

Thanks.

5

I recently encountered the same problem. Apparently, the configuration file (or the default setting) needs to be overridden to trigger full metrics. I’m not sure if this is intentional or a bug.

6

Then overriding in command: should have the priority, like

    command:
      - "--debug.addr=x.x.x.X:9501"
7

@Alexey,

I don’t think this is a default-value issue. All my nodes have the correct IP/port set in config.yaml under debug.addr.

The environment variable also seems to apply correctly (it matches what’s in debug.addr), but for some reason the config.yaml line isn’t being used—or it’s being overridden (that’s my guess).

After I comment out the debug.addr line in config.yaml (even though it shows the same IP/port as the env var) and restart the container, the metrics start working normally.

So I don’t think the problem is the IP/port configuration itself, but something else starting/binding the metrics exporter (or similar) and preventing the full metrics from being exposed.

8

If setting a different port in environment variable it will also work without comment out config.yaml. Maybe a bug? I don’t think it was always like this.

1 Like
9

Accordingly your observation, it’s reverse - the environment variable doesn’t have a higher priority above an option in config.yaml, so I agree:

The correct way to override (higher the number - the higher priority):

  1. config.yaml
  2. environment variable
  3. command line option

So the environment variables are expected to have a higher priority above options in the config.yaml file and the command line options have a highest priority above both.

10

Sure but why it doesn’t show full metrics if both are set to same port (e.g. 5999)?