Docker Compose ERROR

web4yougmbhch · September 25, 2020, 9:30am

I just started with docker compose to start my SN. I am using storjnode, watchtower, storj-exporter and storj-prometheus.

So i have to manage this to start by the same order as listed above.

I created the docker-compose.yml as follows:

#docker_compose.yml
version: "3.3"
services:
  storagenode:
image: storjlabs/storagenode:latest
volumes:
  - type: bind
source: "/home/juerg/.local/share/storj/identity/storagenode"
target: /app/identity
  - type: bind
source: "/DriveShare/StorjNode"
target: /app/config
ports:
  - 28967:28967
  - 14002:14002
deploy:
  restart_policy:
  condition: unless-stopped
environment:
  - WALLET=0xxxxxxxxxxxxxxxxxxxxxxx
  - EMAIL=my-email@here.ch
  - ADDRESS=my-address:28967
  - STORAGE=3200GB
watchtower:
  image: storjlabs/watchtower
  restart: always
  container_name: watchtower
  command: --interval 21600 --cleanup true --stop-timeout 300s --cleanup
  volumes:
    - /var/run/docker.sock:/var/run/docker.sock
  storj-exporter:
  image: anclrii/storj-exporter:latest
  restart: unless-stopped
  container_name: storj-exporter
  links: 
    - storagenode
  ports:
    - 9651:9651
storj-prometheus:
  image: prom/prometheus
  restart: unless-stopped
  container_name: storj-prometheus
  links:
    - storj-exporter
  ports:
   - 9090:9090
  volumes:
    - /srv/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
    - storj-prometheus:/prometheus

On startup i get the following error:
ERROR: Named volume “storj-prometheus:/prometheus:rw” is used in service “storj-prometheus” but no declaration was found in the volumes section.

And here is my original prometheus run command:
docker run -d -p 9090:9090 --name=storj-prometheus --link=storj-exporter -v /srv/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml -v storj-prometheus:/prometheus prom/prometheus

Any hints?

kevink · September 25, 2020, 9:49am

you have to creat the volume “storj-prometheus”?

web4yougmbhch · September 25, 2020, 11:09am

Yes i have to create it. If not i dont have any data on my grafana dashboard.

My shell run command is working well but not in docker-compose

web4yougmbhch · September 25, 2020, 11:23am

In the meantime I’ve changed the following:

#docker-compose-prometheus.yml
version: "3.3"
services:
  storj-prometheus:
    image: prom/prometheus
    restart: unless-stopped
    container_name: storj-prometheus
    external_links:
      - storj-exporter
    ports:
      - 9090:9090
    volumes:
      - type: bind
        source: "/srv/prometheus/prometheus.yml"
        target: /etc/prometheus/prometheus.yml
      - type: bind
        source: "/storj-prometheus"
        target: /prometheus

I m not shure about this external_link. I try now only this docker-compose-prometheus.yml couse i dont like to stop my storjnode every time i test this.

BUT i get now this error:

web4yougmbhch · September 25, 2020, 11:27am

Image could not be loaded so here:

juerg@storj-debian:~/docker-compose$ docker-compose -f docker-compose-prometheus.yml up
Creating network “docker-compose_default” with the default driver
Pulling storj-prometheus (prom/prometheus:)…
latest: Pulling from prom/prometheus
76df9210b28c: Pull complete
559be8e06c14: Pull complete
6a4bb3319487: Pull complete
2cca90a64593: Pull complete
d2014e464a99: Pull complete
70b42590e4a2: Pull complete
54645fcbd6cc: Pull complete
67d9943de656: Pull complete
b9c749b1af90: Pull complete
9723d8eb5323: Pull complete
7d20502d5322: Pull complete
3e519cce6f63: Pull complete
Digest: sha256:d43417c260e516508eed1f1d59c10c49d96bbea93eafb4955b0df3aea5908971
Status: Downloaded newer image for prom/prometheus:latest
Creating storj-prometheus … done
Attaching to storj-prometheus
storj-prometheus | level=info ts=2020-09-25T11:22:56.834Z caller=main.go:310 msg=“No time or size retention was set so using the default time retention” duration=15d
storj-prometheus | level=info ts=2020-09-25T11:22:56.835Z caller=main.go:346 msg=“Starting Prometheus” version=“(version=2.21.0, branch=HEAD, revision=e83ef207b6c2398919b69cd87d2693cfc2fb4127)”
storj-prometheus | level=info ts=2020-09-25T11:22:56.835Z caller=main.go:347 build_context=“(go=go1.15.2, user=root@a4d9bea8479e, date=20200911-11:35:02)”
storj-prometheus | level=info ts=2020-09-25T11:22:56.835Z caller=main.go:348 host_details=“(Linux 4.19.0-8-amd64 #1 SMP Debian 4.19.98-1 (2020-01-26) x86_64 cea589eed6df (none))”
storj-prometheus | level=info ts=2020-09-25T11:22:56.835Z caller=main.go:349 fd_limits=“(soft=1048576, hard=1048576)”
storj-prometheus | level=info ts=2020-09-25T11:22:56.835Z caller=main.go:350 vm_limits=“(soft=unlimited, hard=unlimited)”
storj-prometheus | level=error ts=2020-09-25T11:22:56.835Z caller=query_logger.go:87 component=activeQueryTracker msg=“Error opening query log file” file=/prometheus/queries.active err=“open /prometheus/queries.active: permission denied”
storj-prometheus | panic: Unable to create mmap-ed active query log
storj-prometheus |
storj-prometheus | goroutine 1 [running]:
storj-prometheus | github.com/prometheus/prometheus/promql.NewActiveQueryTracker(0x7ffdd423ff02, 0xb, 0x14, 0x30ca080, 0xc0005f6b70, 0x30ca080)
storj-prometheus | /app/promql/query_logger.go:117 +0x4cf
storj-prometheus | main.main()
storj-prometheus | /app/cmd/prometheus/main.go:377 +0x510c

Alexey · September 25, 2020, 7:43pm

Do not use a links, use a network instead: Overview | Docker Docs or do not use any and use the depends_on instead: Overview | Docker Docs
Also, It complains on missed common volumes section: Overview | Docker Docs
See the volumes: section at the end of the docker-compose.yaml

volumes:
 mydata:
 dbdata:

Also, the deploy clause is applicable only if you use a docker swarm. Otherwise you should use a casual restart: unless-stopped

You can do not stop the storagenode, just after the change use a docker-compose up -d, it will re-create only modified services.

giostino · October 25, 2020, 1:07pm

mind to share a working docker-compose file? Since i´m running storj also in docker-compose and i´m interested in storj-exporter as well

fmoledina · October 28, 2020, 3:32pm

Here’s my working example for Storj and Storj-Exporter using docker-compose.

docker-compose.yml:

version: "3.6"
services:
  storagenode:
    container_name: storagenode
    hostname: storagenode
    image: storjlabs/storagenode:latest
    # restart: always
    ports:
      - "28967:28967"
      - "14002:14002"
    volumes:
      - type: bind
        source: /path/to/identity/storagenode
        target: /app/identity
      - type: bind
        source: /path/to/data/storagenode
        target: /app/config
      # - type: bind
      #   source: /path/to/dbs/storagenode  # only if using dedicated dbs directory separate from /app/config inside container
      #   target: /app/dbs
    env_file:
      - storj.env
    networks:
      - prodnet
    stop_grace_period: 300s

  watchtower:
    container_name: watchtower
    restart: always
    image: storjlabs/watchtower
    networks:
      - prodnet
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    command: storagenode watchtower --stop-timeout 300s # add storj-exporter, prometheus, and grafana to the list if desired

  storj-exporter:
    container_name: storj-exporter
    image: anclrii/storj-exporter:latest
    # ports: # not required if storj-exporter and prometheus are on the same Docker bridge network as in this example
    #   - "9651:9651"
    # build:
    #   context: https://github.com/fmoledina/Storj-Exporter.git  # optional
    environment:
      - STORJ_HOST_ADDRESS=storagenode
      - STORJ_API_PORT=14002
    networks:
      - prodnet
    restart: always

  prometheus:
    container_name: prometheus
    image: prom/prometheus
    ports: # port forward to container not required if using reverse proxy
      - "9090:9090"
    volumes:
      - "/mnt/zfs/appdata/prometheus/etc-prometheus:/etc/prometheus"
      - "/mnt/zfs/appdata/prometheus/prometheus:/prometheus"
    command:
      - "--config.file=/etc/prometheus/prometheus.yml"
      - "--storage.tsdb.path=/prometheus"
      - "--web.console.libraries=/usr/share/prometheus/console_libraries"
      - "--web.console.templates=/usr/share/prometheus/consoles"
    networks:
      - prodnet
    restart: always
    user: "99"
    # labels:  # optional, if using Traefik 1.7.x reverse proxy
    #   - "traefik.enable=true"
    #   - "traefik.backend=prometheus"
    #   - "traefik.frontend.rule=Host:prometheus.example.com"
    #   - "traefik.frontend.whiteList.sourceRange=192.168.0.0/16"
    #   - "traefik.frontend.whiteList.useXForwardedFor=true"
    #   - "traefik.port=9090"
    #   - "traefik.docker.network=prodnet"
    #   - "traefik.frontend.headers.SSLRedirect=true"
    #   - "traefik.frontend.headers.STSSeconds=315360000"
    #   - "traefik.frontend.headers.browserXSSFilter=true"
    #   - "traefik.frontend.headers.contentTypeNosniff=true"
    #   - "traefik.frontend.headers.forceSTSHeader=true"
    #   - "traefik.frontend.headers.SSLHost=example.com"
    #   - "traefik.frontend.headers.STSIncludeSubdomains=true"
    #   - "traefik.frontend.headers.STSPreload=true"
    #   - "traefik.frontend.headers.frameDeny=true"

  grafana:
    container_name: grafana
    image: grafana/grafana
    restart: always
    ports: # port forward to container not required if using reverse proxy
      - "3000:3000"
    volumes:
      - "/mnt/zfs/appdata/grafana/var-lib-grafana:/var/lib/grafana"
      - "/mnt/zfs/appdata/grafana/etc-grafana-provisioning:/etc/grafana/provisioning"
    env_file:
      - grafana.env
    networks:
      - prodnet
    user: "99"
    depends_on:
      - prometheus
    # labels:  # optional, if using Traefik 1.7.x reverse proxy
    #   - "traefik.enable=true"
    #   - "traefik.backend=grafana"
    #   - "traefik.frontend.rule=Host:grafana.example.com"
    #   - "traefik.frontend.whiteList.sourceRange=192.168.0.0/16"
    #   - "traefik.frontend.whiteList.useXForwardedFor=true"
    #   - "traefik.port=3000"
    #   - "traefik.docker.network=prodnet"
    #   - "traefik.frontend.headers.SSLRedirect=true"
    #   - "traefik.frontend.headers.STSSeconds=315360000"
    #   - "traefik.frontend.headers.browserXSSFilter=true"
    #   - "traefik.frontend.headers.contentTypeNosniff=true"
    #   - "traefik.frontend.headers.forceSTSHeader=true"
    #   - "traefik.frontend.headers.SSLHost=example.com"
    #   - "traefik.frontend.headers.STSIncludeSubdomains=true"
    #   - "traefik.frontend.headers.STSPreload=true"
    #   - "traefik.frontend.headers.frameDeny=true"

networks:
  prodnet:
    # external: true # optional, only if you create the Docker bridge network 'prodnet' separately

storj.env used in above docker-compose.yml:

WALLET=< wallet address >
EMAIL=< email address >
ADDRESS=storj.example.com:28967
STORAGE=8TB

grafana.env used in above docker-compose.yml:

GF_SERVER_DOMAIN=grafana.example.com
GF_SERVER_ROOT_URL=https://grafana.example.com
GF_METRICS_ENABLED=true
GF_ANALYTICS_REPORTING_ENABLED=false
GF_USERS_ALLOW_SIGN_UP=false
GF_INSTALL_PLUGINS=yesoreyeram-boomtable-panel

Snippet from prometheus.yml config file for Storj-Exporter:

...
scrape_configs:
  ...
  - job_name: 'storj-exporter'
    static_configs:
      - targets: ['storj-exporter:9651']
        labels:
          instance: 'storj'

jeremyfritzen · July 5, 2021, 1:49am

Hi!
Sorry for digging up a such old thread…

@fmoledina thank you for sharing your docker-compose, it helps a lot
Just a question about your watchtower configuration:

  watchtower:
    container_name: watchtower
    restart: always
    image: storjlabs/watchtower
    networks:
      - prodnet
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    command: storagenode watchtower

=> Why is the volume " - /var/run/docker.sock:/var/run/docker.sock" for?

Thanks a lot!

fmoledina · July 5, 2021, 3:55am

Good question! This is so that watchtower can control the Docker daemon on the host to do things like control actions for other containers. Since watchtower itself is running in a container, by default it doesn’t have access to do any actions at the host level.

In the above docker-compose stack, watchtower is used to pull a new image for storjlabs/storagenode:latest when available, stop the running storagenode container, and start it again. All of these functions require control of the Docker daemon on the host. Using the docker.sock socket is how watchtower achieves this.

Below is an excerpt from my watchtower log that shows this process:

watchtower        | 2021-06-10T10:02:16.753283893Z time="2021-06-10T10:02:16Z" level=info msg="Found new storjlabs/storagenode:latest image (2983dc49145a)"
watchtower        | 2021-06-10T10:03:18.931718959Z time="2021-06-10T10:03:18Z" level=info msg="Stopping /storj_storj03_1 (f8d023f65d19) with SIGTERM"
watchtower        | 2021-06-10T10:04:38.629204389Z time="2021-06-10T10:04:38Z" level=info msg="Creating /storj_storj03_1"

There are some security concerns with exposing the entire Docker daemon to specific containers like watchtower and there are some ways out there to harden this connection by using things like fluencelabs/docker-socket-proxy. I’ve decided that for home use, exposing the Docker socket is okay with me.

Hope this helps!

jeremyfritzen · July 5, 2021, 6:13pm

Thanks for this great explanation!
Really interesting, I’m sure this will help me in other projects/ contexts