Metrics: Reliably Configuring Prometheus and Grafana with Docker.

Tech Stack: Prometheus, Grafana, Docker, Docker-compose.

There is a lot of great guides on how to integrate metrics into your application so you can use them with Prometheus, for example there is an excellent Java Client and first level support from popular web frameworks such as Spring. However, I struggled to find information on how these metrics are collected, and how you would go about versioning and deploying dashboards reliably once you had them setup.

Therefore, this guide will take you through an overview of how Prometheus collects metrics, before moving on to visualizing them with Grafana. As we get things running, we will focus on how we store and version this config, so it can be deployed reliably with Docker.

Step 1: Understanding how metrics are collected

This guide will be running an application exposing metrics on host and port: http://yak-server:9001/metrics.

Given you have your application running, and you have integrated some metrics into it, the first thing to understand is how Prometheus accesses that information.

When you run Prometheus, you configure it to “scrape” certain destinations. What this means is that Prometheus, say every 15 seconds, will call your metrics endpoint and then store the result; this is known as a pull-based architecture (Prometheus is pulling the metrics to it). Therefore, your application just needs to provide a location to get metrics from, and Prometheus will do the rest (Figure 1).

Figure 1: Prom queries your app every 15 seconds to pull in new metrics.

Once Prometheus gathers your metrics, it will store them in a time series database, making them available for querying. In order to query them, Prometheus provides a powerful query language that you can use via the API or in the Prometheus UI. These queries are what you then configure in Grafana to display graphs and information in dashboards (Figure 2).

Figure 2: Grafana queries PromQL based on what you setup in your dashboard.

So with this architecture, in order to reliably deploy it, we need to:

  • Configure the Prometheus scrape config, to tell it where our application is located and how often to get metrics from it.
  • Store the metrics in a persisted volume, so we don’t lose them if the Prometheus Docker image is stopped.
  • Configure Grafana to find the Prometheus instance, so we can visualise the metrics we are collecting.
  • Configure a Grafana dashboard with the queries/graphs we want to display.
  • Store these dashboards so we can deploy them again if the Grafana Docker image is stopped.

Step 2: Deploying the metrics pipeline with Docker Compose

The first step to getting a reliable deployment of this architecture, is to simply get the images we need running locally. Create the following folder structure:

your-folder-location/
┣ grafana/
┃ ┣ dashboards/
┃ ┃ ┗ my-dashboard.json
┃ ┗ provisioning/
┃   ┣ dashboards/
┃ ┃ ┃ ┗ all.yml
┃   ┣ datasources/
┃ ┃ ┃ ┗ all.yml
┃   ┗ config.ini
┣ prom/
┃ ┣ prometheus.yml
┃ ┗ rules.yml
┗ docker-compose.yml

This structure contains everything we will need in this article, and we will go through each file as we incorporate it into our deployment. If you would like to skip this article, and just go to an example deployment, visit here.

Open the docker-compose.yml and copy the starting configuration:

version: '3.7'

services:

  yak-server:
    image: guardiandevelopment/yak-server:1.0.0.SNAPSHOT
    ports:
      - 9000:9000
      - 9001:9001
    deploy:
      resources:
        limits:
          cpus: '1'
          memory: 250M

  prom:
    image: prom/prometheus:v2.27.1
    ports:
      - 9090:9090

  grafana:
    image: grafana/grafana:7.5.7
    ports:
      - 3000:3000

volumes:
# store prom metrics between runs
prometheus_data: {}

networks:
  default:
    name: metrics-network

With this stored, you should be able to run the following command:

docker-compose up

and then access the following links:

  • Prometheus: http://localhost:9090/graph
  • Grafana: http://localhost:3000/ The login by default is admin:admin. Once you have logged in, just skip setting up another user as that is beyond the scope of this guide.
  • The metrics from the app: http://localhost:9001/metrics This will be what we tell Prometheus to scrape, but it gives you an idea of the format and type of metrics available to us in this guide.

Now we have done this, we will go through incorporating each config file to build our architecture from Step 1. For now, you can run the following command to stop everything:

# turn off all images and volumes, remove the -v 
# if you want volumes persisted between between runs
docker-compose down -v

Configuring Prometheus scrape targets

In order for Prometheus to gather metrics from our application, we need to configure its scrape targets. This will tell Prometheus how frequently to gather metrics, and where to find the endpoint for our application. Add the following config to ./prom/prometheus.yml:

global:
  scrape_interval: 15s
  evaluation_interval: 15s
  external_labels:
    monitor: 'yak-monitor'

# extra: include rules for alerting
rule_files:
  - rules.yml

# IMPORTANT BIT HERE: targets for scraping metrics from
scrape_configs:

  - job_name: 'prometheus'
    scrape_interval: 5s
    static_configs:
      - targets: [ 'prom:9090' ]

  - job_name: 'yak-server'
    scrape_interval: 5s
    static_configs:
      - targets: [ 'yak-server:9001' ]

This important part of this config is the scrape_configs section. This configures 2 targets:

  1. It scrapes itself for metrics. This is pretty standard and allows you to see metrics on the performance and health of your Prometheus instance.
  2. We setup yak-server:9001, which scrapes our application. As this is running in docker-compose, the yak-server DNS entry will be available to Prometheus. If you have ran your own app, make sure to update this to match.

With this config in place, add the following to your Prometheus image in docker-compose:

prom:
  image: prom/prometheus:v2.27.1
  ports:
    - 9090:9090
  volumes:
    - ./prom/prometheus.yml:/etc/prometheus/prometheus.yml
    - ./prom/rules.yml:/etc/prometheus/rules.yml

This mounts the prometheus.yml file we just configured to a location in the container where it will be found on startup. This also mounts the rules.yml file if you want to add alerting (see extra section at the bottom of this article if you are interested in this).

To test this, run the following command and navigate to the Prometheus UI.

docker-compose up

If you then run the following query, you should see that Prom has successfully got the JVM info from the application, showing it has scraped the metrics successfully:

Figure 3: showing a PromQL query to grab the jvm_info published by the yak-server metrics endpoint.

Thats it! You now have metrics being pulled into Prometheus from your application.

Storing Prometheus metrics in a persisted volume

In most setups it will be desirable to keep metrics outside of the Prometheus container lifecycle. Luckily for us, Prometheus places all its time series data into a known folder, which means we can mount that as a volume and keep metrics between runs of the application. Add the following to the docker-compose file:

prom:
  image: prom/prometheus:v2.27.1
  ports:
    - 9090:9090
  volumes:
    - ./prom/prometheus.yml:/etc/prometheus/prometheus.yml
    - ./prom/rules.yml:/etc/prometheus/rules.yml
    - prometheus_data:/prometheus

You can see this references the volume in the compose file, and mounts it at /prometheus. This persists our metrics between docker-compose down/up operations. However, if you do want to start fresh, simply run:

docker-compose down -v

This command will not only turn your containers off, but also remove any volumes.

So with this, we now have metrics collection and storage, its time to use them to plot a graph in Grafana!

Configuring Grafana to find Prometheus

Let’s now use our new metrics to create a dashboard. To do this, we need to tell Grafana how to find the Prometheus service. Go to ./grafana/provisioning/datasources/all.yml and add the following config:

apiVersion: 1

# tells grafana where to find the prom connection
datasources:
  - name: 'prometheus'
    type: 'prometheus'
    access: 'proxy'
    url: 'http://prom:9090'

This tells Grafana exactly where our Prometheus instance is located, which is equal to the container name in the docker-compose file. In order for Grafana to pick this config up, you need to add the following to ./grafana/config.ini

# place to find startup config
[paths]
provisioning = /etc/grafana/provisioning

[server]
enable_gzip = true

Finally, we need to make this config available to the Grafana docker image by updating the docker-compose file with:

grafana:
  image: grafana/grafana:7.5.7
  ports:
    - 3000:3000
  volumes:
    - ./grafana/provisioning/datasources:/etc/grafana/provisioning/datasources
    - ./grafana/config.ini:/etc/grafana/config.ini

With this in place, restart your application:

docker-compose down -v
docker-compose up

Then go the Grafana UI. Once you have logged in, you should be able to do the following:

Go to Manage Dashboards:

Then click on “New Dashboard”:

Next, chose to add an empty panel:

Select the Prometheus data source, which is now hooked up to our Prometheus docker container:

Then add in the same test query as before for jvm_info. Be sure to click outside of the query text input, and you should see the following:

This is not an exactly a useful panel, but it shows that we have Grafana querying Prometheus, which is scraping information from our application! The final part of this guide will show you how to create a simple dashboard, and save it so you can check it in to version control.

Creating a Grafana Dashboard

Continuing from the last section, let’s add a useful panel into our dashboard. Enter the following query instead of jvm_info:

sum(rate(thread_heartbeat_total[$__rate_interval])) by (thread_name)

In the Legend box add the following:

{{ thread_name }}

This metric comes directly from our application. This is a counter for each thread the app runs, showing when it has finished cycling through all the work it has to do. The metric lets you visualise the speed of each thread, and how quickly they are getting through work within the app. Save the panel by clicking “Apply” in the top right corner, giving it a Title, such as “Thread Heartbeat Rate”.

You will now be on a page similar looking to:

So now we have a dashboard, lets save it so its always available at startup within our Grafana container.

Storing a Grafana Dashboard

On the dashboard page, go to the top right and click on the save button.

Give it a good name, and click save. Don’t worry about the folder, as will be automating where this dashboard lives next. With it saved, go to the settings wheel in the top right.

Then go to JSON Model:

This model represents the dashboard we have created, and is what we need to store in order to create it again. Copy the contents of this JSON Model into the file ./grafana/dashboards/my-dashboard.json file. Now you have your dashboard, let’s tell Grafana how to load it at startup so we dont need to make it again.

Go to ./grafana/provisioning/dashboards/all.yml, and add the following config:

apiVersion: 1

# tells grafana where to find the pre-created dashboards
providers:
  - name: 'default'
    folder: 'yak-dashboards'
    type: 'file'
    options:
      path: '/var/lib/grafana/dashboards'

This tells Grafana where to load the dashboard from on startup, along with the name of the folder to use. We now need to make these folders available inside our docker container. To do this, update the docker-compose.yml file with:

grafana:
  image: grafana/grafana:7.5.7
  ports:
    - 3000:3000
  volumes:
    - ./grafana/provisioning/dashboards:/etc/grafana/provisioning/dashboards
    - ./grafana/provisioning/datasources:/etc/grafana/provisioning/datasources
    - ./grafana/config.ini:/etc/grafana/config.ini
    - ./grafana/dashboards:/var/lib/grafana/dashboards

You can see this maps the dashboard JSON files into the container at the location we specified in ./grafana/provisioning/dashboards/all.yaml. With your dashboard saved, restart the containers:

docker-compose down -v 
docker-compose up

If you go back to the Grafana UI, you will be asked to login again. However, this time when you land on the Dashboard page, you should see a folder presented to you:

If you click on this, then open the dashboard in it, you should see your previously created dashboard:

From here, you now have end-to-end metrics, dash-boarding, and a reliable way to version and provision your dashboards so you can deploy them to different environments.

This is a great point to conclude the article. I hope this has shown you the power of some of the modern metrics tools available, and how simple they can be to setup and configure reliably. In the real-world you would then check in the files we have created somewhere, and build and publish them when they change; giving you versioning and control over dashboards and alerts. If you want to see a full example of this, integrated within the yak-server application, please visit here.

Now, one final piece of information to wrap up. If you have used Prometheus before, you know it can also provide alerting as well as querying. This final section simply completes the setup if you are interested in alerting.

Extra Information: Configuring alerts in Prometheus

If you go to the Prometheus UI, and click on the alerts tab:

You will see that there are no alerts currently configured. However, we only require the addition of a single file to enable automated alerts (i wont be showing integrating an alert manager to receive these, as its out of scope of the article).

Go to the ./prom/rules.yml file and add the following:

groups:
  - name: Yak Server Health
    rules:
      - alert: unable to gather metrics
        expr: scrape_samples_scraped{instance=~"yak-server.*"} == 0
        for: 1m
        labels:
          severity: page
        annotations:
          summary: "{{ $labels.instance }} has not provided metrics in 1m"
          description: "{{ $labels.instance }} may be offline as prom has been unable to gather metrics from it in over 1m"

This is an alert, that uses the Prometheus metrics, to detect when we are unable to gather metrics from any instance with the label like “yak-server”, for over 1 minute. This alert would fire if the app isn’t running, Prometheus has been configured incorrectly, the network has died…. lots of reasons to want to know when this happens.

With this in place, restart your docker instance:

docker-compose down -v
docker-compose up

Go back to the alerts page in the Prometheus UI, and you should see your alarm in the state of not firing.

Now if you stop the yak-server docker container:

docker ps # find the container for yak app
docker stop <container id>

If you wait around 1 minute, which is what we configured in our alert config, you should see the alarm go to pending then firing. This would then notify any alert manager you have configured, or simply display in the Prom UI if you wanted to manually check the alerts.

This should hopefully now contain everything you need to know for full end to end metrics and alerting.

If you have any questions, feel free to reach out to me:

Joe Honour – Senior Software Engineer – AND Digital | LinkedIn

Metrics: Reliably Configuring Prometheus and Grafana with Docker. was originally published in Level Up Coding on Medium, where people are continuing the conversation by highlighting and responding to this story.


This content originally appeared on Level Up Coding - Medium and was authored by Joe Honour

Tech Stack: Prometheus, Grafana, Docker, Docker-compose.

There is a lot of great guides on how to integrate metrics into your application so you can use them with Prometheus, for example there is an excellent Java Client and first level support from popular web frameworks such as Spring. However, I struggled to find information on how these metrics are collected, and how you would go about versioning and deploying dashboards reliably once you had them setup.

Therefore, this guide will take you through an overview of how Prometheus collects metrics, before moving on to visualizing them with Grafana. As we get things running, we will focus on how we store and version this config, so it can be deployed reliably with Docker.

Step 1: Understanding how metrics are collected

This guide will be running an application exposing metrics on host and port: http://yak-server:9001/metrics.

Given you have your application running, and you have integrated some metrics into it, the first thing to understand is how Prometheus accesses that information.

When you run Prometheus, you configure it to “scrape” certain destinations. What this means is that Prometheus, say every 15 seconds, will call your metrics endpoint and then store the result; this is known as a pull-based architecture (Prometheus is pulling the metrics to it). Therefore, your application just needs to provide a location to get metrics from, and Prometheus will do the rest (Figure 1).

Figure 1: Prom queries your app every 15 seconds to pull in new metrics.

Once Prometheus gathers your metrics, it will store them in a time series database, making them available for querying. In order to query them, Prometheus provides a powerful query language that you can use via the API or in the Prometheus UI. These queries are what you then configure in Grafana to display graphs and information in dashboards (Figure 2).

Figure 2: Grafana queries PromQL based on what you setup in your dashboard.

So with this architecture, in order to reliably deploy it, we need to:

  • Configure the Prometheus scrape config, to tell it where our application is located and how often to get metrics from it.
  • Store the metrics in a persisted volume, so we don’t lose them if the Prometheus Docker image is stopped.
  • Configure Grafana to find the Prometheus instance, so we can visualise the metrics we are collecting.
  • Configure a Grafana dashboard with the queries/graphs we want to display.
  • Store these dashboards so we can deploy them again if the Grafana Docker image is stopped.

Step 2: Deploying the metrics pipeline with Docker Compose

The first step to getting a reliable deployment of this architecture, is to simply get the images we need running locally. Create the following folder structure:

your-folder-location/
┣ grafana/
┃ ┣ dashboards/
┃ ┃ ┗ my-dashboard.json
┃ ┗ provisioning/
┃   ┣ dashboards/
┃ ┃ ┃ ┗ all.yml
┃   ┣ datasources/
┃ ┃ ┃ ┗ all.yml
┃   ┗ config.ini
┣ prom/
┃ ┣ prometheus.yml
┃ ┗ rules.yml
┗ docker-compose.yml

This structure contains everything we will need in this article, and we will go through each file as we incorporate it into our deployment. If you would like to skip this article, and just go to an example deployment, visit here.

Open the docker-compose.yml and copy the starting configuration:

version: '3.7'

services:

  yak-server:
    image: guardiandevelopment/yak-server:1.0.0.SNAPSHOT
    ports:
      - 9000:9000
      - 9001:9001
    deploy:
      resources:
        limits:
          cpus: '1'
          memory: 250M

  prom:
    image: prom/prometheus:v2.27.1
    ports:
      - 9090:9090

  grafana:
    image: grafana/grafana:7.5.7
    ports:
      - 3000:3000

volumes:
  # store prom metrics between runs
  prometheus_data: {}

networks:
  default:
    name: metrics-network

With this stored, you should be able to run the following command:

docker-compose up

and then access the following links:

  • Prometheus: http://localhost:9090/graph
  • Grafana: http://localhost:3000/ The login by default is admin:admin. Once you have logged in, just skip setting up another user as that is beyond the scope of this guide.
  • The metrics from the app: http://localhost:9001/metrics This will be what we tell Prometheus to scrape, but it gives you an idea of the format and type of metrics available to us in this guide.

Now we have done this, we will go through incorporating each config file to build our architecture from Step 1. For now, you can run the following command to stop everything:

# turn off all images and volumes, remove the -v 
# if you want volumes persisted between between runs
docker-compose down -v

Configuring Prometheus scrape targets

In order for Prometheus to gather metrics from our application, we need to configure its scrape targets. This will tell Prometheus how frequently to gather metrics, and where to find the endpoint for our application. Add the following config to ./prom/prometheus.yml:

global:
  scrape_interval: 15s
  evaluation_interval: 15s
  external_labels:
    monitor: 'yak-monitor'

# extra: include rules for alerting
rule_files:
  - rules.yml

# IMPORTANT BIT HERE: targets for scraping metrics from
scrape_configs:

  - job_name: 'prometheus'
    scrape_interval: 5s
    static_configs:
      - targets: [ 'prom:9090' ]

  - job_name: 'yak-server'
    scrape_interval: 5s
    static_configs:
      - targets: [ 'yak-server:9001' ]

This important part of this config is the scrape_configs section. This configures 2 targets:

  1. It scrapes itself for metrics. This is pretty standard and allows you to see metrics on the performance and health of your Prometheus instance.
  2. We setup yak-server:9001, which scrapes our application. As this is running in docker-compose, the yak-server DNS entry will be available to Prometheus. If you have ran your own app, make sure to update this to match.

With this config in place, add the following to your Prometheus image in docker-compose:

prom:
  image: prom/prometheus:v2.27.1
  ports:
    - 9090:9090
  volumes:
    - ./prom/prometheus.yml:/etc/prometheus/prometheus.yml
    - ./prom/rules.yml:/etc/prometheus/rules.yml

This mounts the prometheus.yml file we just configured to a location in the container where it will be found on startup. This also mounts the rules.yml file if you want to add alerting (see extra section at the bottom of this article if you are interested in this).

To test this, run the following command and navigate to the Prometheus UI.

docker-compose up

If you then run the following query, you should see that Prom has successfully got the JVM info from the application, showing it has scraped the metrics successfully:

Figure 3: showing a PromQL query to grab the jvm_info published by the yak-server metrics endpoint.

Thats it! You now have metrics being pulled into Prometheus from your application.

Storing Prometheus metrics in a persisted volume

In most setups it will be desirable to keep metrics outside of the Prometheus container lifecycle. Luckily for us, Prometheus places all its time series data into a known folder, which means we can mount that as a volume and keep metrics between runs of the application. Add the following to the docker-compose file:

prom:
  image: prom/prometheus:v2.27.1
  ports:
    - 9090:9090
  volumes:
    - ./prom/prometheus.yml:/etc/prometheus/prometheus.yml
    - ./prom/rules.yml:/etc/prometheus/rules.yml
    - prometheus_data:/prometheus

You can see this references the volume in the compose file, and mounts it at /prometheus. This persists our metrics between docker-compose down/up operations. However, if you do want to start fresh, simply run:

docker-compose down -v

This command will not only turn your containers off, but also remove any volumes.

So with this, we now have metrics collection and storage, its time to use them to plot a graph in Grafana!

Configuring Grafana to find Prometheus

Let’s now use our new metrics to create a dashboard. To do this, we need to tell Grafana how to find the Prometheus service. Go to ./grafana/provisioning/datasources/all.yml and add the following config:

apiVersion: 1

# tells grafana where to find the prom connection
datasources:
  - name: 'prometheus'
    type: 'prometheus'
    access: 'proxy'
    url: 'http://prom:9090'

This tells Grafana exactly where our Prometheus instance is located, which is equal to the container name in the docker-compose file. In order for Grafana to pick this config up, you need to add the following to ./grafana/config.ini

# place to find startup config
[paths]
provisioning = /etc/grafana/provisioning

[server]
enable_gzip = true

Finally, we need to make this config available to the Grafana docker image by updating the docker-compose file with:

grafana:
  image: grafana/grafana:7.5.7
  ports:
    - 3000:3000
  volumes:
    - ./grafana/provisioning/datasources:/etc/grafana/provisioning/datasources
    - ./grafana/config.ini:/etc/grafana/config.ini

With this in place, restart your application:

docker-compose down -v
docker-compose up

Then go the Grafana UI. Once you have logged in, you should be able to do the following:

Go to Manage Dashboards:

Then click on “New Dashboard”:

Next, chose to add an empty panel:

Select the Prometheus data source, which is now hooked up to our Prometheus docker container:

Then add in the same test query as before for jvm_info. Be sure to click outside of the query text input, and you should see the following:

This is not an exactly a useful panel, but it shows that we have Grafana querying Prometheus, which is scraping information from our application! The final part of this guide will show you how to create a simple dashboard, and save it so you can check it in to version control.

Creating a Grafana Dashboard

Continuing from the last section, let’s add a useful panel into our dashboard. Enter the following query instead of jvm_info:

sum(rate(thread_heartbeat_total[$__rate_interval])) by (thread_name)

In the Legend box add the following:

{{ thread_name }}

This metric comes directly from our application. This is a counter for each thread the app runs, showing when it has finished cycling through all the work it has to do. The metric lets you visualise the speed of each thread, and how quickly they are getting through work within the app. Save the panel by clicking “Apply” in the top right corner, giving it a Title, such as “Thread Heartbeat Rate”.

You will now be on a page similar looking to:

So now we have a dashboard, lets save it so its always available at startup within our Grafana container.

Storing a Grafana Dashboard

On the dashboard page, go to the top right and click on the save button.

Give it a good name, and click save. Don’t worry about the folder, as will be automating where this dashboard lives next. With it saved, go to the settings wheel in the top right.

Then go to JSON Model:

This model represents the dashboard we have created, and is what we need to store in order to create it again. Copy the contents of this JSON Model into the file ./grafana/dashboards/my-dashboard.json file. Now you have your dashboard, let’s tell Grafana how to load it at startup so we dont need to make it again.

Go to ./grafana/provisioning/dashboards/all.yml, and add the following config:

apiVersion: 1

# tells grafana where to find the pre-created dashboards
providers:
  - name: 'default'
    folder: 'yak-dashboards'
    type: 'file'
    options:
      path: '/var/lib/grafana/dashboards'

This tells Grafana where to load the dashboard from on startup, along with the name of the folder to use. We now need to make these folders available inside our docker container. To do this, update the docker-compose.yml file with:

grafana:
  image: grafana/grafana:7.5.7
  ports:
    - 3000:3000
  volumes:
    - ./grafana/provisioning/dashboards:/etc/grafana/provisioning/dashboards
    - ./grafana/provisioning/datasources:/etc/grafana/provisioning/datasources
    - ./grafana/config.ini:/etc/grafana/config.ini
    - ./grafana/dashboards:/var/lib/grafana/dashboards

You can see this maps the dashboard JSON files into the container at the location we specified in ./grafana/provisioning/dashboards/all.yaml. With your dashboard saved, restart the containers:

docker-compose down -v 
docker-compose up

If you go back to the Grafana UI, you will be asked to login again. However, this time when you land on the Dashboard page, you should see a folder presented to you:

If you click on this, then open the dashboard in it, you should see your previously created dashboard:

From here, you now have end-to-end metrics, dash-boarding, and a reliable way to version and provision your dashboards so you can deploy them to different environments.

This is a great point to conclude the article. I hope this has shown you the power of some of the modern metrics tools available, and how simple they can be to setup and configure reliably. In the real-world you would then check in the files we have created somewhere, and build and publish them when they change; giving you versioning and control over dashboards and alerts. If you want to see a full example of this, integrated within the yak-server application, please visit here.

Now, one final piece of information to wrap up. If you have used Prometheus before, you know it can also provide alerting as well as querying. This final section simply completes the setup if you are interested in alerting.

Extra Information: Configuring alerts in Prometheus

If you go to the Prometheus UI, and click on the alerts tab:

You will see that there are no alerts currently configured. However, we only require the addition of a single file to enable automated alerts (i wont be showing integrating an alert manager to receive these, as its out of scope of the article).

Go to the ./prom/rules.yml file and add the following:

groups:
  - name: Yak Server Health
    rules:
      - alert: unable to gather metrics
        expr: scrape_samples_scraped{instance=~"yak-server.*"} == 0
        for: 1m
        labels:
          severity: page
        annotations:
          summary: "{{ $labels.instance }} has not provided metrics in 1m"
          description: "{{ $labels.instance }} may be offline as prom has been unable to gather metrics from it in over 1m"

This is an alert, that uses the Prometheus metrics, to detect when we are unable to gather metrics from any instance with the label like “yak-server”, for over 1 minute. This alert would fire if the app isn't running, Prometheus has been configured incorrectly, the network has died…. lots of reasons to want to know when this happens.

With this in place, restart your docker instance:

docker-compose down -v
docker-compose up

Go back to the alerts page in the Prometheus UI, and you should see your alarm in the state of not firing.

Now if you stop the yak-server docker container:

docker ps # find the container for yak app
docker stop <container id>

If you wait around 1 minute, which is what we configured in our alert config, you should see the alarm go to pending then firing. This would then notify any alert manager you have configured, or simply display in the Prom UI if you wanted to manually check the alerts.

This should hopefully now contain everything you need to know for full end to end metrics and alerting.

If you have any questions, feel free to reach out to me:

Joe Honour - Senior Software Engineer - AND Digital | LinkedIn

Metrics: Reliably Configuring Prometheus and Grafana with Docker. was originally published in Level Up Coding on Medium, where people are continuing the conversation by highlighting and responding to this story.


This content originally appeared on Level Up Coding - Medium and was authored by Joe Honour


Print Share Comment Cite Upload Translate Updates
APA

Joe Honour | Sciencx (2021-06-20T16:33:12+00:00) Metrics: Reliably Configuring Prometheus and Grafana with Docker.. Retrieved from https://www.scien.cx/2021/06/20/metrics-reliably-configuring-prometheus-and-grafana-with-docker/

MLA
" » Metrics: Reliably Configuring Prometheus and Grafana with Docker.." Joe Honour | Sciencx - Sunday June 20, 2021, https://www.scien.cx/2021/06/20/metrics-reliably-configuring-prometheus-and-grafana-with-docker/
HARVARD
Joe Honour | Sciencx Sunday June 20, 2021 » Metrics: Reliably Configuring Prometheus and Grafana with Docker.., viewed ,<https://www.scien.cx/2021/06/20/metrics-reliably-configuring-prometheus-and-grafana-with-docker/>
VANCOUVER
Joe Honour | Sciencx - » Metrics: Reliably Configuring Prometheus and Grafana with Docker.. [Internet]. [Accessed ]. Available from: https://www.scien.cx/2021/06/20/metrics-reliably-configuring-prometheus-and-grafana-with-docker/
CHICAGO
" » Metrics: Reliably Configuring Prometheus and Grafana with Docker.." Joe Honour | Sciencx - Accessed . https://www.scien.cx/2021/06/20/metrics-reliably-configuring-prometheus-and-grafana-with-docker/
IEEE
" » Metrics: Reliably Configuring Prometheus and Grafana with Docker.." Joe Honour | Sciencx [Online]. Available: https://www.scien.cx/2021/06/20/metrics-reliably-configuring-prometheus-and-grafana-with-docker/. [Accessed: ]
rf:citation
» Metrics: Reliably Configuring Prometheus and Grafana with Docker. | Joe Honour | Sciencx | https://www.scien.cx/2021/06/20/metrics-reliably-configuring-prometheus-and-grafana-with-docker/ |

Please log in to upload a file.



There are no updates yet.
Click the Upload button above to add an update.

You must be logged in to translate posts. Please log in or register.

Related Posts