MI User Guide

This document guides the reader to set up their own monitoring target and and monitor their instance using the EFPF monitoring system.

Metrics Monitoring using Prometheus

1. Exporting the metrics to Prometheus

Export metrics using cAdvisor

If you are running your containers in a virtual machine and want to expose metrics related to these containers to prometheus, cAdvisor can be a handy tool. Please follow the following steps to run a cAdvisor docker container.

Note: It is necessary to secure the endpoint using authentication layer. Since this depends on individual deployments, they are not covered in this tutorial.

  1. Clone the target repository
git clone https://gitlab.fit.fraunhofer.de/efpf-pilots/monitoring-target/-/tree/master
  1. cd to monitoring-target/cadvisor
  2. run the cadvisor docker container
bash run.sh

This command runs the cadvisor and also sets up a cron job to keep updating the cadvisor container. Note that cAdvidor runs in previleged mode and mounts the root of the filesystem as a volume. Updating this frequently ensures that we do not miss any security updates against the vlnerabilities.

Use your own prometheus exporters

If you want to use a custom exporter, there are plenty of other exporters listed in the prometheus official page.

2. Adding the scrape config in prometheus

Once you have set-up your exporter, the Prometheus instance should be configured to to receive the metrics from the new exporter. This can be done only by the administrator of the monitoring infrastructure.

If you are not an admin of the monitoring instance, please contact the monitoring infrastructure administrators (FIT) to add your exporter to the prometheus targets.

If you are admin, then please follow the instructions specified in admin guide

Log Monitoring Using Loki

1. Exporting the Logs to Loki

If you are running your containers in server and want to expose logs related to these containers to Loki, vector can be a handy tool. Please follow the following steps to run a vector docker container.

  1. Clone the target repository
git clone https://gitlab.fit.fraunhofer.de/efpf-pilots/monitoring-target/-/tree/master
  1. cd to monitoring-target/vector. The docker-compose.yml in this directory runs two containers.
    1. Container which runs vector itself
    2. Obtainer which runs the script obtain-token.sh which constantly obtains the client token from keycloak and updates the vector config with latest token.
  2. Edit the docker-compose.yml file to update the client details for bearer authentication.
  3. Run the vector docker container using the command
docker-compose up

Visualization Using Grafana

Grafana visualization is accessible to the EFPF users at https://monitor.efpf.linksmart.eu/grafana/.

Exploring Loki and Prometheus:

Both the data sources of Loki and Prometheus provide exploring functionalities. Loki can be explored using LogQL queries and Prometheus can be explored using PromQL.

LogQL examples

In this section we shall give an example of exploring NGINX logs to get an insight about http requests:

  • Get the nginx logs where there was HTTP error : This query shall show the logs there are 4xx or 5xx errors:

    {containers="<container_name>"} |~ "/grafana" |~ "HTTP.1[.]1.. [4-5][0-9][0-9]"

  • Requests to an enpoint per day:

    count_over_time({containers="<container_name>"} |~ "${endpoints}"[1d])

  • Rate of HTTP errors for 10 minutes:

    count_over_time({containers="<container_name>"} |~ "/grafana" |~ "HTTP.1[.]1.. [4-5][0-9][0-9]" [10m])

    The result is a metric. Therefore this can be visualized using panels

PromQL examples

  • Overall CPU usage (percentage) of all the containers running in a host for 5 minutes

    rate(container_cpu_user_seconds_total{image!="",job="<target name>"}[5m]) * 100

  • Overall Memory usage of all the containers running in a host

    container_memory_usage_bytes{image!="",job="<target name>"}

Dashboard for Metrics Monitored by Prometheus:

If the exporter to Prometheus is cAdvisor, then a ready made cadvisor-prometheus panel can be used to visualize the docker containers.

Docker monitoring Figure 1: Monitoring the docker containers running in a virtual machine

1. Load the template Json

Import JSON Figure 2: Creating a Grafana dashboard by importing json

Load the contents of panel.json to create a new dashboard. Then give a name of your choice and select the folder where the dashboard should reside.

2. Set the right job_name as environmental variable

Now, to set the right cAdvisor source, change the variable associated with the Dashboard to the matching job_name set in prometheus as follows

Setting environmental variables for the dashboard Figure 3: Setting the right job name using the environmental variables

Dashboard for service status monitoring:

Service status can be monitored using statusmap Plugin.

Displaying service status Figure 4: Displaying service status

Each rows of the panel can follow below query sample:

count((time()-container_last_seen{job="${cadvisor_source}",name="mycontainername"})<15)*1 or vector(0)

where, ${cadvisor_source} is set as the environmental variable of the dashboard (see figure 3).

Custom Panels

Additional custom panels can be created using various panels offered for Grafana. Custom queries can be written in PromQL.