Beginner's Guide to Prometheus: Installation and Key Architecture

Introduction

Think of Prometheus as a super helpful friend who always keeps an eye on your cloud-based apps to make sure they're working perfectly. This open-source tool is great for monitoring and sending alerts when something goes wrong, making it essential for anyone working with cloud-native applications.

In this blog post, we'll break down what Prometheus is all about. We’ll look at the problems it solves, understand how it works, and provide easy-to-follow steps to install Prometheus on different platforms.

Whether you’re an experienced DevOps engineer or just starting out, this guide will help you learn how to use Prometheus to keep your applications running smoothly. Let’s get started and see how Prometheus can make monitoring your apps easier and more effective.

Table of Contents

1. What is Prometheus?

2. Problems Solved by Prometheus

3. Prometheus Architecture

4. Installing Prometheus

   • Using Docker

   • On Kubernetes

5. Conclusion

What is Prometheus?

Prometheus is an open-source monitoring and alerting toolkit, designed to provide a reliable and scalable solution for monitoring cloud-native applications and dynamic environments. Originally developed at SoundCloud, it has become a foundational tool for many organizations.

Problems Solved by Prometheus

Prometheus addresses several key challenges in modern infrastructure:

1. Monitoring and Alerting:

   • Traditional monitoring systems often fail in dynamic environments.

   • Prometheus provides robust, flexible monitoring with a powerful query language (PromQL) and efficient alerting mechanisms.

2. Scalability:

   • As infrastructures grow, traditional monitoring solutions struggle to scale.

   • Prometheus scales horizontally by sharding data and distributing workloads.

3. Reliability:

   • Ensuring the reliability of the monitoring system itself.

   • Prometheus uses local storage and is self-contained, making it highly resilient.

Prometheus Architecture: A Full Explanation

Its architecture consists of several components that work together to collect, store, and analyze metrics. Here’s a detailed breakdown of Prometheus's architecture and how each part functions:

1. Prometheus Server

• Function: The core component responsible for scraping and storing metrics.

• How it Works:

  • The Prometheus server periodically retrieves (scrapes) metrics from configured targets over HTTP.

  • These targets can be application instances, servers, or any other system that exposes metrics.

  • The data is stored locally in a time-series database (TSDB), meaning each data point is stored with a timestamp.

  • The server also provides an interface for querying this data using PromQL, Prometheus’s powerful query language.

2. Exporters

• Function: Components that expose metrics from various systems in a format that Prometheus can scrape.

• Types of Exporters:

  • Node Exporter: Exposes hardware and OS metrics (CPU, memory, disk usage).

  • Database Exporters: For example, MySQL Exporter for MySQL databases.

  • Custom Exporters: You can create custom exporters to expose metrics from any application or system.

• How it Works:

  • Exporters run on the target systems and expose metrics over HTTP.

  • Prometheus scrapes these metrics at regular intervals.

3. Pushgateway

• Function: Allows short-lived or batch jobs to push metrics to Prometheus.

• How it Works:

  • Some jobs, like batch jobs or cron jobs, may not run long enough to be scraped by Prometheus.

  • These jobs push their metrics to the Pushgateway.

  • The Pushgateway then exposes these metrics for Prometheus to scrape.

• It’s important to note that the Pushgateway does not store metrics long-term; it simply acts as an intermediary.

4. Service Discovery

• Function: Automatically discovers targets to scrape, particularly useful in dynamic environments like Kubernetes.

• How it Works:

  • Prometheus can integrate with various service discovery mechanisms (Kubernetes, Consul, DNS, etc.).

  • It dynamically updates its list of targets based on the current state of the infrastructure.

  • This allows Prometheus to automatically adjust to changes, such as new services being deployed or old ones being removed.

5. Alertmanager

• Function: Manages alerts generated by Prometheus.

• How it Works:

  • Prometheus can be configured to trigger alerts based on specific conditions (e.g., high error rate, low disk space).

  • These alerts are sent to Alertmanager, which handles deduplication, grouping, and routing.

  • Alertmanager can send notifications through various channels, such as email, Slack, or PagerDuty.

6. PromQL (Prometheus Query Language)

• Function: A powerful language for querying time-series data.

• How it Works:

  • PromQL allows users to select and aggregate time-series data in flexible ways.

  • You can write queries to extract specific metrics, calculate averages, create complex aggregations, and more.

  • For example, you might query the average response time of a web server over the past hour or the CPU usage of all servers over the past day.

7. Web UI and Grafana

• Function: Interfaces for visualizing and querying metrics.

• How it Works:

  • Prometheus has a built-in web UI that allows users to perform basic queries and view metrics.

  • For more advanced visualizations, Prometheus integrates seamlessly with Grafana.

  • Grafana provides a rich set of features for creating custom dashboards and visualizations, making it easier to monitor and analyze metrics.

Let's walk through an example to see how these components work together in a real-world scenario:

Scenario: Monitoring Microservices in a Kubernetes Cluster

Step 1: Service Discovery

• Prometheus is deployed in a Kubernetes cluster.

• It uses Kubernetes service discovery to automatically find and monitor all microservices running in the cluster.

• As new microservices are deployed or old ones are removed, Prometheus dynamically updates its target list.

Step 2: Metrics Collection

• Each microservice exposes metrics through an HTTP endpoint using an appropriate exporter.

• Prometheus server scrapes these metrics at regular intervals.

Step 3: Storing Metrics

• The scraped metrics are stored in Prometheus’s time-series database.

• Each data point is stored with a timestamp, allowing for historical analysis.

Step 4: Querying Metrics

• You can use PromQL to query the metrics. For instance, you can calculate the average response time of a specific microservice or the error rate of another.

• Example query: avg(rate(http_requests_total[5m])) to find the average rate of HTTP requests over the last 5 minutes.

Step 5: Setting Alerts

• You configure Prometheus to trigger alerts based on conditions. For example, an alert is triggered if the error rate exceeds a certain threshold.

• These alerts are sent to Alertmanager.

Step 6: Managing Alerts

• Alertmanager receives the alerts, deduplicates them, and groups related alerts together.

• It then routes the alerts to the appropriate notification channels, such as Slack or email.

Step 7: Visualization

• You use Grafana to create a dashboard that visualizes key metrics in real-time.

• The dashboard displays metrics such as CPU usage, memory usage, error rates, and response times.

Installing Prometheus

Using Docker

docker run -d --name=prometheus -p 9090:9090 prom/prometheus

To use a custom configuration:

docker run -d --name=prometheus -p 9090:9090 -v /path/to/prometheus.yml:/etc/prometheus/prometheus.yml prom/prometheus

The Prometheus GUI dashboard should be viewed on port 9090.

On Kubernetes

1. Add the Prometheus Helm Repository:

    helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
   

   

2. Update Helm Repositories:

    helm repo update
   

   

3. Install Prometheus using Helm:

    helm install prometheus prometheus-community/prometheus
   

   

   Check the Services Created by Helm

   • After installation, Helm creates several Kubernetes resources, including services. You can list these services using the following command:

       kubectl get pods
     

     

       kubectl get svc
     

     

     You should see services like prometheus-server, prometheus-alertmanager, etc.

Accessing Prometheus via the Service

• The prometheus-server service exposes the Prometheus UI and API. By default, this service might be a ClusterIP type, which is only accessible within the cluster.

• To access Prometheus from outside the cluster, you have a few options:

  • Port Forwarding: You can use kubectl port-forward to forward a local port to the Prometheus server service:

      kubectl port-forward svc/prometheus-server 9090:80
    

    After running this command, you can access Prometheus by navigating to http://localhost:9090 in your web browser.

    

  • NodePort or LoadBalancer Service: Modify the service type to NodePort or LoadBalancer to expose Prometheus outside the cluster. You can edit the service configuration using kubectl edit service prometheus-server and change the spec.type to NodePort or LoadBalancer.

Using Prometheus UI and API

• Once you have access to Prometheus, you can use the Prometheus web UI to visualize metrics, configure alerts, and query data using PromQL.

• The Prometheus API can be accessed via HTTP endpoints, allowing you to integrate Prometheus with other monitoring tools and automation scripts.

Conclusion

Prometheus is a powerful monitoring and alerting tool designed for modern, dynamic environments like Kubernetes clusters. It includes key components such as the Prometheus server, exporters, Pushgateway, service discovery, Alertmanager, and PromQL, which work together to collect, store, and analyze metrics efficiently.

Deploying Prometheus with Helm simplifies the setup and management process, providing automated resource creation and easy access methods. This allows you to quickly use Prometheus for monitoring, alerting, and visualization, ensuring the health and performance of your applications and infrastructure.