Monitoring and Logging with Prometheus and Grafana

In today’s dynamic DevOps environments, monitoring and logging are essential for ensuring application reliability and performance. Tools like Prometheus and Grafana provide powerful solutions for collecting, visualizing, and analyzing metrics and logs in real-time.

This guide will walk you through the process of automating monitoring and logging for applications using Prometheus and Grafana, with hands-on examples, step-by-step processes, and official references.

Why Monitor Applications?

The Importance of Monitoring

1. Proactive Issue Detection:
   • Identify and resolve issues before they affect end-users.
2. Performance Optimization:
   • Analyze bottlenecks and improve application performance.
3. Operational Insights:
   • Gain visibility into application behavior and infrastructure health.
4. Compliance and Auditing:
   • Track logs and metrics for security and compliance purposes.

Monitoring and Logging with Prometheus and Grafana

Prometheus

Prometheus is an open-source monitoring and alerting toolkit that collects metrics from targets at specified intervals, stores them, and provides a query language (PromQL) to retrieve and analyze metrics.

• Key Features:
  • Pull-based metric collection.
  • Multi-dimensional data model.
  • Built-in alerting.

Grafana

Grafana is an open-source visualization tool that integrates with Prometheus and other data sources to create interactive dashboards for metrics visualization.

• Key Features:
  • Customizable dashboards.
  • Integration with multiple data sources.
  • Alerting and notifications.

Prerequisites

1. A running application:
   • Use any web application or containerized service.
2. Docker installed:
   • Install Docker:

     sudo apt update
     sudo apt install docker.io -y
     

   • Verify Docker installation:

     docker --version
     

3. Docker Compose installed:
   • Install Docker Compose:

     sudo apt install docker-compose -y
     

   • Verify installation:

     docker-compose --version
     

Step 1: Set Up Prometheus

1.1 Create Prometheus Configuration

1. Create a directory for Prometheus:

   mkdir prometheus && cd prometheus
   

2. Create a prometheus.yml configuration file:

   global:
     scrape_interval: 15s
   
   scrape_configs:
     - job_name: "node"
       static_configs:
         - targets: ["localhost:9100"]
   

   • scrape_interval: How often Prometheus scrapes metrics.
   • job_name: Identifier for the metrics source.

1.2 Create a Docker Compose File

Create a docker-compose.yml file:

version: "3.7"
services:
  prometheus:
    image: prom/prometheus:latest
    container_name: prometheus
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml

1.3 Start Prometheus

1. Start the Prometheus container:

   docker-compose up -d
   

2. Access Prometheus:
   • Open http://localhost:9090 in your browser.

Step 2: Server monitoring with grafana​ : Set Up Grafana

2.1 Create a Docker Compose File

Add Grafana to docker-compose.yml:

  grafana:
    image: grafana/grafana:latest
    container_name: grafana
    ports:
      - "3000:3000"

2.2 Start Grafana

1. Start Grafana:

   docker-compose up -d grafana
   

2. Access Grafana:
   • Open http://localhost:3000 in your browser.
   • Default credentials: admin / admin.

Step 3: Configure Prometheus as a Data Source in Grafana

1. Log in to Grafana (http://localhost:3000).
2. Navigate to Configuration > Data Sources > Add Data Source.
3. Select Prometheus.
4. Enter the URL for Prometheus:

   http://prometheus:9090
   

5. Click Save & Test.

Step 4: Visualize Metrics with Grafana Dashboards

1. Create a new dashboard:
   • Go to Dashboards > New Dashboard > Add a New Panel.
2. Query Prometheus for metrics:
   • Example PromQL query:

     node_cpu_seconds_total
     

3. Customize visualizations:
   • Choose graph types, time ranges, and thresholds.

Step 5: Enable Monitoring for Applications

5.1 Monitor a Node.js Application

1. Install the Prometheus Client library:

   npm install prom-client
   

2. Add metrics to your application:

   const express = require('express');
   const client = require('prom-client');
   const app = express();
   
   const counter = new client.Counter({
       name: 'http_requests_total',
       help: 'Total number of HTTP requests',
   });
   
   app.get('/', (req, res) => {
       counter.inc();
       res.send('Hello, Prometheus!');
   });
   
   client.collectDefaultMetrics();
   app.get('/metrics', (req, res) => {
       res.set('Content-Type', client.register.contentType);
       res.end(client.register.metrics());
   });
   
   app.listen(3000, () => console.log('App running on port 3000'));
   

3. Update Prometheus configuration:

   scrape_configs:
     - job_name: "nodejs"
       static_configs:
         - targets: ["localhost:3000"]
   

4. Restart Prometheus:

   docker-compose restart prometheus
   

5.2 Enable Alerting

1. Add alerting rules to prometheus.yml:

   rule_files:
     - "alerts.yml"
   
   alerting:
     alertmanagers:
       - static_configs:
           - targets: ["localhost:9093"]
   

2. Create alerts.yml:

   groups:
     - name: example
       rules:
         - alert: HighCPUUsage
           expr: node_cpu_seconds_total > 0.5
           for: 2m
           labels:
             severity: critical
           annotations:
             summary: "High CPU usage detected"
             description: "CPU usage is above 50% for the last 2 minutes."
   

3. Reload Prometheus configuration:

   docker-compose restart prometheus
   

Advanced Features

1. Centralized Logging

1. Set up a logging stack using the ELK Stack (Elasticsearch, Logstash, Kibana).
2. Stream application logs to Elasticsearch:
   • Use tools like Fluentd or Filebeat.

2. Distributed Tracing

1. Implement tracing with tools like Jaeger or Zipkin.
2. Correlate logs, traces, and metrics for deeper insights.

3. Exporters

Prometheus relies on exporters for collecting metrics:

• Node Exporter: System metrics.
• Blackbox Exporter: Network probing.

Install the Node Exporter:

docker run -d -p 9100:9100 prom/node-exporter

Conclusion

Monitoring and logging with Prometheus and Grafana provide a robust solution for tracking application performance and system health. By automating monitoring workflows and visualizing metrics in Grafana, you can ensure reliability, scalability, and operational excellence.

Would you like me to proceed with post #5 (Configuration Management)? Let me know!

References

• Prometheus Documentation
• Grafana Documentation
• Node Exporter
• monitor kubernetes cluster with grafana