Monitor Kubernetes node resource usage

This guide combines the “Monitor Kubernetes node resource usage” script and detailed implementation steps to provide a seamless experience. It covers prerequisites, script functionality, implementation steps, and automation scenarios.

Key Features of the Script

1. Retrieve and Display Resource Usage:
   • Tracks CPU and memory usage for each node.
   • Highlights nodes exceeding predefined thresholds.
2. Log Resource Usage:
   • Saves metrics to timestamped files for future analysis.
3. Alerting System:
   • Alerts when resource usage exceeds thresholds (optional email alerts).
4. Flexible Scheduling:
   • Automates periodic monitoring via cron jobs.

Prerequisites

1. Install and Configure kubectl

1. Install Kubernetes CLI (kubectl):

   sudo apt-get install -y kubectl  # For Debian/Ubuntu
   

   For other operating systems, follow the official Kubernetes documentation.

2. Configure kubectl access to your cluster:

   gcloud container clusters get-credentials <CLUSTER_NAME> --region <REGION>
   

   Or, set up a kubeconfig file:

   kubectl config use-context <CONTEXT_NAME>
   

3. Verify connectivity:

   kubectl cluster-info
   

2. Deploy Metrics Server

Metrics Server is required for kubectl top to fetch node resource usage:

1. Deploy Metrics Server:

   kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml
   

2. Verify Metrics Server functionality:

   kubectl top nodes
   

   If the command returns node metrics, the server is operational.

The Bash Script

Here is the script to monitor Kubernetes node resource usage:

#!/bin/bash

# Kubernetes Node Resource Usage Monitoring Script
# Author: [Your Name]
# Date: [Date]
# Version: 1.0

# Icons for visual feedback
CHECK="\u2714" # ✔
CROSS="\u274C" # ✘
INFO="\u2139"  # ℹ
WARNING="\u26A0" # ⚠️

# Thresholds for resource usage
CPU_THRESHOLD=80  # 80% CPU usage
MEMORY_THRESHOLD=80  # 80% memory usage

# Function: Check if kubectl is installed
function check_kubectl() {
    if ! command -v kubectl &> /dev/null; then
        echo -e "${CROSS} Error: kubectl is not installed. Please install kubectl to use this script."
        exit 1
    fi
    echo -e "${CHECK} kubectl is installed."
}

# Function: Check if Metrics Server is installed
function check_metrics_server() {
    if ! kubectl top nodes &> /dev/null; then
        echo -e "${CROSS} Error: Metrics Server is not installed or not functioning. Please install it."
        exit 1
    fi
    echo -e "${CHECK} Metrics Server is installed and functioning."
}

# Function: Monitor node resource usage
function monitor_node_usage() {
    echo -e "${INFO} Fetching node resource usage..."

    local nodes=$(kubectl get nodes --no-headers -o custom-columns=":metadata.name")
    echo -e "${INFO} Nodes in the cluster:"
    echo "$nodes"

    echo -e "\nNode Resource Usage:"
    echo -e "NODE\tCPU(%)\tMEMORY(%)"
    echo "-----------------------------------"

    while read -r node; do
        local usage=$(kubectl top node "$node" --no-headers)
        local cpu=$(echo "$usage" | awk '{print $2}' | sed 's/%//')
        local memory=$(echo "$usage" | awk '{print $4}' | sed 's/%//')

        if [[ $cpu -gt $CPU_THRESHOLD || $memory -gt $MEMORY_THRESHOLD ]]; then
            echo -e "$node\t${WARNING} ${cpu}%\t${WARNING} ${memory}%"
        else
            echo -e "$node\t${CHECK} ${cpu}%\t${CHECK} ${memory}%"
        fi
    done <<< "$nodes"
}

# Function: Log resource usage
function log_usage() {
    local timestamp=$(date +"%Y%m%d%H%M%S")
    local log_file="node_usage_$timestamp.log"

    echo -e "${INFO} Logging node resource usage to $log_file"
    kubectl top nodes > "$log_file"

    if [[ $? -eq 0 ]]; then
        echo -e "${CHECK} Node resource usage logged successfully."
    else
        echo -e "${CROSS} Error: Failed to log node resource usage."
    fi
}

# Main script
echo "=== Kubernetes Node Resource Usage Monitoring ==="

# Step 1: Check prerequisites
check_kubectl
check_metrics_server

# Step 2: Monitor node resource usage
monitor_node_usage

# Step 3: Optional logging
read -p "Do you want to log the resource usage? (yes/no, default: no): " log
log=${log:-no}

if [[ $log == "yes" ]]; then
    log_usage
fi

echo -e "${CHECK} Script execution completed."

Step-by-Step Implementation

1. Create and Set Up the Script

1. Open a terminal and create the script file:

   nano monitor_node_usage.sh
   

2. Paste the script into the file, then save and exit:
   • Press CTRL + O to save.
   • Press CTRL + X to exit.
3. Make the script executable:

   chmod +x monitor_node_usage.sh
   

2. Run the Script

Execute the script:

./monitor_node_usage.sh

Example Output:

=== Kubernetes Node Resource Usage Monitoring ===
✔ kubectl is installed.
✔ Metrics Server is installed and functioning.
ℹ Fetching node resource usage...
ℹ Nodes in the cluster:
node1
node2
node3

Node Resource Usage:
NODE        CPU(%)  MEMORY(%)
-----------------------------------
node1       ✔ 45%   ✔ 60%
node2       ⚠️ 85%   ✔ 70%
node3       ⚠️ 90%   ⚠️ 88%

3. Schedule the Script

Use cron to automate periodic monitoring:

1. Open the cron editor:

   crontab -e
   

2. Add a cron job to run the script hourly:

   0 * * * * /path/to/monitor_node_usage.sh >> /var/log/k8s_node_usage.log 2>&1
   

3. Save and exit.

Verify the scheduled jobs:

crontab -l

4. Log Resource Usage

If prompted, choose to log resource usage:

Do you want to log the resource usage? (yes/no, default: no): yes

The log file is saved in the format node_usage_<TIMESTAMP>.log.

View the log:

cat node_usage_20250109093000.log

Advanced Features

1. Add Email Alerts

Send email alerts when thresholds are exceeded:

1. Install mailx:

   sudo apt-get install -y mailx
   

2. Modify the script to send email alerts:

   if [[ $cpu -gt $CPU_THRESHOLD || $memory -gt $MEMORY_THRESHOLD ]]; then
       echo "High resource usage on $node: CPU=$cpu%, Memory=$memory%" | mail -s "Kubernetes Node Alert" [email protected]
   fi
   

Use Cases

1. Pre-Deployment Check: Monitor node resource availability before deploying new workloads:

   ./monitor_node_usage.sh
   

2. Scheduled Monitoring: Schedule periodic monitoring and log resource usage:

   crontab -e
   

3. High Traffic Monitoring: Run the script during peak hours to detect potential bottlenecks:

   ./monitor_node_usage.sh
   

Benefits

1. Real-Time Monitoring: Quickly identify nodes under stress.
2. Proactive Alerts: Highlight nodes exceeding thresholds with visual icons.
3. Logging: Maintain historical data for trend analysis.
4. Automation-Ready: Easily integrates with cron jobs or CI/CD pipelines.

This script provides a comprehensive and practical solution for monitoring Kubernetes node resource usage. Let me know if you’d like further enhancements or explanations!