Incremental Backup Using rsync

Incremental backups save only the changes made since the last backup, significantly reducing storage requirements and improving backup efficiency. The Linux rsync command is an excellent tool for creating incremental backups due to its speed, reliability, and ability to transfer only changed files.

This guide explains how to use rsync for incremental backups with examples and scripts.

How Incremental Backups Work with rsync

1. Initial Full Backup:
   • A complete copy of the source directory is created at the destination.
2. Subsequent Incremental Backups:
   • Only files that have changed (modified, added, or deleted) since the last backup are synced to the destination.
3. Optional Versioning:
   • Use hard links to maintain multiple backup snapshots without duplicating unchanged files.

Key Features of rsync for Incremental Backups

1. Delta Transfer:
   • Transfers only the differences between source and destination files.
2. Hard Links:
   • Enables efficient snapshot creation.
3. Customizable Options:
   • Preserve file permissions, ownership, and timestamps.

Basic rsync Command

rsync -av --delete /source/directory/ /destination/directory/

Options Explained:

• -a: Archive mode (preserves file permissions, ownership, and symbolic links).
• -v: Verbose (shows detailed progress).
• --delete: Removes files at the destination that no longer exist in the source.

Incremental Backup Using rsync

Bash Script: incremental_backup.sh

#!/bin/bash

# Incremental Backup Script Using rsync
# Author: [Your Name]
# Version: 1.0

# Configuration
SOURCE_DIR="/path/to/source"           # Source directory to back up
DEST_DIR="/path/to/backups"            # Destination directory for backups
SNAPSHOT_NAME="backup_$(date +%Y%m%d%H%M%S)" # Unique name for the current snapshot
LOG_FILE="/var/log/incremental_backup.log"   # Log file for backup operations

# Ensure the destination directory exists
mkdir -p "$DEST_DIR"

# Log function
log_message() {
    echo "[$(date '+%Y-%m-%d %H:%M:%S')] $1" | tee -a "$LOG_FILE"
}

# Perform the backup
log_message "Starting incremental backup..."
rsync -a --delete --link-dest="$DEST_DIR/latest" "$SOURCE_DIR/" "$DEST_DIR/$SNAPSHOT_NAME/"

if [[ $? -eq 0 ]]; then
    log_message "Backup completed successfully: $DEST_DIR/$SNAPSHOT_NAME"
    ln -snf "$DEST_DIR/$SNAPSHOT_NAME" "$DEST_DIR/latest"
else
    log_message "Backup failed."
    exit 1
fi

How the Script Works

1. Initial Setup:
   • Creates a directory for storing backups ($DEST_DIR).
2. Hard-Linking to Previous Snapshot:
   • The --link-dest option links unchanged files to the previous snapshot:

     --link-dest="$DEST_DIR/latest"
     

3. Efficient Storage:
   • Hard links ensure that unchanged files share the same physical storage, saving space.
4. Version Management:
   • Each backup is stored in a uniquely named subdirectory ($SNAPSHOT_NAME).
   • A symbolic link latest always points to the most recent backup.

Example Usage

1. Save the Script: Save the script as incremental_backup.sh and make it executable:

   chmod +x incremental_backup.sh
   

2. Run the Script:

   ./incremental_backup.sh
   

3. Verify Backups:
   • Backups are stored in the destination directory:

     ls /path/to/backups
     

     Output:

     backup_20250109120000
     backup_20250109130000
     latest -> backup_20250109130000
     

4. Inspect Logs: Check the log file for backup details:

   cat /var/log/incremental_backup.log
   

Automate Incremental Backups with cron

To run the script at regular intervals, add it to the cron scheduler:

1. Edit the crontab:

   crontab -e
   

2. Add the schedule:

   0 2 * * * /path/to/incremental_backup.sh >> /var/log/incremental_backup_cron.log 2>&1
   

This runs the script daily at 2 AM.

Advanced Features

1. Exclude Certain Files or Directories

Use the --exclude option to skip specific files or directories:

rsync -a --exclude="*.tmp" --exclude="cache/" /source/ /destination/

2. Retention Policy

Automatically delete backups older than a specific number of days:

Add the following lines to the script:

find "$DEST_DIR" -type d -name "backup_*" -mtime +7 -exec rm -rf {} \;
log_message "Deleted backups older than 7 days."

3. Network Backups

Perform backups over SSH for remote systems:

rsync -a --delete --link-dest="/remote/backups/latest" -e "ssh -p 22" /source/ user@remote-server:/remote/backups/$(date +%Y%m%d%H%M%S)/

Benefits of Using rsync for Incremental Backups

1. Efficiency:
   • Transfers only changed files, saving time and bandwidth.
2. Storage Optimization:
   • Hard-linking avoids duplicate storage for unchanged files.
3. Reliability:
   • Provides detailed logs for troubleshooting.

Scenario Examples

Scenario 1: Backing Up a Web Server

• Source: /var/www/html
• Destination: /backups/web
• Exclude cache files:

  rsync -a --exclude="cache/" /var/www/html/ /backups/web/
  

Scenario 2: Remote Backup

• Source: /data
• Destination: Remote server 192.168.1.10 at /mnt/backups:

  rsync -a --link-dest="/mnt/backups/latest" -e "ssh" /data/ [email protected]:/mnt/backups/$(date +%Y%m%d%H%M%S)/
  

Scenario 3: Retention Management

• Retain only the last 30 days of backups:

  find /backups -type d -name "backup_*" -mtime +30 -exec rm -rf {} \;
  

Conclusion

Using rsync for incremental backups provides a powerful, efficient, and reliable way to manage backups. With hard-linking, version control, and robust logging, rsync ensures that only changed files are backed up while maintaining a history of snapshots. By automating with scripts and scheduling tools like cron, you can ensure consistent and optimized backup routines.

Let me know if you need further customization or help setting up!

Learn more aout: Automating Full Database Backups, How to create incremental backups using rsync on Linux