STORAGE ENGINEERING

Linux backup & disaster recovery: rsync, Borg, Restic

Category: Backup & RecoveryTechnologies: rsync, BorgBackup, Restic, pg_dump, LVM/Btrfs/ZFS snapshots

Backups are the last line of defense against data loss from hardware failure, human error, ransomware, and natural disasters. This guide covers the foundational principles, the three primary Linux backup tools, database-specific strategies, and disaster recovery planning.

The 3-2-1 rule (and 3-2-1-1-0 extension)

ClauseRequirementProtects against
3Three copies of data (original + 2 backups)Single-point failure
2Two different media typesMedia-specific failures (controller bug, format incompatibility)
1One copy offsitePhysical disaster (fire, flood, datacenter outage)
1 (extension)One immutable or air-gapped copyRansomware and insider threats
0 (extension)Zero errors after verified restoreSilent backup corruption

RPO and RTO

MetricDefinitionDetermines
RPO (Recovery Point Objective)Maximum acceptable data loss, measured in time backward from failureBackup frequency and replication strategy
RTO (Recovery Time Objective)Maximum acceptable downtime, measured forward from failureRecovery architecture and procedures
RPO targetStrategy
Near-zero (seconds)Continuous replication, synchronous DB streaming, ZFS send/receive with frequent snapshots
1-15 minutesFrequent incremental backups, WAL/binlog shipping
1-24 hoursDaily Borg/restic archives, nightly database dumps
>24 hoursWeekly full backups with daily differentials

rsync

rsync is the foundation of Linux file synchronization. Its delta-transfer algorithm sends only the changed portions of files, making it efficient for incremental backups.

# Basic local backup
rsync -a --delete /source/ /backup/

# Over SSH with progress
rsync -a -e ssh --progress --partial /source/ user@host:/backup/

# Incremental backup with hardlinks (space-efficient)
rsync -a --delete --link-dest=/backup/previous /source/ /backup/current/

# Full system backup (excluding virtual filesystems)
rsync -aAXHv \
  --exclude='/dev/*' --exclude='/proc/*' --exclude='/sys/*' \
  --exclude='/tmp/*' --exclude='/run/*' --exclude='/mnt/*' \
  --exclude='/media/*' --exclude='/lost+found/' \
  / /path/to/backup

Key flags: -a (archive = -rlptgoD), --delete (remove files not in source), --link-dest (hardlink-based incrementals), -e ssh (remote shell).

rsync limitations

rsync has no built-in deduplication (beyond --link-dest hardlinks), no built-in encryption of stored data (SSH provides transport encryption only), and no snapshot management. For deduplicated, encrypted, snapshot-managed backups, use Borg or Restic.

BorgBackup (borg)

Borg uses content-defined chunking (CDC) for deduplication: files are split into variable-length chunks based on content, not fixed offsets. Chunks are identified by cryptographic hash and deduplicated across all archives in the repository.

Repository initialization

# repokey encryption (key + passphrase in repository)
borg init --encryption=repokey /path/to/repo

# keyfile encryption (key in ~/.config/borg/keys/, passphrase separate)
borg init --encryption=keyfile user@host:/path/to/repo

Full backup workflow

#!/bin/bash
export BORG_PASSPHRASE="your-passphrase"
REPO="/backup/borg"

# Create backup with compression
borg create --stats --progress --compression zstd,3 \
  --exclude 'home/*/.cache' --exclude '*.tmp' \
  "$REPO::{hostname}-{now:%Y-%m-%d_%H-%M}" \
  /etc /home /var /root

# Prune old backups
borg prune --list --glob-archives '{hostname}-*' \
  --keep-daily 7 --keep-weekly 4 --keep-monthly 6 \
  "$REPO"

# Compact (reclaim space after prune)
borg compact "$REPO"

# Verify
borg check --verify-data "$REPO"

Listing, extraction, and mounting

# List all archives
borg list /path/to/repo

# List contents of an archive
borg list /path/to/repo::Monday

# Extract
borg extract /path/to/repo::Monday
borg extract /path/to/repo::Monday home/user/Documents

# Mount as FUSE filesystem
borg mount /path/to/repo /mnt/borg
borg umount /mnt/borg

Compression options

AlgorithmLevelsNotes
lz4-Default. Very fast, low ratio.
zstd1-22 (default 3)Modern, good ratio/speed balance.
zlib0-9 (default 6)Medium speed, medium ratio.
lzma0-9 (default 6)High ratio, slow.
auto,lzma,6-Heuristic, test with lz4, only compress if beneficial.

Restic

Restic is a Go-based backup tool with mandatory client-side encryption (AES-256-CTR + Poly1305-AES) and native support for cloud backends.

Repository initialization

# Local
export RESTIC_REPOSITORY=/srv/restic-repo
export RESTIC_PASSWORD="your-password"
restic init

# S3
export RESTIC_REPOSITORY=s3:s3.amazonaws.com/bucket/prefix
export AWS_ACCESS_KEY_ID=...
export AWS_SECRET_ACCESS_KEY=...
restic init

# SFTP
restic -r sftp:user@host:/backup/restic init

Supported backends

BackendSyntax
Local/path/to/repo
S3 / S3-compatibles3:https://endpoint/bucket/prefix
Backblaze B2b2:bucketname/prefix
Azure Blobazure:container:/prefix
Google Cloud Storagegs:bucket-name:/prefix
SFTPsftp:user@host:/path
REST serverrest:https://host:8000/prefix

Full S3 workflow

#!/bin/bash
export RESTIC_REPOSITORY="s3:s3.amazonaws.com/my-bucket/restic"
export AWS_ACCESS_KEY_ID="..."
export AWS_SECRET_ACCESS_KEY="..."
export AWS_DEFAULT_REGION="us-east-1"
export RESTIC_PASSWORD="complex-passphrase"

# Backup with tags
restic backup /etc /home /var --tag daily --tag server1 \
  --exclude '*.tmp' --exclude '/home/*/.cache'

# Retention
restic forget --keep-daily 7 --keep-weekly 4 --keep-monthly 6 \
  --tag daily --prune

# Spot-check 5% of data
restic check --read-data-subset=5%

unset RESTIC_PASSWORD AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY

Restore and mount

# Restore latest snapshot
restic restore latest --target /tmp/restore

# Restore specific snapshot
restic restore 79766175 --target /tmp/restore

# Restore specific path
restic restore latest --path /home/user --target /tmp/restore

# Mount as FUSE
restic mount /mnt/restic

Tool comparison

FeaturersyncBorgRestic
DeduplicationNo (hardlinks with --link-dest)Yes (content-defined)Yes (content-defined)
EncryptionNo (SSH transport only)Yes (multiple modes)Yes (mandatory AES-256)
CompressionTransfer only (-z)Yes (lz4, zstd, zlib, lzma)Yes (built-in)
Cloud backendsNoSSH onlyS3, B2, Azure, GCS, SFTP, REST
FUSE mountNoYesYes
Retention policyManualborg prunerestic forget
VerificationChecksum onlyborg check --verify-datarestic check --read-data

Database backups

Never file-backup a running database

File-level backups of a running database are unsafe, the database may have uncommitted changes in memory, and tar/rsync cannot take an atomic snapshot. Use the database's native dump tool, or use a filesystem snapshot (LVM/Btrfs/ZFS) which is crash-consistent.

PostgreSQL

# Logical backup (custom format, recommended)
pg_dump -Fc -f mydb.dump mydb

# All databases + globals
pg_dumpall --globals-only > globals.sql
pg_dump -Fc -f mydb.dump mydb   # per database

# Point-in-time recovery: enable WAL archiving
# postgresql.conf:
#   wal_level = replica
#   archive_mode = on
#   archive_command = 'cp %p /wal_archive/%f'

# Base backup
pg_basebackup -h localhost -D /backup/base -U replicator -P -v -R

MySQL / MariaDB

# Logical backup (consistent for InnoDB)
mysqldump --single-transaction --routines --triggers mydb > mydb.sql

# All databases
mysqldump --single-transaction --all-databases > all.sql

# Physical backup with Percona XtraBackup
xtrabackup --backup --target-dir=/backup/full --user=root --password=pass
xtrabackup --prepare --target-dir=/backup/full

Snapshot-based backups

Filesystem snapshots enable near-zero-RPO backups with minimal performance impact.

LVM snapshot

lvcreate -L 10G -s -n db_snap /dev/vg00/lvol1
mount /dev/vg00/db_snap /mnt/snap
rsync -a /mnt/snap/ /backup/
umount /mnt/snap
lvremove /dev/vg00/db_snap

Btrfs snapshot + send

btrfs subvolume snapshot -r /data /.snapshots/2026-07-21
btrfs send /.snapshots/2026-07-21 | ssh backup "btrfs receive /backup"
# Incremental:
btrfs send -p /.snapshots/2026-07-20 /.snapshots/2026-07-21 \
  | ssh backup "btrfs receive /backup"

ZFS snapshot + send

zfs snapshot tank/data@2026-07-21
zfs send tank/data@2026-07-21 | ssh backup "zfs receive tank/data"
# Incremental:
zfs send -i tank/data@2026-07-20 tank/data@2026-07-21 \
  | ssh backup "zfs receive tank/data"

Backup encryption

Borg and Restic provide built-in authenticated encryption. For tools without encryption (rsync, tar), use age or gpg:

# age (modern, simple)
age-keygen -o key.txt
tar -czf - /data | age -r age1... -o backup.tar.gz.age
age -d -i key.txt backup.tar.gz.age | tar -xzf -

# gpg (asymmetric)
tar -czf - /data | gpg --encrypt --recipient user@example.com > backup.tar.gz.gpg

Backup verification

A backup you haven't tested is not a backup

Backups can complete successfully but be corrupted, incomplete, or unrestorable. Test restores regularly.

# Restic: basic check
restic check

# Restic: read all data (slow, comprehensive)
restic check --read-data

# Restic: spot-check 10%
restic check --read-data-subset=10%

# Borg: repository check
borg check /path/to/repo

# Borg: cryptographic data verification
borg check --verify-data /path/to/repo

Verification cadence

Check typeFrequency
Tool integrity checkDaily/weekly
Read-data verificationMonthly
Partial restore testMonthly
Full restore testQuarterly
DR drillAnnually

Automation

systemd timer for Restic

# /etc/systemd/system/restic-backup.timer
[Unit]
Description=Daily restic backup

[Timer]
OnCalendar=*-*-* 02:30:00
Persistent=true
RandomizedDelaySec=20min

[Install]
WantedBy=timers.target
# /etc/systemd/system/restic-backup.service
[Unit]
Description=Restic backup
After=network-online.target
Wants=network-online.target

[Service]
Type=oneshot
Environment=RESTIC_REPOSITORY=/backup/restic
Environment=RESTIC_PASSWORD_FILE=/etc/restic/password
ExecStart=/usr/bin/restic backup /etc /home /var
ExecStart=/usr/bin/restic forget --keep-daily 7 --keep-weekly 4 --prune
Nice=19
IOSchedulingClass=idle
systemctl daemon-reload
systemctl enable --now restic-backup.timer
systemctl list-timers

Offsite and immutability

S3 Object Lock (WORM)

# Enable Object Lock on bucket (at creation time)
aws s3api create-bucket --bucket my-backup \
  --object-lock-enabled-for-bucket

# Set retention on an object
aws s3api put-object-retention --bucket my-backup \
  --key backup.tar.gz \
  --retention 'Mode=COMPLIANCE,RetainUntilDate=2027-01-15T00:00:00Z'

Compliance mode prevents deletion by anyone, including root. Governance mode allows privileged users to change retention.

Air-gapped backups

For ransomware protection, maintain at least one copy that is physically or logically unreachable from the network:

Disaster recovery planning

Every production runbook should contain:

  1. Service impact: what breaks, who is affected, regulatory implications
  2. Recovery tier, RPO, RTO: Tier 1 (critical), Tier 2 (important), Tier 3 (optional)
  3. Roles: incident commander, recovery operator, validator, communications
  4. Triggers: conditions that activate the DR plan
  5. Dependency map: recovery order based on system dependencies
  6. Approved recovery sources: which backup, which snapshot, which offsite copy
  7. Explicit commands: exact commands with verification steps after each
  8. Credentials: where keys are stored, how to access them
  9. Validation checks: infrastructure, application, data, and business validation
  10. Communication plan: stakeholder notification and status updates
  11. Failback procedures: how to return to primary
  12. Evidence collection: record actual RPO/RTO achieved, lessons learned