Linux Software RAID with mdadm

mdadm (Multiple Devices Admin) manages Linux software RAID through the kernel's MD driver. It creates virtual block devices (/dev/md0, /dev/md1, …) from two or more physical disks or partitions, providing redundancy and performance without dedicated hardware.

RAID levels

RAID 10 layouts

The --layout option controls how mirror copies are distributed:

• near (default): copies placed consecutively, classic RAID 1+0
• far: copies placed far apart, improved sequential read performance, slight write penalty
• offset: copies striped consecutively, similar read characteristics to far without as much seeking

Creating arrays

# RAID 0
mdadm --create /dev/md0 --level=0 --raid-devices=2 /dev/sdb1 /dev/sdc1

# RAID 1
mdadm --create /dev/md0 --level=1 --raid-devices=2 /dev/sdb1 /dev/sdc1

# RAID 5
mdadm --create /dev/md0 --level=5 --raid-devices=3 /dev/sdb1 /dev/sdc1 /dev/sdd1

# RAID 6
mdadm --create /dev/md0 --level=6 --raid-devices=4 /dev/sd[bcde]1

# RAID 10 with far layout
mdadm --create /dev/md0 --level=10 --layout=f2 --raid-devices=4 /dev/sd[bcde]1

# With hot spare
mdadm --create /dev/md0 --level=5 --raid-devices=3 --spare-devices=1 /dev/sd[bcde]1

# With custom chunk size (default 512 KiB)
mdadm --create /dev/md0 --level=5 --raid-devices=3 --chunk=64 /dev/sd[bcd]1

# With write-intent bitmap
mdadm --create /dev/md0 --level=5 --raid-devices=3 --bitmap=internal /dev/sd[bcd]1

Examining arrays

# Detailed view of an active array
mdadm --detail /dev/md0

# Brief view (suitable for mdadm.conf)
mdadm --detail --brief /dev/md0

# Scan all arrays and output ARRAY lines
mdadm --detail --scan

# Examine a component device's superblock
mdadm --examine /dev/sdb1

# Kernel view of active arrays
cat /proc/mdstat

Example /proc/mdstat output:

Personalities : [raid1] [raid5] [raid6] [raid10]
md0 : active raid5 sda1[0] sdb1[1] sdc1[2]
      209584128 blocks super 1.2 level 5, 512k chunk, algorithm 2 [3/3] [UUU]
      bitmap: 0/1 pages [0KB], 65536KB chunk
unused devices: <none>

The [UUU] field shows device health: U = up, _ = down/failed.

Monitoring

# Start monitor daemon (scans mdadm.conf)
mdadm --monitor --scan --daemonize

# With email alerts
mdadm --monitor --scan --mail=admin@example.com --daemonize

# Send a test alert
mdadm --monitor --scan --test

# Via systemd
systemctl enable --now mdmonitor.service

mdadm.conf

The configuration file (/etc/mdadm.conf or /etc/mdadm/mdadm.conf on Debian) tells the initramfs which arrays to assemble at boot. Generate it after every array change:

# Generate ARRAY lines from active arrays
mdadm --detail --scan >> /etc/mdadm.conf

# Update initramfs so boot can find the array
update-initramfs -u     # Debian/Ubuntu
dracut -f               # RHEL/Fedora

A typical file:

MAILADDR root
ARRAY /dev/md0 metadata=1.2 UUID=12345678:12345678:12345678:12345678 name=host:0

Adding, removing, and replacing disks

# Mark a disk as failed
mdadm /dev/md0 --fail /dev/sdb1

# Remove the failed disk
mdadm /dev/md0 --remove /dev/sdb1

# Add a new (or replacement) disk
mdadm /dev/md0 --add /dev/sdb1

# Re-add a previously removed disk
mdadm /dev/md0 --re-add /dev/sdb1

Complete disk replacement procedure

# 1. Identify the failed disk
mdadm --detail /dev/md0
cat /proc/mdstat
smartctl -a /dev/sdb | grep -i serial

# 2. Fail and remove
mdadm /dev/md0 --fail /dev/sdb1
mdadm /dev/md0 --remove /dev/sdb1

# 3. Physically replace the disk

# 4. Copy partition table from a healthy disk
sfdisk -d /dev/sdc | sfdisk /dev/sdb
# For GPT:
sgdisk -R /dev/sdb /dev/sdc
sgdisk -G /dev/sdb          # randomize the disk GUID

# 5. Add the new partition to the array
mdadm /dev/md0 --add /dev/sdb1

# 6. Watch the rebuild
watch cat /proc/mdstat

Growing arrays

# Add a disk, then grow to include it (RAID 5: 3 → 4 disks)
mdadm /dev/md0 --add /dev/sdd1
mdadm --grow /dev/md0 --raid-devices=4 --backup-file=/tmp/grow_backup

# RAID level migration (0 → 5)
mdadm --grow /dev/md0 --level=5 --raid-devices=3

# Change chunk size
mdadm --grow /dev/md0 --chunk=128

Stopping and assembling

# Stop an array
mdadm --stop /dev/md0

# Assemble from explicit devices
mdadm --assemble /dev/md0 /dev/sdb1 /dev/sdc1 /dev/sdd1

# Assemble all arrays from mdadm.conf
mdadm --assemble --scan

# Force assembly of a dirty/degraded array
mdadm --assemble --scan --force

Superblock versions

# Specify at creation time
mdadm --create /dev/md0 --level=1 --raid-devices=2 --metadata=1.0 /dev/sd[bc]1

# Zero out a superblock before reusing a disk
mdadm --zero-superblock /dev/sdb1

RAID 5/6 write hole

The write hole is a fundamental timing problem in parity RAID: updating a stripe requires writing both data and parity blocks, and these are separate writes. If power is lost between them, data and parity become inconsistent. On a degraded array, inconsistent parity cannot be recalculated, leading to silent data corruption during rebuild.

Mitigations

# Write-intent bitmap (tracks dirty chunks for fast recovery)
mdadm --grow /dev/md0 --bitmap=internal

# Write journal (dedicated journal device, RAID 4/5/6)
mdadm --create /dev/md0 --level=5 --raid-devices=3 \
  --write-journal=/dev/sdj1 /dev/sd[abc]1

# Consistency scrub
echo check > /sys/block/md0/md/sync_action
cat /sys/block/md0/md/mismatch_cnt
echo repair > /sys/block/md0/md/sync_action

Write-intent bitmaps

# Add internal bitmap
mdadm --grow /dev/md0 --bitmap=internal

# Add external bitmap on a separate file
mdadm --grow /dev/md0 --bitmap=/raidbitmap.bin

# Remove bitmap
mdadm --grow /dev/md0 --bitmap=none

Partitions vs whole disks

Recommendation: use partitions slightly smaller than full disk capacity (e.g., leave a 100 MB margin) to accommodate size variations between nominally identical disks from different manufacturers.

Degraded boot

If the root filesystem is on RAID and a disk is missing at boot, the initramfs may refuse to start a dirty degraded array. Force it from the emergency shell:

# In initramfs emergency shell
mdadm --assemble --scan --force
mdadm --run /dev/md0

Kernel parameter to allow dirty degraded starts:

md-mod.start_dirty_degraded=1

Command quick reference