Linux storage troubleshooting & incident runbooks

Storage incidents in production demand rapid, deterministic diagnosis. This hub contains battle-tested runbooks for the most critical Linux storage failures: hidden disk space leaks, emergency read-only remounts, unmount locks, and degraded RAID arrays.

Runbook 1: 100% Full disk with missing files (Unlinked open file leak)

Root Cause

A log rotation script or administrator deleted a massive file (e.g. rm /var/log/app.log), but the active application process (Java, Nginx, Python) still holds the open file descriptor. Linux keeps data blocks allocated until the process closes the descriptor.

Diagnosis & Non-Disruptive Remediation

# 1. Locate all open unlinked files greater than 100MB
sudo lsof +L1 / | sort -k 7 -n -r | head -n 10

# Example output:
# COMMAND   PID USER   FD   TYPE DEVICE   SIZE/OFF NLINK   NODE NAME
# java    14820 app     4w   REG  259,1 85899345920     0 786432 /var/log/app.log (deleted)

# 2. Reclaim the 80 GB instantly WITHOUT killing the application by truncating via procfs:
# Syntax: : > /proc/<PID>/fd/<FD>
sudo sh -c ': > /proc/14820/fd/4'

# 3. Verify disk space is immediately restored
df -h /

Runbook 2: Emergency read-only filesystem remounts

Diagnosis

# 1. Check kernel dmesg logs for hardware I/O or filesystem errors
sudo dmesg -T | grep -E -i "EXT4-fs error|XFS|I/O error|blk_update_request|aborted journal"

# 2. Check SMART health on underlying physical disk
sudo smartctl -H -A /dev/nvme0n1  # or /dev/sda

Remediation Procedure

# 1. If secondary data disk, unmount cleanly
sudo umount /srv/data

# 2. Run filesystem check and repair (for ext4):
sudo e2fsck -f -y /dev/sdb1

# For XFS filesystems:
sudo xfs_repair -v /dev/sdb1

# 3. Remount read-write and test
sudo mount -o remount,rw /srv/data

For detailed repair strategies (including superblock recovery and dirty journal replay), see the Filesystem Repair & e2fsck Reference and the SMART Disk Health Guide.

Runbook 3: "Device or resource busy" on unmount

Diagnosis & Resolution

# 1. Identify all processes with open files or active working directories
sudo fuser -vm /mnt/storage

# 2. Check open file descriptors specifically
sudo lsof +f -- /mnt/storage

# 3. Gracefully terminate blocking processes
sudo fuser -k -15 -m /mnt/storage

# 4. If processes refuse to die and immediate detach is required:
sudo umount -l /mnt/storage  # Lazy unmount

Runbook 4: Degraded software RAID & missing LVM recovery

Recovering a degraded mdadm RAID array

For full multi-disk recovery scenarios, see the mdadm Software RAID Guide:

# 1. Check array status
cat /proc/mdstat

# 2. Remove failed drive (e.g. /dev/sdb1)
sudo mdadm /dev/md0 --fail /dev/sdb1 --remove /dev/sdb1

# 3. Insert new drive, partition identically, and hot-add to rebuild
sudo mdadm /dev/md0 --add /dev/sdc1

# 4. Monitor rebuild progress
watch -n 1 'cat /proc/mdstat'

Recovering LVM Volume Group with missing physical volume

For thin pool and snapshot recovery, refer to the LVM Architecture Guide:

# 1. Inspect VG status
sudo vgdisplay --partial

# 2. If physical disk is permanently lost, purge missing PV from metadata:
sudo vgreduce --removemissing --force vg_storage

# 3. Restore VG metadata from automatic backup if accidentally corrupted:
sudo vgcfgrestore -f /etc/lvm/backup/vg_storage vg_storage