DM-Multipath: SAN path redundancy & failover

DM-Multipath provides redundant I/O paths between servers and SAN storage. It aggregates multiple physical paths (cables, switches, HBAs) into a single logical device, providing both redundancy (failover) and performance (load balancing).

Active/active vs active/passive

Components

WWID (World Wide Identifier)

The WWID is a globally unique, unchanging identifier for storage devices, derived from SCSI Inquiry data (VPD page 0x83 or Unit Serial Number page 0x80). Multipath uses the WWID to detect multiple paths to the same device. Default device naming uses WWID: /dev/mapper/WWID.

Configuration

/etc/multipath.conf sections

Sections are evaluated in priority order: multipaths > overrides > devices > defaults.

defaults          # General default settings
blacklist         # Devices to exclude from multipath
blacklist_exceptions  # Devices to include despite blacklist
multipaths        # Settings for individual multipath devices (by WWID)
devices           # Settings for specific storage controllers
overrides         # Settings that override device-specific settings

Key parameters

Example configuration

defaults {
    user_friendly_names yes
    path_grouping_policy multibus
    path_selector "service-time 0"
    path_checker tur
    polling_interval 5
    failback immediate
    no_path_retry 12
    find_multipaths yes
}

blacklist {
    devnode "^sda"
    devnode "^sr[0-9]+"
    wwid "36001405.*"
}

devices {
    device {
        vendor "NETAPP"
        product "LUN.*"
        path_grouping_policy group_by_prio
        prio ontap
        path_checker tur
    }
}

multipaths {
    multipath {
        wwid 3600508b4000156d70001200000b0000
        alias yellow
        path_grouping_policy multibus
        failback manual
    }
}

Commands

multipath command

# List topology (sysfs + device mapper)
multipath -l

# List topology (all available info)
multipath -ll

# Verbose
multipath -v2 -l
multipath -v3 -l

# Flush (remove) a specific device
multipath -f /dev/mapper/mpatha

# Flush all unused devices
multipath -F

# Reload configuration
multipath -r

# Show currently used configuration
multipath -t

# Show built-in configuration template
multipath -T

# Add WWID to wwids file
multipath -a /dev/sdb

# Check if device should be a multipath path
multipath -c /dev/sdb

multipathd interactive commands

# Enter interactive mode
multipathd -k

# Show paths being monitored
multipathd -k 'show paths'

# Show multipath devices
multipathd -k 'show maps'

# Show topology (same as multipath -ll)
multipathd -k 'show topology'

# Show configuration
multipathd -k 'show config'

# Add a path
multipathd -k 'add path sda'

# Remove a path
multipathd -k 'remove path sda'

# Reinstate a failed path
multipathd -k 'reinstate path sda'

# Disable queueing for specific device
multipathd -k 'disablequeueing map mpathc'

# Restore queueing for all devices
multipathd -k 'restorequeueing maps'

Rescanning SCSI paths

# Rescan specific SCSI host (channel, target, LUN)
echo "c t l" > /sys/class/scsi_host/hosth/scan

# Rescan all channels, targets, and LUNs on a host
echo "- - -" > /sys/class/scsi_host/host0/scan

# Rescan all SCSI hosts
for host in /sys/class/scsi_host/host*; do
    echo "- - -" > "$host/scan"
done

# Issue LIP (Loop Initialization Protocol) for FC hosts
for host in /sys/class/fc_host/host*; do
    echo "1" > "$host/issue_lip"
done

# After rescanning, reload multipath
multipath -r

Path states

Path status in multipath -ll output

3600508b4000156d70001200000b0000
`-+- policy='round-robin 0' prio=1 status=active
  `- 6:0:0:0 sdb 8:16  active ready  running
  `- 7:0:0:0 sdf 8:80  active ghost running

Ghost paths indicate the device is in standby or passive state. These return valid responses to SCSI commands but fail read/write I/O. Normal for ALUA devices with Standby state.

Multipath and LVM

When using multipath, configure LVM to only use multipath devices to avoid duplicate PV warnings. Each path to a LUN appears as a separate SCSI device with the same LVM metadata.

# /etc/lvm/lvm.conf
devices {
    # Accept only multipath devices, reject raw sd devices
    filter = [ "a|/dev/mapper/.*|", "r|/dev/sd.*|", "r|/dev/disk/by-path/.*|", "a|.*|" ]
}

# Test the filter
lvmconfig --type diff
lvmdiskscan

# Create PV on multipath device
pvcreate /dev/mapper/mpatha
vgcreate myvg /dev/mapper/mpatha
lvcreate -L 10G -n mylv myvg

Multipath and filesystems

Device naming options

# WWID (recommended for clusters)
/dev/mapper/3600508b4000156d70001200000b0000

# User-friendly name
/dev/mapper/mpatha

# Custom alias
/dev/mapper/mydata

# /dev/disk/by-id
/dev/disk/by-id/wwid-3600508b4000156d70001200000b0000

/etc/fstab

# Using user-friendly name
/dev/mapper/mpatha    /data    xfs    defaults,_netdev    0 0

# Using WWID
/dev/mapper/3600508b4000156d70001200000b0000    /data    xfs    defaults,_netdev    0 0

# Using custom alias
/dev/mapper/mydata    /data    xfs    defaults,_netdev    0 0

ALUA (Asymmetric Logical Unit Access)

Troubleshooting

queue_if_no_path I/O hangs

If configured with features "1 queue_if_no_path", processes issuing I/O hang until paths are restored. Use no_path_retry N instead:

defaults {
    no_path_retry 12  # Retry 12 times before failing
}

# Runtime fix
dmsetup message mpathc 0 "fail_if_no_path"
# Or via multipathd
multipathd -k 'disablequeueing map mpathc'

Flapping paths

# Increase polling interval
defaults {
    polling_interval 10
    max_polling_interval 40
}

# Use appropriate path checker
defaults {
    path_checker tur  # For active/passive arrays
}

Diagnostic commands

# Show detailed topology
multipath -ll

# Show path checker states
multipathd -k 'show paths'

# Check daemon status
systemctl status multipathd

# Check logs
journalctl -u multipathd -f
dmesg | grep -i multipath

# Test path checker
multipathd -k 'show paths format "%d %s %c %t"'

Test failover

# Simulate path failure
echo offline > /sys/block/sdb/device/state

# Verify failover
multipath -ll

# Restore path
echo running > /sys/block/sdb/device/state

# Verify recovery
multipath -ll

Best practices

1. Blacklist non-multipath devices to prevent local disks from being multipathed.
2. Use WWID-based naming for clusters: it ensures consistent device names across nodes.
3. Configure no_path_retry appropriately, fail for critical data, queue or N for non-critical.
4. Use find_multipaths so only devices with 2+ paths get multipathed.
5. Set the appropriate path checker. tur for active/passive arrays, directio for active/active.
6. Test failover by simulating path failures before production deployment.
7. Monitor multipathd: ensure the daemon is running and check logs regularly.
8. Use vendor-specific settings. Consult vendor documentation for recommended parameters.
9. Configure the LVM filter to only use /dev/mapper/* devices, avoiding duplicate PVs.
10. Document the configuration and keep /etc/multipath.conf under version control.