Linux I/O schedulers & blk-mq architecture
The Linux blk-mq (Multi-Queue Block I/O) subsystem arbitrates all read and write requests passing between filesystems (ext4, XFS, Btrfs) and underlying storage controllers. Selecting the correct I/O scheduler directly determines storage latency, IOPS ceilings, and system responsiveness under heavy concurrent loads.
Multi-Queue (blk-mq) architecture
Legacy Linux kernels (prior to blk-mq) funneled all block I/O through a single global request queue protected by a single spinlock. On modern NVMe drives delivering millions of IOPS across 64+ CPU cores, that global lock created catastrophic lock contention.
blk-mq solves this with a two-tiered queuing architecture:
- Software Staging Queues: Allocated per-CPU core or per-NUMA node. Applications submit I/O locally without cross-core lock synchronization.
- Hardware Dispatch Queues: Mapped directly to the physical submission queues of the storage hardware (e.g. 64-1024 queues on enterprise NVMe SSDs).
Scheduler comparison & selection matrix
| Scheduler | Primary Design Goal | Recommended Hardware / Workload |
|---|---|---|
none | Zero software overhead; direct hardware dispatch | PCIe / NVMe SSDs (Enterprise & Consumer), fast Virtual Disks (VirtIO-Blk with multi-queue enabled). |
mq-deadline | Strict read latency deadlines; starvation avoidance | SATA SSDs, SAS HBAs, Database servers (PostgreSQL/MySQL), shared hypervisors. |
bfq (Budget Fair Queueing) | Proportional bandwidth fairness & interactivity | Rotational HDDs, slow USB flash, interactive desktop workstations, media playback. |
kyber | Target latency SLA throttling | High-throughput cloud services, web server farms with strict p99 latency constraints on fast flash. |
Inspecting and setting schedulers
Check active scheduler
# Check active scheduler for NVMe drive (selected scheduler shown in brackets)
cat /sys/block/nvme0n1/queue/scheduler
# Output: [none] mq-deadline kyber bfq
# Check active scheduler for SATA drive
cat /sys/block/sda/queue/scheduler
# Output: none [mq-deadline] kyber bfqTemporarily switch scheduler
# Set NVMe device to none (instantaneous runtime switch)
echo none > /sys/block/nvme0n1/queue/scheduler
# Set SATA SSD device to mq-deadline
echo mq-deadline > /sys/block/sda/queue/schedulerPersistent configuration via udev rules
Kernel sysfs settings reset on reboot. Use persistent udev rules in /etc/udev/rules.d/ to automatically apply optimal schedulers based on disk characteristics:
# Create /etc/udev/rules.d/60-io-schedulers.rules
cat <<'EOF' | sudo tee /etc/udev/rules.d/60-io-schedulers.rules
# 1. Set 'none' for high-speed NVMe drives
ACTION=="add|change", KERNEL=="nvme[0-9]*n[0-9]*", ATTR{queue/scheduler}="none"
# 2. Set 'mq-deadline' for non-rotational SATA/SAS SSDs (rotational == 0)
ACTION=="add|change", KERNEL=="sd[a-z]|vd[a-z]", ATTR{queue/rotational}=="0", ATTR{queue/scheduler}="mq-deadline"
# 3. Set 'bfq' for mechanical rotational spinning disks (rotational == 1)
ACTION=="add|change", KERNEL=="sd[a-z]", ATTR{queue/rotational}=="1", ATTR{queue/scheduler}="bfq"
EOF
# Reload and apply udev rules immediately
sudo udevadm control --reload-rules
sudo udevadm trigger --type=devices --action=changeAdvanced block queue sysfs tunables
Additional queue parameters located in /sys/block/<dev>/queue/ can be tuned for specialized workloads:
| Sysfs Path | Default | Tuned Value | Impact & Rationale |
|---|---|---|---|
queue/nr_requests | 128 | 512-1024 | Increases maximum queued I/O requests before blocking applications. Recommended for heavy database write batches. |
queue/read_ahead_kb | 128 | 256-2048 | Increases kernel pre-read buffer for sequential streaming reads. Boosts sequential throughput on large file analytics. |
queue/nomerges | 0 (Merges on) | 2 (Complete no-merge) | Disables software I/O request coalescing. Reduces CPU overhead on high-speed NVMe drives handling pure random 4K I/O. |
queue/add_random | 0 or 1 | 0 | Disables feeding disk I/O entropy to kernel random pool, eliminating CPU interrupt overhead on high-IOPS devices. |
Frequently asked questions
What is blk-mq (Multi-Queue Block Layer)?
blk-mq is the modern Linux kernel block I/O layer introduced to handle high-IOPS storage. It replaces the legacy single global request lock with two-stage queues: per-CPU Software Staging Queues mapped to multiple Hardware Dispatch Queues on the NVMe/SAS controller, eliminating CPU lock contention on multicore systems.
Why is none the recommended scheduler for NVMe drives?
Modern NVMe drives support up to 65,535 parallel hardware submission queues directly in controller silicon. Software I/O scheduling introduces CPU cache thrashing and context switching overhead without performance benefit. Setting none (the multiqueue successor to the legacy noop scheduler) passes I/O requests directly to the NVMe driver with zero software queue overhead.
When should mq-deadline be used instead of none or bfq?
mq-deadline is ideal for SATA SSDs, SAS arrays, database engines, and virtual guest disks. It guarantees strict latency bounds by assigning deadlines to read and write requests (favoring reads to prevent application thread starvation) while maintaining high throughput.
What is BFQ (Budget Fair Queueing) designed for?
BFQ allocates disk bandwidth proportionally according to process priority. It is best suited for rotational HDDs, slow USB drives, and desktop interactive systems where preventing a background copy or torrent from freezing the GUI or media playback is critical.
What is Facebook's Kyber scheduler?
Kyber is a latency-targeted I/O scheduler developed by Facebook (Meta) for fast flash devices. It actively monitors request latencies against target thresholds (e.g. 2ms for reads, 10ms for writes) and automatically throttles queue depth to meet latency SLAs under heavy burst load.