Linux disk performance benchmarking

Storage benchmarking measures three fundamental metrics: IOPS (random access), bandwidth (sequential throughput), and latency (response time). The right tool and parameters depend on what you are trying to measure and why.

IOPS vs bandwidth vs latency

fio (Flexible I/O Tester)

fio is the standard for storage benchmarking. It simulates configurable I/O workloads with precise control over block size, queue depth, I/O engine, and read/write patterns.

Command-line examples

# Random read IOPS (4K, queue depth 256, 4 jobs, 60s)
fio --name=randread --filename=/dev/sda --rw=randread --bs=4k \
    --direct=1 --ioengine=libaio --iodepth=256 --numjobs=4 \
    --time_based --runtime=60 --group_reporting

# Sequential write bandwidth (1M blocks)
fio --name=seqwrite --filename=/dev/sda --rw=write --bs=1M \
    --direct=1 --ioengine=libaio --iodepth=64 --size=5G \
    --numjobs=4 --group_reporting

# Mixed 70/30 read/write
fio --name=mixed --filename=/dev/sda --rw=randrw --rwmixread=70 \
    --bs=4k --direct=1 --ioengine=libaio --iodepth=128 \
    --size=10G --numjobs=4 --time_based --runtime=60 --group_reporting

Job file format

[global]
ioengine=libaio
direct=1
iodepth=16
bs=4k
size=1G
numjobs=4
time_based
runtime=60
group_reporting

[rand-read]
rw=randread
filename=/dev/sda

[seq-write]
rw=write
bs=1M
iodepth=64
filename=/dev/sda

Key parameters

Interpreting fio output

• IOPS: random access performance
• Bandwidth (MB/s or GB/s), sequential throughput
• Latency:
  • slat, submission latency (time to submit I/O)
  • clat, completion latency (time to complete)
  • lat, total latency (slat + clat)

dd (basic sequential test)

# Write 10 GB with direct I/O
dd if=/dev/zero of=testfile bs=1M count=10000 oflag=direct conv=fdatasync

# Read test with direct I/O
dd if=testfile of=/dev/null bs=1M iflag=direct

iostat (extended statistics)

# Extended stats every 1 second
iostat -x 1

# Every 5 seconds, 10 times, specific device
iostat -x 5 10 -p sda

# Human-readable
iostat -h 1

Identifying bottlenecks: high %util + high await = device saturated. High await with low %util may indicate controller or network bottleneck.

iotop (per-process I/O)

# Show only processes doing I/O
iotop -o

# Batch mode for logging
iotop -b -n 10 > iotop.log

# Accumulated I/O since start
iotop -a

Requires kernel options CONFIG_TASK_DELAY_ACCT, CONFIG_TASK_IO_ACCOUNTING, CONFIG_TASKSTATS.

Other tools

hdparm (quick read test)

# Buffered disk reads
hdparm -t /dev/sda

# Cache reads (shows RAM speed)
hdparm -T /dev/sda

# Direct I/O (bypass cache)
hdparm -t --direct /dev/sda

# Both
hdparm -tT /dev/sda

biolatency (eBPF, kernel latency histogram)

# Block I/O latency histogram
biolatency

# Per-device histograms
biolatency -D

# Include OS queued time
biolatency -Q

# 1-second summaries, 10 times
biolatency 1 10

ioping (latency test)

# Basic latency
ioping .

# Disk seek rate (IOPS)
ioping -R /dev/sda

# Sequential speed (bandwidth)
ioping -RL /dev/sda

# Direct I/O
ioping -D .

Best practices

1. Always bypass cache. Use direct=1 in fio or oflag=direct in dd. Without it you are benchmarking RAM, not storage.
2. Warm up: run tests multiple times, since SSDs may behave differently after initial writes.
3. Match block size to workload: 4k for databases, 64k-256k for virtualization, 1M for backups.
4. Use realistic queue depths. HDDs: 1-8, SATA SSDs: 32, NVMe: 128-256.
5. Multiple runs: run 3-5 times, take the average, discard outliers.
6. Watch for thermal throttling. NVMe drives can exceed 80°C under sustained load and throttle performance, so monitor temperature during long tests.
7. Test on the target device, not a file. Testing on a file in another filesystem adds overhead and may not reflect raw device performance.

Matching benchmarks to workloads