Linux disk performance benchmarking

Storage benchmarking measures three fundamental metrics: IOPS (random access), bandwidth (sequential throughput), and latency (response time). The right tool and parameters depend on what you are measuring and why. fio is the standard for serious benchmarking; dd and hdparm serve quick checks.

IOPS vs bandwidth vs latency

fio (Flexible I/O Tester)

fio is the standard for storage benchmarking. It simulates configurable I/O workloads with precise control over block size, queue depth, I/O engine, and read/write patterns. Use direct=1 to bypass the page cache; filesystem caches like ZFS ARC will otherwise serve reads from RAM.

Command-line examples

# Random read IOPS (4K, queue depth 256, 4 jobs, 60s)
fio --name=randread --filename=/dev/sda --rw=randread --bs=4k \
    --direct=1 --ioengine=libaio --iodepth=256 --numjobs=4 \
    --time_based --runtime=60 --group_reporting

# Sequential write bandwidth (1M blocks)
fio --name=seqwrite --filename=/dev/sda --rw=write --bs=1M \
    --direct=1 --ioengine=libaio --iodepth=64 --size=5G \
    --numjobs=4 --group_reporting

# Mixed 70/30 read/write
fio --name=mixed --filename=/dev/sda --rw=randrw --rwmixread=70 \
    --bs=4k --direct=1 --ioengine=libaio --iodepth=128 \
    --size=10G --numjobs=4 --time_based --runtime=60 --group_reporting

Job file format

[global]
ioengine=libaio
direct=1
iodepth=16
bs=4k
size=1G
numjobs=4
time_based
runtime=60
group_reporting

[rand-read]
rw=randread
filename=/dev/sda

[seq-write]
rw=write
bs=1M
iodepth=64
filename=/dev/sda

Key parameters

Interpreting fio output

• IOPS: random access performance
• Bandwidth (MB/s or GB/s), sequential throughput
• Latency:
  • slat, submission latency (time to submit I/O)
  • clat, completion latency (time to complete)
  • lat, total latency (slat + clat)

dd (basic sequential test)

# Write 10 GB with direct I/O
dd if=/dev/zero of=testfile bs=1M count=10000 oflag=direct conv=fdatasync

# Read test with direct I/O
dd if=testfile of=/dev/null bs=1M iflag=direct

iostat (extended statistics)

# Extended stats every 1 second
iostat -x 1

# Every 5 seconds, 10 times, specific device
iostat -x 5 10 -p sda

# Human-readable
iostat -h 1

Identifying bottlenecks: high %util + high await = device saturated. High await with low %util may indicate controller or network bottleneck. That includes network-attached backends like iSCSI and NVMe-oF, where the fabric adds latency the disk never sees.

iotop (per-process I/O)

# Show only processes doing I/O
iotop -o

# Batch mode for logging
iotop -b -n 10 > iotop.log

# Accumulated I/O since start
iotop -a

Requires kernel options CONFIG_TASK_DELAY_ACCT, CONFIG_TASK_IO_ACCOUNTING, CONFIG_TASKSTATS.

Other tools

hdparm (quick read test)

# Buffered disk reads
hdparm -t /dev/sda

# Cache reads (shows RAM speed)
hdparm -T /dev/sda

# Direct I/O (bypass cache)
hdparm -t --direct /dev/sda

# Both
hdparm -tT /dev/sda

biolatency (eBPF, kernel latency histogram)

# Block I/O latency histogram
biolatency

# Per-device histograms
biolatency -D

# Include OS queued time
biolatency -Q

# 1-second summaries, 10 times
biolatency 1 10

ioping (latency test)

# Basic latency
ioping .

# Disk seek rate (IOPS)
ioping -R /dev/sda

# Sequential speed (bandwidth)
ioping -RL /dev/sda

# Direct I/O
ioping -D .

Best practices

1. Always bypass cache. Use direct=1 in fio or oflag=direct in dd. Without it you are benchmarking RAM, not storage.
2. Warm up: run tests multiple times, since SSDs may behave differently after initial writes.
3. Match block size to workload: 4k for databases, 64k-256k for virtualization, 1M for backups.
4. Use realistic queue depths. HDDs: 1-8, SATA SSDs: 32, NVMe: 128-256.
5. Multiple runs: run 3-5 times, take the average, discard outliers.
6. Watch for thermal throttling. NVMe drives can exceed 80°C under sustained load and throttle performance, so monitor temperature during long tests.
7. Test on the target device, not a file. Testing on a file in another filesystem adds overhead and may not reflect raw device performance.

Matching benchmarks to workloads

Run TRIM on a used SSD before benchmarking so garbage collection isn't mid-cycle. If the device sits behind a cache layer, measure under real load (see storage tiering and caching for cache effectiveness).