STORAGE ENGINEERING

Linux storage tiering & caching

Category: Block CachingTechnologies: bcache, lvmcache, dm-cache, dm-writecache

Block-level caching places a fast device (SSD/NVMe) in front of a slow device (HDD) to accelerate I/O. The cache holds hot data; cold data stays on the backing device. Linux offers four main implementations: bcache, dm-cache, lvmcache (which wraps dm-cache), and dm-writecache.

Cache modes

Mode	Write behavior	Safety	Performance
writethrough	Writes go to both cache and backing device	Safe (no data loss if cache fails)	Reads accelerated, writes not
writeback	Writes go to cache only, flushed to backing later	Risky (data loss if cache fails before flush)	Both reads and writes accelerated
writearound	Writes go directly to backing, bypassing cache	Safe	Reads of recently-written data not accelerated
none / passthrough	I/O passes through to backing device	Safe	No caching

bcache

bcache is a kernel block-layer cache that uses an SSD as a cache for one or more HDDs. It uses bucket-based allocation with a B+ tree index to track cached extents, and supports writeback, writethrough, writearound, and none modes.

Setup

# 1. Format the SSD as a cache device (bucket size should match SSD erase block)
make-bcache -C -b 512K /dev/nvme0n1p1

# 2. Format the HDD as a backing device
make-bcache -B /dev/sdb

# 3. Register devices with the kernel (if not auto-registered by udev)
echo /dev/nvme0n1p1 > /sys/fs/bcache/register
echo /dev/sdb > /sys/fs/bcache/register

# 4. Attach backing device to cache set (UUID from /sys/fs/bcache/)
ls /sys/fs/bcache/
echo <UUID> > /sys/block/bcache0/bcache/attach

# 5. Create filesystem on the bcache device
mkfs.ext4 /dev/bcache0
mount /dev/bcache0 /mnt/data

Cache mode

# View current mode (brackets indicate active)
cat /sys/block/bcache0/bcache/cache_mode

# Set mode at runtime
echo writeback > /sys/block/bcache0/bcache/cache_mode
echo writethrough > /sys/block/bcache0/bcache/cache_mode
echo writearound > /sys/block/bcache0/bcache/cache_mode
echo none > /sys/block/bcache0/bcache/cache_mode

Key tunables

Sysfs path	Default	Effect
`cache_mode`	writethrough	Cache policy
`writeback_percent`	10	Target dirty cache percentage before throttling writes
`writeback_delay`	30	Seconds to delay writeback after last write
`sequential_cutoff`	4M	Sequential I/O threshold to bypass cache (0 = cache everything)
`congested_read_threshold_us`	0	Latency threshold to bypass cache on reads
`dirty_data`	-	Read-only: amount of unflushed dirty data

Sequential bypass

bcache detects sequential I/O and bypasses the cache to avoid evicting useful random data. Disable it if your workload benefits from caching large files:

# Disable sequential bypass (cache everything)
echo 0 > /sys/block/bcache0/bcache/sequential_cutoff

# Restore default (4M)
echo 4M > /sys/block/bcache0/bcache/sequential_cutoff

Stopping and detaching

# Detach cache (revert to passthrough)
echo 1 > /sys/block/bcache0/bcache/detach

# Stop a backing device
echo 1 > /sys/block/bcache0/bcache/stop

# Unregister a cache device
echo 1 > /sys/fs/bcache/<UUID>/unregister

Replacing a cache device

# 1. Switch to writethrough to ensure no dirty data
echo writethrough > /sys/block/bcache0/bcache/cache_mode

# 2. Wait for dirty data to flush
watch cat /sys/block/bcache0/bcache/dirty_data

# 3. Detach
echo 1 > /sys/block/bcache0/bcache/detach

# 4. Register new cache device
make-bcache -C -b 512K /dev/new_nvme
echo /dev/new_nvme > /sys/fs/bcache/register

# 5. Attach backing device to new cache set
echo <NEW_UUID> > /sys/block/bcache0/bcache/attach

dm-cache (device-mapper cache)

dm-cache is the kernel's device-mapper caching target. It is rarely used directly, lvmcache wraps it with a friendlier interface. dm-cache supports writethrough, writeback, and passthrough modes, and uses the smq (Stochastic Multi-Queue) policy by default.

lvmcache

lvmcache is the LVM interface to dm-cache. It creates a cache pool (metadata LV + cache data LV) on a fast device and attaches it to an existing LV on a slow device.

Setup with cache volume (simpler, recommended)

# Create cache volume on the fast device
lvcreate -L 20G -n fast_vol myvg /dev/nvme0n1p1

# Attach to existing LV (writeback mode)
lvconvert --type cache --cachevol myvg/fast_vol \
  --cachemode writeback myvg/main_lv

Setup with cache pool (traditional, more control)

# Create cache data and metadata LVs
lvcreate -L 20G -n cache_data myvg /dev/nvme0n1p1
lvcreate -L 1G -n cache_meta myvg /dev/nvme0n1p1

# Convert to cache pool
lvconvert --type cache-pool --poolmetadata myvg/cache_meta \
  myvg/cache_data

# Attach cache pool to origin LV
lvconvert --type cache --cachepool myvg/cache_pool \
  --cachemode writeback myvg/main_lv

One-step creation

lvcreate --type cache -L 100G -n cached_lv \
  --cachedevice /dev/nvme0n1p1 --cachesize 20G \
  --cachemode writeback myvg /dev/sdb1

Cache modes

# Set at creation
lvconvert --type cache --cachepool myvg/cache_pool \
  --cachemode writeback myvg/main_lv

# Change mode of existing cache
lvconvert --cachemode writethrough myvg/main_lv

Monitoring hit rate

# Show cache statistics
lvs -o lv_name,cache_mode,cache_policy,cache_read_hits,cache_read_misses, \
  cache_write_hits,cache_write_misses myvg/main_lv

# Cache utilization
lvs -o lv_name,data_percent myvg/main_lv

# Raw dm-cache status
dmsetup status myvg-main_lv

Removing cache

# Flush dirty data and detach (safe for writeback)
lvconvert --splitcache myvg/main_lv

# Alternative: uncache
lvconvert --uncache myvg/main_lv

dm-writecache (LVM writecache)

dm-writecache is a simpler writeback-only cache designed specifically for SSD/NVMe write acceleration. Unlike dm-cache, it does not cache reads. It only buffers writes and flushes them to the backing device asynchronously.

# Create deactivated writecache LV on SSD
lvcreate -an -L 20G -n writecache myvg /dev/nvme0n1p1

# Unmount and deactivate origin LV
umount /data
lvchange -an myvg/main_lv

# Attach writecache
lvconvert --type writecache --cachevol myvg/writecache myvg/main_lv

# Reactivate and mount
lvchange -ay myvg/main_lv
mount /data

dm-writecache is best for write-heavy workloads on HDDs where read caching is less important. For mixed read/write workloads, dm-cache (lvmcache) is usually better.

Tool comparison

Feature	bcache	lvmcache (dm-cache)	dm-writecache
Read caching	Yes	Yes	No
Write caching	Yes (writeback)	Yes (writeback)	Yes (writeback only)
Writethrough	Yes	Yes	No
Writearound	Yes	No	No
Management	sysfs + make-bcache	LVM commands	LVM commands
Works with LVM	Can put bcache under LVM	Native	Native
Online attach/detach	Yes	Yes	No (requires offline)
Best for	Max performance, non-LVM	General use, LVM environments	Write-intensive workloads

ZFS ARC/L2ARC/SLOG (comparison)

ZFS provides its own caching layers, which are filesystem-level rather than block-level:

ARC (Adaptive Replacement Cache): in-memory read cache (RAM). Most effective layer; add RAM before considering L2ARC.
L2ARC (Level 2 ARC): optional second-level read cache on SSD. Helps when working set exceeds RAM. Does nothing for writes. Costs RAM for headers.
SLOG (ZIL): write accelerator for synchronous writes only. Not a cache - it's a log device for crash recovery.

Best practices

Size cache at 5-10% of backing device for most workloads; 10-20% for write-intensive.
Use NVMe/SSD for cache. HDD cache provides minimal benefit.
Use enterprise SSDs for writeback mode: consumer SSDs can lose data on power failure.
Start with writethrough until you trust the setup, then switch to writeback.
Monitor hit ratios: if below 50%, the cache may not be helping.
Have a recovery plan. Know how to recover from cache failure before you need to.
Don't cache backups. Backup workloads are typically sequential and won't benefit.