OpenZFS on Linux: pools, datasets, snapshots & replication

OpenZFS is a combined filesystem and volume manager that provides end-to-end checksumming, copy-on-write snapshots, native compression, deduplication, and replication. It is licensed under CDDL, which is incompatible with the Linux GPLv2. This is why it cannot be merged into the mainline kernel and ships as an out-of-tree DKMS module.

Architecture

Transaction groups (txg)

Writes are batched into transaction groups. Three txgs are in flight at once: one open (accepting writes), one quiescing, one syncing. A group commits atomically, either all writes in the group land or none do. On crash, the pool returns to the last committed state.

Merkle tree of checksums

Every block pointer records its child's location, size, and checksum. Checksums are stored in the parent, not next to the block. A damaged block cannot vouch for itself. This provides end-to-end integrity: corruption is detected on read, not just on scrub.

Pool creation

# Two-way mirror (always use ashift=12 for modern drives)
zpool create -o ashift=12 tank mirror sda sdb

# Stripe of mirrors (recommended over RAIDZ for most workloads)
zpool create -o ashift=12 tank \
  mirror sda sdb \
  mirror sdc sdd \
  mirror sde sdf

# RAIDZ1 (single parity)
zpool create -o ashift=12 tank raidz sda sdb sdc sdd sde sdf

# RAIDZ2 (double parity)
zpool create -o ashift=12 tank raidz2 sda sdb sdc sdd sde sdf sdg

# RAIDZ3 (triple parity)
zpool create -o ashift=12 tank raidz3 sda sdb sdc sdd sde sdf sdg sdh sdi

Online RAIDZ expansion (OpenZFS 2.4.0+)

# Widen a RAIDZ vdev by attaching a device
zpool attach tank raidz2-0 sdX

Fault tolerance is unchanged (RAIDZ2 stays RAIDZ2). Blocks written before expansion keep the original data-to-parity ratio; only newly written blocks use the wider ratio.

Dataset management

# Create a dataset
zfs create tank/home
zfs create tank/home/alice

# List datasets
zfs list
zfs list -r tank

# Set properties
zfs set compression=lz4 tank/home
zfs set recordsize=16K tank/database
zfs set atime=off tank
zfs set mountpoint=/export/data tank/data

# Get properties
zfs get all tank/home
zfs get -r compression,recordsize tank

# Destroy
zfs destroy tank/home/alice

Key properties

Volumes (zvols)

# Create a 40 GiB thick volume
zfs create -V 40G tank/vm/disk0

# Create a sparse (thin) volume
zfs create -s -V 100G tank/vm/thin

# The block device appears at:
ls -l /dev/zvol/tank/vm/disk0

zvols are used for iSCSI targets, VM disks, and swap. volblocksize is set at creation (default 16 KiB since OpenZFS 2.2) and cannot be changed.

Snapshots

# Create a snapshot
zfs snapshot tank/home@2026-07-21

# Recursive snapshot
zfs snapshot -r tank/home@2026-07-21

# List snapshots
zfs list -t snapshot

# Destroy
zfs destroy tank/home@2026-07-21
zfs destroy -r tank/home@2026-07-21

Snapshots are instant and initially consume zero space, they share all blocks with the source via CoW. As the source diverges, the snapshot retains the old blocks. Access snapshots under .zfs/snapshot/ in the dataset root (visibility controlled by the snapdir property).

Clones

# Create a writable clone from a snapshot
zfs clone tank/project/production@today tank/project/beta

# Promote clone (swap parent/child relationship)
zfs promote tank/project/beta

A clone is a writable dataset whose initial contents are a snapshot. It diverges as written to. The origin snapshot cannot be destroyed while clones exist.

Send / receive (replication)

# Full send
zfs send tank/data@snap | ssh backup zfs receive tank/data

# Incremental send
zfs send -i tank/data@yesterday tank/data@today | ssh backup zfs receive tank/data

# Replication stream (recursive, preserves properties)
zfs send -R tank/home@snap | ssh backup "zfs receive -u tank/home"

# Raw send (encrypted datasets stay encrypted in transit)
zfs send -w tank/secret@snap | ssh backup zfs receive tank/secret

Bookmarks

# Create a bookmark (survives snapshot destruction)
zfs bookmark tank/data@2026-07-21 tank/data#2026-07-21

# Use bookmark as incremental source after snapshot is destroyed
zfs send -i tank/data#2026-07-21 tank/data@2026-07-22 | ssh backup zfs receive tank/data

Scrubbing

# Start scrub
zpool scrub tank

# Pause / stop
zpool scrub -p tank    # pause
zpool scrub -s tank    # stop

# Wait for completion
zpool scrub -w tank

# View status and error details
zpool status -v tank

Scrub walks all data and verifies every checksum. Where redundancy exists (mirror, RAIDZ, copies>1), it repairs corrupted blocks. Scrub and resilver are mutually exclusive, only one runs at a time. Schedule monthly scrubs for most pools.

Compression

zfs set compression=lz4 tank
zfs set compression=zstd-5 tank/archive

# Check achieved ratio
zfs get compressratio tank

Deduplication

# Enable dedup (think twice)
zfs set dedup=on tank/data

# Fast Dedup (OpenZFS 2.4.0+) with table quota
zpool set dedup_table_quota=100G tank
zpool ddtprune tank

# Dedicated dedup vdev on fast SSD
zpool add tank dedup mirror nvme0n1 nvme1n1

Only use dedup for workloads with genuine, substantial duplication: VM images from a common base, or backup targets receiving many similar systems.

ARC, L2ARC, and SLOG

Encryption

# Create an encrypted dataset with a passphrase
zfs create -o encryption=aes-256-gcm -o keyformat=passphrase tank/secret

# Load keys at boot or after import
zfs load-key tank/secret
zfs load-key -a          # all encryption roots

# Unload key (dataset becomes inaccessible)
zfs unload-key tank/secret

# Change key
zfs change-key tank/secret

Raw sends (zfs send -w) keep data encrypted in transit. The receiving machine never needs the keys.

Device management

# Add a vdev
zpool add tank mirror sdc sdd

# Remove a top-level vdev (mirror or cache, not RAIDZ)
zpool remove tank mirror-2

# Replace a failed drive
zpool replace tank sda sdX

# Attach to convert a single disk to a mirror
zpool attach tank sda sdb

# Detach from a mirror
zpool detach tank sdb

# Offline / online
zpool offline tank sda
zpool online tank sda
zpool online -e tank sda    # expand to use all available space

Resilvering

Resilver rebuilds only the data ZFS knows is out of date (after replace, attach, or online), not the entire disk. Sequential reconstruction (zpool replace -s) rebuilds sequentially for faster redundancy restoration. Checksum verification behavior during sequential rebuild varies by vdev type: for mirrors, a scrub is automatically started afterward to verify; for dRAID the verification is integrated.

systemd integration

OpenZFS 2.4.0 features

• Fast Dedup: new dedup table format with dedup_table_quota and zpool ddtprune
• Direct I/O: zfs set direct=standard|always|disabled controls O_DIRECT behavior
• Online RAIDZ expansion: zpool attach pool raidzN-0 sdX widens an existing RAIDZ vdev

Best practices

1. Always use ashift=12 at pool creation.
2. Enable compression=lz4 on all datasets. It is a net performance win.
3. Do not use dedup unless you have a specific, measured reason and sufficient RAM.
4. Prefer mirrors over RAIDZ1: better random IOPS, faster rebuild, more predictable performance.
5. Use whole disks, not partitions. ZFS manages its own layout.
6. Set recordsize to match your workload (16K for databases, 128K default).
7. Set atime=off to eliminate a write for every read.
8. Test your backups, verify zfs send | zfs receive restores regularly.
9. Schedule monthly scrubs and monitor zpool status.
10. Use ECC RAM when possible. ZFS trusts RAM for checksums.