OverlayFS & container storage

OverlayFS is a union filesystem that merges multiple directories into a single unified view. It is the foundation of modern container storage, enabling efficient layering where read-only base images are combined with writable container layers.

OverlayFS architecture

OverlayFS combines a lowerdir (read-only) and an upperdir (writable) into a merged view. A workdir is required for atomic operations when an upperdir is present.

Basic mount

# Single lower layer with writable upper
mount -t overlay overlay \
  -olowerdir=/lower,upperdir=/upper,workdir=/work \
  /merged

# Multiple lower layers (read-only)
mount -t overlay overlay \
  -olowerdir=/lower1:/lower2:/lower3 \
  /merged

# Read-only overlay (no upperdir/workdir)
mount -t overlay overlay \
  -olowerdir=/lower1:/lower2 \
  /merged

fstab entry

overlay  /merged  overlay  lowerdir=/lower,upperdir=/var/lib/overlay/upper,workdir=/var/lib/overlay/work  0 0

Multiple lower layers

Multiple lower directories use colon separators. Stacking order is right-to-left: the rightmost directory is the bottom layer, the leftmost is the top (closest to upperdir).

Whiteouts and opaque directories

To support deletion without modifying the lower filesystem, OverlayFS uses whiteouts and opaque directories stored in the upper layer.

• Whiteout: marks a file as deleted. Created as a character device with 0/0 device number, or a zero-size regular file with the xattr trusted.overlay.whiteout.
• Opaque directory: hides the contents of the same-named directory in lower layers. Set with trusted.overlay.opaque xattr to "y".

# Mark a directory as opaque
setfattr -n trusted.overlay.opaque -v "y" /upper/mydir

Copy-up behavior

When a file from the lower layer is accessed in a way that requires write access (opening for write, changing metadata, creating a hardlink), OverlayFS performs a copy_up operation to copy the file to the upper layer.

1. Ensure the containing directory exists in upperdir (create parents if needed)
2. Create the object with the same metadata (owner, mode, mtime, symlink target)
3. If a file, copy data from lower to upper (may use clone_file_range if same filesystem)
4. Copy extended attributes from lower to upper
5. For directories, mark as opaque if needed

OverlayFS mount options

Kernel configuration

OverlayFS limitations

• Rename across layers: without redirect_dir=on, renaming a directory from a lower layer fails with EXDEV. mv falls back to recursive copy-delete.
• Device nodes: limited support in lower layers. Container runtimes handle devices via cgroups, not OverlayFS.
• mmap consistency: if a lower file is memory-mapped and then copied up, the mapping may see stale data. Use read() for security-sensitive scenarios.
• Backing filesystem: upperdir must support xattrs and d_type in readdir. XFS must be formatted with ftype=1. NFS is not suitable as an upper filesystem.

Docker storage drivers

The overlay2 driver is the default and recommended driver on modern Linux systems. It uses the kernel's OverlayFS implementation and stores data in /var/lib/docker/overlay2/.

Directory structure

/var/lib/docker/overlay2/
├── l/                          # Shortened layer identifiers (symlinks)
│   ├── ABCDEF... -> ../abcdef...
│   └── ...
├── abcdef.../                   # Layer directory
│   ├── diff/                    # Layer contents (actual files)
│   ├── link                     # Shortened identifier name
│   ├── lower                    # References to parent layers (colon-separated)
│   ├── merged/                  # Unified view (mount point)
│   └── work/                    # OverlayFS work directory
└── ...

Checking storage driver

docker info
...
Storage Driver: overlay2
 Backing Filesystem: xfs
 Supports d_type: true
 Native Overlay Diff: true
 Using metacopy: false

Prerequisites

• Linux kernel 4.0+ or RHEL/CentOS 3.10.0-514+
• XFS backing filesystem with ftype=1 (check with xfs_info)
• ext4 is also supported

# Verify XFS ftype
xfs_info /var/lib/docker | grep ftype

# Format XFS with ftype=1 if needed
mkfs.xfs -n ftype=1 /dev/sdb1

Configuring overlay2

# /etc/docker/daemon.json
{
  "storage-driver": "overlay2"
}

sudo systemctl restart docker
docker info | grep "Storage Driver"

Legacy storage drivers

Podman / containerd / CRI-O storage

Podman, containerd, and CRI-O use the containers/storage library. By default, data lives in /var/lib/containers/storage (root) or $HOME/.local/share/containers/storage (rootless).

# /etc/containers/storage.conf
[storage]
driver = "overlay"
runroot = "/run/containers/storage"
graphroot = "/var/lib/containers/storage"

[storage.options]
mount_program = "/usr/bin/fuse-overlayfs"  # For rootless on older kernels
mountopt = "nodev"

Checking storage driver

podman info -f '{{.Store.GraphDriverName}}'
podman info -f '{{index .Store.GraphStatus "Native Overlay Diff"}}'

Rootless overlay support

Rootless containers can use native OverlayFS since kernel 5.13. Before that, or on older kernels, fuse-overlayfs is used as a fallback.

BuildKit and build cache

BuildKit is Docker's modern build engine. It caches build results to speed up subsequent builds.

# Registry cache
docker buildx build --push -t myimage:latest \
  --cache-to type=registry,ref=myrepo/myimage:cache \
  --cache-from type=registry,ref=myrepo/myimage:cache \
  .

# Cache mode (min vs max)
# min: only cache layers for resulting image (default)
# max: cache all intermediate layers
docker buildx build --cache-to type=registry,ref=mycache,mode=max .

Cache mounts

# Dockerfile with cache mount for package manager
RUN --mount=type=cache,target=/var/cache/apt \
  apt-get update && apt-get install -y python3

Volumes, bind mounts, and tmpfs

Named volumes

# Create and use
docker volume create mydata
docker run -v mydata:/data myimage
docker volume rm mydata

Bind mounts

# Bind mount a directory
docker run -v /host/path:/container/path myimage

# Read-only bind mount
docker run -v /host/path:/container/path:ro myimage

tmpfs mounts

# tmpfs with size limit
docker run --tmpfs /tmp:rw,size=100m myimage

Kubernetes persistent volumes

Kubernetes provides persistent storage through PersistentVolumes (PV) and PersistentVolumeClaims (PVC). StorageClasses enable dynamic provisioning via CSI drivers.

PersistentVolume (PV)

apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv-example
spec:
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: fast
  hostPath:
    path: /data

PersistentVolumeClaim (PVC)

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pvc-example
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
  storageClassName: fast

StorageClass

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: fast
provisioner: csi.driver.example.com
parameters:
  type: ssd
reclaimPolicy: Delete
allowVolumeExpansion: true
volumeBindingMode: WaitForFirstConsumer
mountOptions:
  - discard

Reclaim policies

Best practices

Storage driver selection by backing filesystem

Disk space management

# Check disk usage
docker system df
podman system df

# Prune unused resources
docker system prune -a --volumes
podman system prune -a --volumes

# Prune specific types
docker image prune -a
docker volume prune
docker builder prune

OverlayFS and SELinux

SELinux labels are supported on OverlayFS (fixed in RHEL 7.3). Volume mounts require proper labeling:

# Automatically relabel volume mount (private label)
docker run -v /host/data:/data:z myimage

# Recursively relabel
docker run -v /host/data:/data:Z myimage

Production recommendations

1. Use overlay2 on ext4 or xfs (ftype=1). It is the most tested and performant combination.
2. Use named volumes for persistent data; avoid bind mounts for production databases.
3. Enable SELinux/AppArmor. Do not disable security features to work around permission issues.
4. Automate regular pruning of unused images and containers.
5. Monitor disk usage and set up alerts for /var/lib/docker and /var/lib/containers.
6. Use BuildKit cache backends to export cache to a registry for CI/CD.
7. Avoid writing to the container layer. Use volumes for write-heavy workloads.
8. Use native overlayfs for rootless containers, preferring kernel 5.13+ to avoid fuse-overlayfs overhead.