Ceph distributed storage: RADOS, RBD & CephFS

Q: What is the difference between Ceph RBD and CephFS?

RBD (RADOS Block Device) exposes thin-provisioned, striped block devices over the network to a single host (or VM) as a virtual hard disk ( /dev/rbd0 ). CephFS is a shared POSIX-compliant distributed filesystem that can be mounted simultaneously by hundreds of client servers via kernel or FUSE drivers.

Q: Why is the _netdev mount option essential for Ceph in /etc/fstab?

The _netdev option tells the Linux systemd boot sequence that the filesystem resides on a network-dependent device. This prevents systemd from attempting to mount the volume before the network interfaces and Ceph kernel modules are initialized.

Category: Distributed StorageCore Subsystems: RADOS, BlueStore, RBD, CephFSStandard: Ceph Reef / Quincy / Tentacle

Ceph is the industry-standard open-source distributed software-defined storage platform. It delivers object, block, and file storage unified under a single distributed cluster. Built on top of the self-healing RADOS (Reliable Autonomic Distributed Object Store) layer, Ceph eliminates single points of failure and scales seamlessly to exabytes of capacity across commodity hardware.

Ceph cluster architecture & core daemons

Daemon / Layer	Acronym	Architectural Role
Object Storage Daemon	`ceph-osd`	Manages physical storage drives via the BlueStore engine; handles replication, scrub, recovery, and rebalancing.
Monitor Daemon	`ceph-mon`	Maintains cluster consensus and maps (mon map, osd map, pg map, crush map) using Paxos consensus.
Manager Daemon	`ceph-mgr`	Provides cluster monitoring, dashboard metrics, Prometheus exporters, and automated balancing.
Metadata Server	`ceph-mds`	Manages POSIX namespace directory hierarchies and metadata caching specifically for CephFS.
RADOS Gateway	`ceph-rgw`	Provides S3 and OpenStack Swift compatible HTTP REST API object storage endpoints.

Linux client: RADOS Block Device (RBD)

RBD stripes virtual disk images across hundreds of OSDs in 4 MiB chunks, providing high-performance network block storage for Linux hypervisors and database hosts.

1. Map an RBD image to a kernel block device

# Install Ceph client tools
sudo apt install ceph-common

# Map image 'db_disk_01' from pool 'rbd_data'
sudo rbd map rbd_data/db_disk_01 --name client.admin

# Output: /dev/rbd0

# Verify mapped block devices
rbd device list

2. Format and mount with persistence

Once mapped as /dev/rbd0, format the block device with an enterprise filesystem like XFS or Ext4:

# Format with XFS (disable TRIM discard via -K if cluster handles reclamation)
sudo mkfs.xfs -f -K /dev/rbd0

# Mount directory
sudo mkdir -p /srv/ceph_storage
sudo mount -o noatime,_netdev /dev/rbd0 /srv/ceph_storage

# Persistent /etc/fstab entry
/dev/rbd0  /srv/ceph_storage  xfs  noatime,_netdev,x-systemd.requires=rbdmap.service  0  0

Linux client: CephFS distributed filesystem

Unlike point-to-point block SAN protocols (iSCSI & NVMe-oF), CephFS provides a POSIX-compliant shared distributed filesystem mounted natively across multiple servers, comparable to high-availability NFS clusters:

# Mount CephFS directly using the in-tree kernel client (mount.ceph)
sudo mkdir -p /mnt/shared_docs
sudo mount -t ceph 10.0.0.10,10.0.0.11,10.0.0.12:/ /mnt/shared_docs \
  -o name=admin,secretfile=/etc/ceph/admin.secret,noatime,_netdev

# Persistent /etc/fstab entry for CephFS
10.0.0.10,10.0.0.11,10.0.0.12:/  /mnt/shared_docs  ceph  name=admin,secretfile=/etc/ceph/admin.secret,noatime,_netdev  0  0

Snapshots & fast copy-on-write cloning

Ceph provides instant sub-second snapshots and layered clones for disaster recovery and backup workflows, as well as dynamic scaling alongside cloud block storage:

# Create snapshot of RBD image
rbd snap create rbd_data/db_disk_01@snap_20260820

# Protect snapshot and create instant thin-provisioned clone
rbd snap protect rbd_data/db_disk_01@snap_20260820
rbd clone rbd_data/db_disk_01@snap_20260820 rbd_data/db_disk_01_clone

Frequently asked questions

What is Ceph RADOS and how does the CRUSH algorithm work?

RADOS (Reliable Autonomic Distributed Object Store) is the foundational object layer of Ceph. The CRUSH (Controlled Replication Under Scalable Hashing) algorithm deterministically calculates which physical Object Storage Daemons (OSDs) store a given data object based on a weighted hierarchy tree (rack, row, host, disk). This eliminates central lookup table bottlenecks, enabling Ceph clusters to scale to thousands of storage nodes.

What is the difference between Ceph RBD and CephFS?

RBD (RADOS Block Device) exposes thin-provisioned, striped block devices over the network to a single host (or VM) as a virtual hard disk (/dev/rbd0). CephFS is a shared POSIX-compliant distributed filesystem that can be mounted simultaneously by hundreds of client servers via kernel or FUSE drivers.

What is Ceph BlueStore?

BlueStore is Ceph's high-performance native storage engine for OSDs. It bypasses local Linux filesystems (ext4/XFS) and writes directly to raw block devices, utilizing an embedded RocksDB database (via the BlueFS adapter, which provides a minimal filesystem-like interface for RocksDB) to manage internal metadata, with an optional separate WAL device for journaling.

Why is the _netdev mount option essential for Ceph in /etc/fstab?

The _netdev option tells the Linux systemd boot sequence that the filesystem resides on a network-dependent device. This prevents systemd from attempting to mount the volume before the network interfaces and Ceph kernel modules are initialized.