iSCSI & NVMe over Fabrics (NVMe-oF)

iSCSI and NVMe-oF are the two primary block-level network storage protocols on Linux. iSCSI encapsulates SCSI commands over TCP/IP and is universally supported. NVMe-oF extends the NVMe protocol over network fabrics (TCP, RDMA, Fibre Channel) for lower latency and higher performance with flash storage.

iSCSI architecture

iSCSI target setup with targetcli

# Install
sudo dnf install targetcli          # RHEL/Fedora
sudo apt install targetcli-fb       # Debian/Ubuntu
sudo systemctl enable --now target

Backstores

Complete target setup

sudo targetcli

# Create block backstore
/> backstores/block create my_disk /dev/sdb

# Create iSCSI target
/> iscsi/ create iqn.2006-04.com.example:storage.target

# Navigate to TPG1
/> cd iscsi/iqn.2006-04.com.example:storage.target/tpg1

# Create LUN
/iscsi/.../tpg1> luns/ create /backstores/block/my_disk

# Create portal
/iscsi/.../tpg1> portals/ create 192.168.1.100

# Create ACL for initiator
/iscsi/.../tpg1> acls/ create iqn.2005-03.org.open-iscsi:initiator1

# Save configuration (persists to /etc/target/saveconfig.json)
/> saveconfig
/> exit

Demo mode (testing only, NOT for production)

/iscsi/.../tpg1> set attribute authentication=0 demo_mode_write_protect=0 \
  generate_node_acls=1 cache_dynamic_acls=1

iSCSI initiator setup

# Install
sudo dnf install iscsi-initiator-utils    # RHEL/Fedora
sudo apt install open-iscsi               # Debian/Ubuntu
sudo systemctl enable --now iscsid

# View current initiator IQN
cat /etc/iscsi/initiatorname.iscsi

Discovery and login

# Discover targets
iscsiadm -m discovery -t st -p 192.168.1.100:3260

# Login to specific target
iscsiadm -m node -T iqn.2006-04.com.example:storage.target \
  -p 192.168.1.100 -l

# Login to all discovered targets
iscsiadm -m node -L all

# Set automatic login at boot
iscsiadm -m node -T iqn.2006-04.com.example:storage.target \
  -p 192.168.1.100 --op=update -n node.startup -v automatic

Sessions and logout

# List active sessions
iscsiadm -m session

# Detailed session info
iscsiadm -m session -P 3

# Logout
iscsiadm -m node -T iqn.2006-04.com.example:storage.target \
  -p 192.168.1.100 -u

# Rescan for new LUNs
iscsiadm -m node -T iqn.2006-04.com.example:storage.target \
  -p 192.168.1.100 --rescan

CHAP authentication

Target configuration (targetcli)

/iscsi/.../acls/iqn.2005-03.org.open-iscsi:initiator1> set auth userid=initiator_user
/iscsi/.../acls/iqn.2005-03.org.open-iscsi:initiator1> set auth password=initiator_password

# Mutual CHAP (bidirectional)
/iscsi/.../acls/...> set auth mutual_userid=target_user
/iscsi/.../acls/...> set auth mutual_password=target_password

# Enable authentication on TPG
/iscsi/.../tpg1> set attribute authentication=1

Initiator configuration (/etc/iscsi/iscsid.conf)

node.session.auth.authmethod = CHAP
node.session.auth.username = initiator_user
node.session.auth.password = initiator_password

# Mutual CHAP
node.session.auth.username_in = target_user
node.session.auth.password_in = target_password

iSCSI tuning

# /etc/iscsi/iscsid.conf

# Queue depth (high-performance)
node.session.cmds_max = 2048
node.session.queue_depth = 128

# Multipath (aggressive timeout)
node.session.timeo.replacement_timeout = 0
node.conn[0].timeo.noop_out_interval = 1
node.conn[0].timeo.noop_out_timeout = 1

# TCP window (0 = kernel autotune, recommended)
node.conn[0].tcp.window_size = 0

iSER (iSCSI over RDMA)

iSER extends iSCSI to use RDMA for zero-copy data transfer, reducing CPU overhead and latency. Requires InfiniBand or RoCE hardware.

# Target: create portal, then enable iSER on it
targetcli /iscsi/.../tpg1/portals> create 192.168.1.100
targetcli /iscsi/.../tpg1/portals/192.168.1.100:3260> iser_enable

# Initiator: discover and login via iSER
iscsiadm -m discovery -t st -I iser -p 192.168.1.100
iscsiadm -m node -T iqn.2006-04.com.example:storage.target -I iser -l

NVMe-oF architecture

Transport types

NVMe-oF target setup (nvmet)

# Install
sudo dnf install nvmetcli    # RHEL/Fedora
sudo apt install nvmetcli    # Debian/Ubuntu

# Load kernel modules
modprobe nvmet
modprobe nvmet-tcp           # For TCP transport
modprobe nvmet-rdma          # For RDMA transport

# Mount configfs (if not already)
mount -t configfs none /sys/kernel/config/

Manual configuration via configfs

# Create subsystem
mkdir /sys/kernel/config/nvmet/subsystems/nqn.2016-06.com.example:storage

# Allow any host (disable for explicit access control)
echo 1 > /sys/kernel/config/nvmet/subsystems/nqn.2016-06.com.example:storage/attr_allow_any_host

# Create namespace
mkdir /sys/kernel/config/nvmet/subsystems/nqn.2016-06.com.example:storage/namespaces/1

# Set device path
echo -n /dev/sdb > /sys/kernel/config/nvmet/subsystems/nqn.2016-06.com.example:storage/namespaces/1/device_path

# Enable namespace
echo 1 > /sys/kernel/config/nvmet/subsystems/nqn.2016-06.com.example:storage/namespaces/1/enable

# Create port
mkdir /sys/kernel/config/nvmet/ports/1

# Configure transport
echo tcp > /sys/kernel/config/nvmet/ports/1/addr_trtype
echo ipv4 > /sys/kernel/config/nvmet/ports/1/addr_adrfam
echo 192.168.1.100 > /sys/kernel/config/nvmet/ports/1/addr_traddr
echo 4420 > /sys/kernel/config/nvmet/ports/1/addr_trsvcid

# Link subsystem to port
ln -s /sys/kernel/config/nvmet/subsystems/nqn.2016-06.com.example:storage \
  /sys/kernel/config/nvmet/ports/1/subsystems/nqn.2016-06.com.example:storage

Host access control

# Disable allow_any_host
echo 0 > /sys/kernel/config/nvmet/subsystems/nqn.2016-06.com.example:storage/attr_allow_any_host

# Create host
mkdir /sys/kernel/config/nvmet/hosts/nqn.2016-06.com.example:host1

# Link host to subsystem
ln -s /sys/kernel/config/nvmet/hosts/nqn.2016-06.com.example:host1 \
  /sys/kernel/config/nvmet/subsystems/nqn.2016-06.com.example:storage/hosts/nqn.2016-06.com.example:host1

NVMe-oF initiator setup

# Install
sudo dnf install nvme-cli    # RHEL/Fedora
sudo apt install nvme-cli    # Debian/Ubuntu

# View host NQN
nvme show-hostnqn

Discovery and connect

# Discover subsystems
nvme discover -t tcp -a 192.168.1.100 -s 4420

# Connect to specific subsystem
nvme connect -t tcp -a 192.168.1.100 -s 4420 \
  -n nqn.2016-06.com.example:storage

# Connect with queue depth
nvme connect -t tcp -a 192.168.1.100 -s 4420 \
  -n nqn.2016-06.com.example:storage --queue-size 1024

# Connect with controller loss timeout (seconds)
nvme connect -t tcp -a 192.168.1.100 -s 4420 \
  -n nqn.2016-06.com.example:storage -l 3600

# Discover and connect to all available subsystems
nvme connect-all

Listing and disconnect

# List NVMe devices
nvme list

# List subsystems and paths
nvme list-subsys

# Disconnect by device
nvme disconnect -d /dev/nvme0n1

# Disconnect by NQN
nvme disconnect -n nqn.2016-06.com.example:storage

# Disconnect all
nvme disconnect-all

ANA (Asymmetric Namespace Access)

ANA is the NVMe-oF equivalent of SCSI ALUA, providing multipathing and path optimization. NVMe multipath is built into the kernel (not DM-Multipath).

# Enable NVMe multipath
modprobe nvme-multipath

# View multipath status
nvme list-subsys

# View ANA log
nvme ana-log /dev/nvme0

iSCSI vs NVMe-oF comparison

Best practices

1. Network isolation: use dedicated storage networks or VLANs.
2. Always use authentication: CHAP for iSCSI, DH-HMAC-CHAP for NVMe-oF.
3. Use mutual CHAP for bidirectional authentication.
4. Configure multipathing. DM-Multipath for iSCSI, native nvme-multipath for NVMe-oF.
5. Tune queue depth based on workload and storage capabilities.
6. Monitor sessions: iscsiadm -m session, nvme list-subsys.
7. Use jumbo frames (9000B MTU) for better throughput on high-bandwidth links.
8. For RDMA, ensure lossless Ethernet (DCB/PFC) for RoCEv2.
9. Test failover by simulating path failures in a non-production environment.