STORAGE ENGINEERING

SMART monitoring & disk health prediction

Category: Disk HealthTechnologies: smartmontools, smartctl, smartd, nvme-cli, Backblaze stats

SMART (Self-Monitoring, Analysis and Reporting Technology) is built into most ATA/SATA and SCSI/SAS drives. It monitors internal attributes and can run self-tests to predict failure, though it is not a guarantee. Research shows roughly 36% of drives fail without tripping any SMART threshold.

smartctl commands

# All SMART information
smartctl -a /dev/sda

# Health status only (PASSED/FAILED)
smartctl -H /dev/sda

# Device identity
smartctl -i /dev/sda

# SMART attributes
smartctl -A /dev/sda

# Error log
smartctl -l error /dev/sda

# Self-test log
smartctl -l selftest /dev/sda

# Run short self-test (1-2 minutes)
smartctl -t short /dev/sda

# Run long self-test (can take hours)
smartctl -t long /dev/sda

# Run conveyance test (transport damage)
smartctl -t conveyance /dev/sda

# Comprehensive output (more than -a)
smartctl -x /dev/sda

Key SMART attributes (ATA)

ID	Name	Meaning	When to worry
5	Reallocated_Sector_Ct	Sectors remapped to spare area	Any non-zero value. Google 2007 study: ~14× higher failure rate.
187	Reported_Uncorrectable_Errors	Uncorrectable errors reported to host	Non-zero: data loss likely
197	Current_Pending_Sector	Sectors pending reallocation (unreadable)	Non-zero is urgent
198	Offline_Uncorrectable	Sectors that failed offline scan	Non-zero: elevated failure risk (treat like SMART 197)
199	UDMA_CRC_Error_Count	Cable/backplane/HBA errors	Increasing: infrastructure issue, not drive failure
9	Power_On_Hours	Hours powered on	Not a failure indicator; track for age/EOL planning
194	Temperature_Celsius	Drive temperature	Sustained high temps or sharp swings; consult vendor spec for thresholds
233	Media_Wearout_Indicator (SSD)	Remaining endurance (100 → 1)	Below 10-20

Backblaze, monitoring 67,000+ drives, uses attributes 5, 187, 188, 197, and 198 as their main failure predictors.

smartd configuration

smartd is a daemon that monitors drives continuously and sends alerts. Configuration lives in /etc/smartd.conf (or /etc/smartmontools/smartd.conf on Debian).

# Monitor all devices, all attributes, email alerts
DEVICESCAN -a -m root@example.com

# Health status only + email
DEVICESCAN -H -m root@example.com

# Specific device: short test daily at 2 AM, long test Saturday 3 AM
/dev/sda -a -m root@example.com -s S/../.././02 -s L/../../6/03

Test scheduling format: -s TYPE/../../DAY/HR where TYPE is S (short), L (long), or C (conveyance); DAY is 0-6 (0=Sunday); HR is 0-23.

# Enable and start
sudo systemctl enable --now smartd

# Test email alert
smartctl -t short /dev/sda  # trigger a test
# or:
smartd --testmail

NVMe SMART

NVMe uses a standardized log page format instead of vendor-specific attributes. Use nvme-cli:

# Display SMART log
nvme smart-log /dev/nvme0

# Display for specific namespace
nvme smart-log /dev/nvme0n1

Attribute	Meaning
`critical_warning`	Bit field; non-zero means a critical issue
`percentage_used`	Endurance consumed (0 → 100). Most important NVMe wear indicator.
`data_units_read` / `data_units_written`	512-byte units read/written (reported in TB/PB)
`media_errors`	Count of unrecoverable data errors
`available_spare`	Percentage of spare capacity remaining
`temperature`	Reported in Kelvin and Celsius directly

Interpreting SMART data

SMART is not a guarantee

Manufacturer SMART algorithms have a failure detection rate of only 3-10% with a 0.1% false alarm rate. Some failures (controller failure, firmware bugs) occur suddenly with no SMART warning. Always maintain backups regardless of SMART status.

Normalized vs raw values

ATA SMART attributes have a normalized value (typically 1-253, higher is better), a worst value, a threshold, and a raw value. The normalized value is what the manufacturer's algorithm watches. The raw value is the actual count, useful for trend analysis but vendor-specific in meaning.

When to replace a drive

Reallocated sectors (ID 5) growing: any increase is concerning; consistent growth over weeks means replace soon.
Pending sectors (ID 197) non-zero. Data is at risk. Try writing to force reallocation, and replace if the count doesn't decrease.
Offline uncorrectable (ID 198): data loss has likely occurred.
NVMe media_errors non-zero: serious. percentage_used approaching 100% means end of life.
NVMe available_spare dropping indicates wear; compare against available_spare_threshold.
Multiple attributes degrading: replace preventatively.
SMART self-tests failing. Replace immediately.

Monitor trends rather than absolute values. A single reallocated sector on a 5-year-old drive is less urgent than 50 new ones in a week.

Automated monitoring

# /etc/smartd.conf, email alerts with scheduled tests
DEVICESCAN -a -m admin@example.com -M daily \
  -s S/../.././02 \
  -s L/../../6/03

# -M daily: send one email per device per day (not per alert)
# -M test:  send a test email on daemon start
# -M exec /path/to/script: run a script instead of email

# Example alert script
#!/bin/bash
# /etc/smartd_warning.sh
# $SMARTD_DEVICE, $SMARTD_FAILTYPE, $SMARTD_MESSAGE available
echo "SMART alert on $SMARTD_DEVICE: $SMARTD_MESSAGE" | \
  mail -s "SMART warning: $SMARTD_DEVICENAME" admin@example.com
# Or send to Slack/PagerDuty/etc.