Disk pressure
Automatic eviction of the oldest daily ClickHouse partitions when local disk fills up, and the alerts it raises.
The disk pressure service checks ClickHouse's local disk usage (system.disks) on a fixed interval. When the disk fills up, it drops the oldest daily partitions so the host never runs out of space. It runs in the API service; in multi-replica deployments only the elected leader drops partitions.
Pressure levels
| Level | Default trigger | What happens |
|---|---|---|
| Normal | Below the high watermark | Nothing. Resolves an open disk-pressure health issue and resumes ingest if it was paused. |
| Elevated | ≥ 80% | Drops the oldest daily partitions until usage is below the low watermark (70%). |
| Critical | ≥ 90% | As elevated, plus a disk_pressure_critical audit event and an admin notification. |
| Emergency | ≥ 95% | As critical, plus an optional ingest pause. |
Partition drops:
- Oldest first, at most 5 partition dates per check.
- Tables:
logs,signals,ingestion_errors,identity_observations,nat_candidates. - Each drop writes a
partition_droppedaudit event and a notification. - Dropped data can't be recovered.
If storage tiering to object storage is active, the service doesn't drop partitions; ClickHouse moves aged parts off the local disk instead. It still raises the critical audit event and notification.
Configuration
Set these environment variables on the API service. The defaults suit most installs.
| Variable | Default | Meaning |
|---|---|---|
DISK_PRESSURE_CHECK_INTERVAL_SECS | 60 | Seconds between checks |
DISK_PRESSURE_HIGH_WATERMARK | 0.80 | Start dropping partitions above this fraction of disk used |
DISK_PRESSURE_LOW_WATERMARK | 0.70 | Stop dropping once usage is below this |
DISK_PRESSURE_CRITICAL_THRESHOLD | 0.90 | Critical level: audit event and notification |
DISK_PRESSURE_EMERGENCY_THRESHOLD | 0.95 | Emergency level |
DISK_PRESSURE_PAUSE_INGESTION | false | Pause ingest at the emergency level |
WARM_RETENTION_DAYS | 180 | With cold storage on, how long data stays on the warm volume before deletion |
Values are fractions from 0.0 to 1.0. For a small disk where space is tight:
DISK_PRESSURE_HIGH_WATERMARK=0.50
DISK_PRESSURE_LOW_WATERMARK=0.40
DISK_PRESSURE_CRITICAL_THRESHOLD=0.75
DISK_PRESSURE_EMERGENCY_THRESHOLD=0.85
DISK_PRESSURE_PAUSE_INGESTION=trueMonitoring
Settings → Storage & Retention shows a disk pressure panel on the ClickHouse · Logs tab. The same data is in the disk_pressure object of GET /api/settings/storage/overview:
| Field | Meaning |
|---|---|
usage_fraction | Disk used, 0.0–1.0 |
total_bytes, used_bytes, free_bytes | Absolute disk figures |
level | normal, elevated, critical or emergency |
estimated_retention_days | Days of data the disk can hold at the current rate |
partitions_dropped | Partitions dropped since the service started |
ingestion_paused | Whether ingest is paused |
Notifications are sent once per pressure episode, not on every check. Elevated and higher levels also appear in SIEM Health.