anomaly
Find statistical outliers with z-score or MAD: unusual numeric values, unusual per-group totals, or values that occur unusually often.
Use anomaly to identify unusual numeric values or group totals in your results. You can score them with z-score or MAD (median absolute deviation).
Syntax
... | anomaly field=<field> [by <field> ...] [threshold=<n>] [method=zscore|mad]
... | anomaly <agg>(<field>) by <field> ... [threshold=<n>] [method=zscore|mad]
... | anomaly <field> [by <field> ...] [threshold=<n>] [method=zscore|mad]| Option | Default | Description |
|---|---|---|
field=<f> | Numeric field to score, event by event. | |
<agg>(<f>) | Aggregate first, then score the groups: count(), sum(traffic.bytes_out), avg(duration). | |
<f> | A bare field. A numeric field is scored like field=; a string field is scored by how often each value occurs. | |
by | Group fields. Also accepted as by=<field>. | |
threshold | 3 | How many standard deviations (or MAD equivalents) count as an outlier. |
method | zscore | zscore uses mean and standard deviation. mad uses the median and MAD, which is less distorted by the outliers you are looking for. |
span= and sensitivity= are accepted and ignored.
How it works
Three forms, with different output:
| Form | What is scored | Rows returned |
|---|---|---|
anomaly field=f [by g] | Each event's f, against the mean of its by group (or of all events) | Only outliers (is_anomaly = 1), highest score first |
anomaly f on a string field | How often each value of f occurs, against the other values | Only outliers: one row per unusually frequent (or infrequent) value |
anomaly agg(f) by g | The aggregate per group, against the distribution of all groups | Every group, scored, highest score first |
Columns added:
| Column | Meaning |
|---|---|
anomaly_score | Distance from the center in standard deviations (z-score) or scaled MAD units. Higher is more unusual. |
is_anomaly | 1 when the score is above threshold, otherwise 0. |
avg_val, stddev_val | Mean and standard deviation (z-score method). |
median_val, mad_val | Median and MAD (MAD method). The score uses MAD × 1.4826, so it is comparable to a z-score. |
expected | The center value: the mean, or the median for MAD. |
count, sum_<field>, … | In the aggregate form, the aggregate value per group. |
Examples
Unusually large HTTP responses for each web host:
class_uid=4002 | anomaly field=http_response.length by http_request.url.hostname threshold=3 | table time, src_endpoint.ip, http_request.url.hostname, http_response.length, anomaly_scoreInternal hosts that sent far more data than the rest (aggregate form, every host scored):
class_uid=4002 | anomaly sum(traffic.bytes_out) by src_endpoint.ip | where is_anomaly=1Source IPs with an unusual number of failed logons, using MAD:
class_uid=3002 status_id=2 | anomaly count() by src_endpoint.ip method=mad threshold=3Unusually slow requests, robust to existing outliers:
class_uid=4002 | anomaly field=duration method=mad threshold=4Hourly logon counts per user, scored against that user's own history (run over several days):
class_uid=3002 | bin span=1h | stats count as logons by time_bucket, user.name | anomaly field=logons by user.name threshold=3Notes
- Enough data. A group with only a few values can't produce a high score: with n values, the largest possible z-score is about √(n−1). Use a longer time range or a coarser
bybefore you lower the threshold. - Later commands can use
anomaly_score,is_anomaly, andexpected. For z-score, they can also useavg_valandstddev_val; for MAD,median_valandmad_val. The aggregate form also exposes its aggregate column. Use eventstats when you want to calculate your own comparison. thresholdvalues of 2 to 3 suit exploration; use 3 or more in detections to keep noise down.
What's next
- eventstats: compare events to their group with your own math
- bin: bucket events before scoring per time window
- rare: least common values
- where: filter on
anomaly_scoreoris_anomaly