nano SIEM
Search Commands

anomaly

Supported in:Self-hosted

Find statistical outliers with z-score or MAD: unusual numeric values, unusual per-group totals, or values that occur unusually often.

Use anomaly to identify unusual numeric values or group totals in your results. You can score them with z-score or MAD (median absolute deviation).

Syntax

... | anomaly field=<field> [by <field> ...] [threshold=<n>] [method=zscore|mad]
... | anomaly <agg>(<field>) by <field> ... [threshold=<n>] [method=zscore|mad]
... | anomaly <field> [by <field> ...] [threshold=<n>] [method=zscore|mad]
OptionDefaultDescription
field=<f>Numeric field to score, event by event.
<agg>(<f>)Aggregate first, then score the groups: count(), sum(traffic.bytes_out), avg(duration).
<f>A bare field. A numeric field is scored like field=; a string field is scored by how often each value occurs.
byGroup fields. Also accepted as by=<field>.
threshold3How many standard deviations (or MAD equivalents) count as an outlier.
methodzscorezscore uses mean and standard deviation. mad uses the median and MAD, which is less distorted by the outliers you are looking for.

span= and sensitivity= are accepted and ignored.

How it works

Three forms, with different output:

FormWhat is scoredRows returned
anomaly field=f [by g]Each event's f, against the mean of its by group (or of all events)Only outliers (is_anomaly = 1), highest score first
anomaly f on a string fieldHow often each value of f occurs, against the other valuesOnly outliers: one row per unusually frequent (or infrequent) value
anomaly agg(f) by gThe aggregate per group, against the distribution of all groupsEvery group, scored, highest score first

Columns added:

ColumnMeaning
anomaly_scoreDistance from the center in standard deviations (z-score) or scaled MAD units. Higher is more unusual.
is_anomaly1 when the score is above threshold, otherwise 0.
avg_val, stddev_valMean and standard deviation (z-score method).
median_val, mad_valMedian and MAD (MAD method). The score uses MAD × 1.4826, so it is comparable to a z-score.
expectedThe center value: the mean, or the median for MAD.
count, sum_<field>, …In the aggregate form, the aggregate value per group.

Examples

Unusually large HTTP responses for each web host:

class_uid=4002 | anomaly field=http_response.length by http_request.url.hostname threshold=3 | table time, src_endpoint.ip, http_request.url.hostname, http_response.length, anomaly_score

Internal hosts that sent far more data than the rest (aggregate form, every host scored):

class_uid=4002 | anomaly sum(traffic.bytes_out) by src_endpoint.ip | where is_anomaly=1

Source IPs with an unusual number of failed logons, using MAD:

class_uid=3002 status_id=2 | anomaly count() by src_endpoint.ip method=mad threshold=3

Unusually slow requests, robust to existing outliers:

class_uid=4002 | anomaly field=duration method=mad threshold=4

Hourly logon counts per user, scored against that user's own history (run over several days):

class_uid=3002 | bin span=1h | stats count as logons by time_bucket, user.name | anomaly field=logons by user.name threshold=3

Notes

  • Enough data. A group with only a few values can't produce a high score: with n values, the largest possible z-score is about √(n−1). Use a longer time range or a coarser by before you lower the threshold.
  • Later commands can use anomaly_score, is_anomaly, and expected. For z-score, they can also use avg_val and stddev_val; for MAD, median_val and mad_val. The aggregate form also exposes its aggregate column. Use eventstats when you want to calculate your own comparison.
  • threshold values of 2 to 3 suit exploration; use 3 or more in detections to keep noise down.

What's next

  • eventstats: compare events to their group with your own math
  • bin: bucket events before scoring per time window
  • rare: least common values
  • where: filter on anomaly_score or is_anomaly

On this page