dedup
Keep one event per unique value, or combination of values, of the fields you name.
Use dedup to keep one row for each distinct value or combination of values in your chosen fields. uniq is an alias.
Syntax
... | dedup <field> [, <field> ...]
... | uniq <field> [, <field> ...]| Argument | Description |
|---|---|
field | One or more fields that form the key. Separate them with commas or spaces. |
For compatibility with other SIEM query languages, dedup also accepts a leading count (dedup 3 user.name), keepfirst= / keeplast= and a trailing sortby clause. nano parses them and ignores them.
How it works
- For each key,
dedupkeeps the oldest event in the time range. Which event wins when two share the sametimeis not defined. - Events where a key field is empty are not dropped: they form their own group, and one of them is kept.
- The output keeps every field of the surviving event. Results are ordered newest first, like any event search.
- To keep the newest event per key instead, aggregate:
stats latest(time) as last_seen by <field>. To count events per key, use stats.
Examples
The first logon seen for each user in the window:
class_uid=3002 status_id=1 | dedup user.name | table time, user.name, src_endpoint.ip, device.hostnameOne row per user and source address pair:
class_uid=3002 status_id=2 | dedup user.name, src_endpoint.ip | table time, user.name, src_endpoint.ipOne example of each distinct command line on a host:
class_uid=1007 device.hostname="ws-hr-005.corp.local" | dedup process.cmd_line | table time, process.name, process.cmd_lineThe first CloudTrail call of each API operation:
metadata.log_source=aws_cloudtrail | uniq api.operation | table time, api.operation, actor.user.name, src_endpoint.ipNotes
dedupworks on raw events and after other commands. Put filters before it so fewer events are grouped.- To get the distinct values only,
stats count by <field>is cheaper and also tells you how many events each value has.