Search best practices
Write nPL searches that return quickly and correctly: time range first, field filters over keywords, narrow by log source and class, nano-side stages last.
Use these practices to narrow your nPL searches and check their results. You reduce the data nano reads by choosing a shorter time range and filters the engine can use to skip events that cannot match.
Pick the shortest useful time range
The time range is the biggest lever. Every search is bounded by it, and the engine skips everything outside it.
- Start with the last hour or day while you build a query, then widen it.
- Set it in the time picker, or inline with
earliest=andlatest=(unitss m h d w M y):
class_uid=3002 status_id=2 earliest=-4h | stats count by user.nameFilter on fields, not bare keywords
A bare keyword such as 10.1.3.87 or rundll32 searches the text of every event (message and raw_data). A field filter compares one column, and the engine can skip data where the value can't occur. On a test deployment over 24 hours, the field form below ran more than ten times faster than the keyword form and returned only the events where the value is actually the source IP.
10.1.3.87 | stats count // slow: text of every event, any position
src_endpoint.ip="10.1.3.87" | stats count // fast: one field
src_endpoint.ip="10.1.3.87" OR dst_endpoint.ip="10.1.3.87" | stats count // either sideThe same goes for process, host and domain hunts:
process.name="rundll32.exe" | stats count by device.hostname
http_request.url.hostname="pastebin.com" | table time, src_endpoint.ip, http_request.url.url_string
file.hashes.value="44d88612fea8a8f36de82e1278abb02f" | table time, device.hostname, file.pathUse keywords when you don't know which field holds the value. Then narrow to a field once you find it. See OCSF field reference for which path holds what.
Narrow by log source and class
metadata.log_source and class_uid cut the data down before anything else runs. Add them whenever you know what you're hunting in:
metadata.log_source=windows_event class_uid=3002 status_id=2 | stats count by user.name, src_endpoint.ip
metadata.log_source=windows_sysmon class_uid=1007 process.cmd_line="*-EncodedCommand*" | table time, device.hostname, actor.user.name, process.cmd_lineClass-dependent paths matter too. In Process Activity (1007) the process is process.* and its parent is actor.process.*. In network, file and DNS classes the acting process is actor.process.*. See OCSF schema.
Prefer exact matches and lists over wildcards and regex
field="value"is the cheapest filter. String equality is case-insensitive.- For several values, use
IN:process.name IN ("rundll32.exe", "regsvr32.exe", "mshta.exe"). - Wildcards (
"*ll32.exe","rundll*") and regex (=~,/…/) have to look at each candidate value. Use them when you need them, after the cheaper filters. - For IP ranges, use CIDR:
src_endpoint.ip="10.0.0.0/8"orsrc_endpoint.ip IN CIDR("10.0.0.0/8", "192.168.0.0/16").
class_uid=1007 process.name IN ("rundll32.exe", "regsvr32.exe", "mshta.exe") | stats count by device.hostname, process.name
class_uid=4001 dst_endpoint.port=3389 src_endpoint.ip IN CIDR("10.0.0.0/8") | stats dc(dst_endpoint.ip) as targets by src_endpoint.ipSee Syntax for every operator.
Filter before you aggregate, aggregate before you sort
Put filters in the base search, then aggregate, then sort and limit the small result:
metadata.log_source=apache_access http_response.code>=500 | stats count by http_request.url.path | sort -count | head 20| whereand| searchafter a pipe work, but a base-search filter is cheaper because it runs first.- Sort and dedup the aggregated rows, not raw events. To list distinct values,
stats count by fis cheaper thandedup f. - Give
transactionamaxspanandmaxevents, andtimechartaspanthat fits the range (about 100–500 buckets).
class_uid=3002 | transaction user.name maxspan=30m maxevents=100 | where eventcount > 20
class_uid=4002 earliest=-24h | timechart span=15m count by http_response.codePlace enrichment stages after filters
lookup, prevalence, inputlookup, resolve_identity, ai and lateral run in nano after ClickHouse returns rows, not inside the ClickHouse query. Two consequences:
- Filter hard before them. They work on the rows ClickHouse returns, so put every filter you can before the first one.
- Only some commands work after them. After the first nano-side stage, only these run:
where,head,tail,sort,table,stats,dedup,rename,fields,top,rare,fillnull,eval,timechart,rex,risk,reverse,tree. Any other command placed after one causes the search to fail. Eval functions after the split are also limited; see Functions.
class_uid=1007 device.hostname="ws-hr-005.corp.local" | head 500 | prevalence hash_prevalence < 5 | table time, process.name, process.cmd_lineThese stages also can't run inside a subsearch (append, join, IN [ … ]). Move them to the outer pipeline.
Limits
| Limit | Value |
|---|---|
| Pipe commands per query | 25 |
| Nesting depth (parentheses and subsearches) | 50 |
| Terms in one AND / OR / arithmetic chain | 1,024 |
| Query length | 1 MiB |
head / tail default | 10 rows |
Search concurrency, queueing, post-processing group caps and cost checks are deployment settings. See Search settings.
Check your results
Compare a field filter against a broader query once, such as the keyword form. Confirm that the narrower query includes the events you need.
Troubleshoot
Need help with these checks? Contact support with the affected deployment, request ID and time with timezone.
A filter returns no events
A field that no event in the range populates returns empty values. Check the path in the OCSF field reference, confirm which class carries it, and review the time range.
nano rejects a field name
Use an OCSF 1.9 path or a field created earlier in the pipeline. For source fields with no OCSF home, use unmapped.<key>.
A command fails after enrichment
Move commands that post-processing cannot run before the first enrichment stage. Review post-processing after nano-side stages and the function availability tables in Functions.
What's next
- Syntax: search terms, operators, precedence and comments
- Quick reference: every command on one page
- stats, where, timechart