Detection Rules & Threat Hunting

Detection rules analyze incoming logs to identify suspicious activity and generate alerts. Rules run on cron schedules or as ClickHouse materialized views.

Overview

nano's detection system provides:

Scheduled and real-time rules - Cron-based execution, or materialized views for IOC matching (10-30s latency)
AI rule creation - Generate rules from natural-language descriptions
Prevalence-based filtering - Reduce noise using host-count baselines
Auto-tuning - AI-driven threshold and exclusion adjustments to cut false positives
MITRE ATT&CK mapping - Tactic and technique tagging

Detection Architecture

Rule Types

Scheduled Rules

Execute on cron schedules (every minute, every 5 minutes, hourly, daily)
Query historical data within specified lookback windows
Best for complex correlation and behavioral analysis
Typical latency: 1-15 minutes depending on schedule

Real-time Rules (Materialized Views)

Process logs as they arrive using ClickHouse materialized views
Detection for simple IOC matching at 10-30 second latency
Limited to basic field matching and comparisons

Prevalence-Enhanced Rules

Incorporate statistical baselines to filter common activities
Reduce false positives from legitimate but rare events
Automatically adapt to environmental changes
Support for hash, domain, IP, and process prevalence

Rule Lifecycle

Detection rules progress through modes that control whether they generate alerts:

Mode	Executed?	Alerts?	Use Case
Staging	No	No	Editing and development. Rule is not run
Live	Yes	No	Bake-in testing. Matches are logged and counted, but no alerts are created
Alerting	Yes	Yes	Production. Matches generate real alerts for analyst review
Paused	No	No	Temporarily stopped. Retains alerting context, easy to resume

Move rules through Staging → Live → Alerting as confidence increases. Use Paused to temporarily disable a production rule without losing its alerting state.

Alert Modes

Detection rules support two alert modes that control how matched events map to alerts:

Mode	Behavior	Use Case
`grouped` (default)	All matched events from one execution → 1 alert	Most detection rules, aggregation-based detections
`per_event`	Each matched event → its own alert (1:1 mapping)	Vendor pass-through (CrowdStrike, SentinelOne), IOC matching

When to use per_event:

Vendor pass-through: When forwarding detections from EDR/XDR tools (e.g., CrowdStrike Falcon detections), each vendor detection should become its own nano alert for 1:1 tracking.
IOC matching: When each matched event represents a distinct security finding that warrants individual triage.

Performance note: per_event mode creates N database inserts per rule execution (one per matched event). The existing max_events_per_alert limit (default: 100) caps the number of alerts created per execution.

Set alert_mode in the YAML frontmatter:

---
title: crowdstrike_pass_through
alert_mode: per_event
detection_mode: scheduled
schedule: "*/1 * * * *"
lookback: 5m
---
source_type=crowdstrike | where event_type="detection"

Creating Detection Rules

Manual Rule Creation

Navigate to Detections → New Rule to create rules manually:

Basic Information

Name: Descriptive rule identifier
Description: Detailed explanation of the threat scenario
Severity: Critical, High, Medium, Low, Info
Status: Active, Inactive, Testing

Detection Logic

Query: nPL (nano Pipe Language) query
Schedule: Cron expression for execution timing
Lookback: Time window to analyze (e.g., "15m", "1h", "1d")

Metadata

MITRE ATT&CK: Technique and tactic mappings
References: External threat intelligence links
Tags: Organizational labels for categorization

AI-Powered Rule Creation

Use the AI Detection Creator for natural language rule generation:

Process:

Click "Create with AI" button
Describe the threat in natural language
AI generates a detection rule
Review and customize the generated rule
Test against historical data
Deploy

Example Descriptions:

"Detect PowerShell execution with suspicious command line arguments"
"Find lateral movement using WMI or PSExec"
"Identify data exfiltration to external cloud storage"
"Detect credential dumping tools like Mimikatz"

AI Capabilities:

Query Generation - Converts descriptions to nPL queries
MITRE Mapping - Automatically assigns relevant techniques
Syntax Validation - Ensures query correctness
Historical Testing - Validates against real data
Noise Assessment - Predicts false positive rates
Tuning Recommendations - Suggests optimizations

Query Language (nPL)

nano uses nPL (nano Pipe Language) for detection queries:

Basic Syntax:

# Simple field matching
process_name="powershell.exe"

# Multiple conditions
process_name="powershell.exe" AND command_line CONTAINS "-enc"

# Time-based filtering
process_name="powershell.exe" | where timestamp > now() - 15m

# Statistical analysis
process_name="powershell.exe" | stats count by src_ip | where count > 10

Advanced Features:

# Prevalence filtering
process_hash="abc123..." | prevalence process_hash < 0.01

# Correlation across events
process_name="powershell.exe" | join src_ip [search network_connection]

# Temporal analysis
process_name="powershell.exe" | bucket timestamp span=5m | stats count by timestamp

Field References:

Use UDM (Unified Data Model) field names
Reference the UDM Fields for available fields
Common fields: src_ip, dest_ip, process_name, command_line, user

Rule Scheduling & Execution

Cron Expressions

Detection rules use cron expressions for flexible scheduling:

Common Patterns:

# Every 5 minutes
*/5 * * * *

# Every hour at minute 0
0 * * * *

# Daily at 2 AM
0 2 * * *

# Business hours only (9 AM - 5 PM, weekdays)
0 9-17 * * 1-5

# High-frequency (every 30 seconds) - requires 6-field format
*/30 * * * * *

Schedule Guidelines:

Critical IOCs: Every 1-5 minutes
Behavioral Analysis: Every 15-30 minutes
Statistical Baselines: Hourly or daily
Compliance Checks: Daily or weekly

Execution Context

Lookback Windows:

Short (5-15m): Real-time threat detection
Medium (1-4h): Behavioral pattern analysis
Long (1-7d): Statistical anomaly detection

Resource Considerations:

Complex queries with long lookbacks consume more resources
High-frequency schedules increase system load
Use materialized views for simple, frequent checks
Monitor execution times and optimize slow queries

Materialized Views for Real-Time Detection

When to Use Materialized Views

Ideal Scenarios:

Simple IOC matching (hashes, IPs, domains)
Known bad indicators from threat feeds
Basic field equality or pattern matching
High-value alerts requiring immediate response

Limitations:

No complex joins or correlations
Limited statistical functions
Simple WHERE clause conditions only
No time-based aggregations

Creating Materialized Views

Automatic Generation:

Create a detection rule with simple query
Enable "Real-time Mode" option
System automatically generates materialized view
View processes logs as they arrive

Manual Configuration:

Generated views write into the signals table, which references the source event by matched_log_id rather than copying log fields:

-- Example materialized view for malicious hash detection
CREATE MATERIALIZED VIEW malware_hash_detection_mv
TO signals
AS SELECT
    generateUUIDv4() AS id,
    timestamp,
    'malware_hash_detection' AS rule_name,
    'critical' AS severity,
    process_hash AS risk_entity,
    logs.id AS matched_log_id
FROM logs
WHERE process_hash IN (
    'abc123def456...',  -- Known malware hashes
    '789xyz012abc...'
);

Performance Benefits:

Sub-30 second detection latency
Minimal CPU overhead
Automatic scaling with log volume
No scheduled execution overhead

Prevalence-Based Detection

Understanding Prevalence

Prevalence tracking helps distinguish between common and rare activities by using pre-computed enrichment fields that are added to logs during ingestion.

How Prevalence Works:

Enrichment Process - Logs are enriched with prevalence data during ingestion
Pre-computed Fields - Prevalence values are calculated and stored as enrichment fields
Filtering - Detection rules filter on these enrichment fields using count thresholds

Available Prevalence Fields:

hash_prevalence - Count of how often a file hash has been seen
domain_prevalence - Count of how often a domain has been accessed
hash_first_seen - Timestamp when hash was first observed
domain_first_seen - Timestamp when domain was first observed

Prevalence Thresholds (Count-based):

<5 - Very rare, high suspicion
<10 - Uncommon, moderate suspicion
<50 - Occasional, low suspicion
50+ - Common, baseline activity

Using Prevalence in Rules

Basic Prevalence Filtering:

# Detect rare process executions (requires hash enrichment)
process_name="*.exe" | prevalence hash_prevalence < 5

# Find uncommon domain access (requires domain enrichment)
dns_query=* | prevalence domain_prevalence < 10

# Identify newly seen hashes
process_name="*.exe" | prevalence hash_first_seen > now() - 24h

Advanced Prevalence Analysis:

# Combine prevalence with other indicators
process_name="powershell.exe" 
| prevalence hash_prevalence < 10 
| where command_line CONTAINS "-enc"

# Multi-condition prevalence filtering
dns_query=* 
| prevalence domain_prevalence < 5 
| prevalence domain_first_seen > now() - 7d
| stats count by src_ip 
| where count > 3

Alternative: Manual Prevalence Calculation If enrichment is not configured, calculate prevalence inline:

# Calculate rare process hashes manually
process_name="*.exe" 
| stats count by process_hash 
| where count < 5

# Find uncommon domains manually
dns_query=* 
| stats count by dns_query 
| where count < 10

Configuration:

Prevalence data is computed during log enrichment
Enrichment fields are added to matching log records
Detection rules filter on these pre-computed fields
Requires enrichment pipeline configuration

Cross-Telemetry Detections (Logs · Traces · Metrics)

Detection rules aren't limited to logs. A rule carries a dataset — logs (default), spans (OpenTelemetry traces), or metrics — and the engine routes it to the right table (logs / otel_spans / otel_metrics) and schema at evaluation time. A rule can also correlate across datasets in a single query, using a shared key (trace_id, service_name, user, src_ip) to marry app behavior (traces), security verdicts (logs), and system state (metrics).

The signal in these detections lives in the contradiction between layers — something a logs-only SIEM or a traces-only APM each can't see on its own.

How it works

Set the rule's Dataset to Traces or Metrics. Spans/metrics rules are scheduled-only (real-time materialized views are logs-only).
Set a risk_entity_field that exists on that dataset — e.g. service_name for spans (the entity isn't auto-detected on non-log datasets).
Correlate to another dataset with <key> IN [dataset=<name> … | return <key>] (filter) or | join <key> [dataset=<name> …] (merge — brings the other side's fields onto the row). See join.
Use a generous lookback (e.g. 15 min): each run is time-bounded, and both sides of the correlation must land in the same window.

Example: privileged operation succeeded despite a failed auth check

Broken access control / auth bypass (MITRE T1068 / T1190). The trace knows the operation hit a privileged endpoint and returned success; the security log knows that exact request failed authentication. Joined on trace_id, the contradiction is the finding.

# Dataset: Traces · risk_entity_field: service_name · Alerting · Critical
status_code="OK"
span_name=/.*(admin|\/users\/|password|token|role|grant).*/
trace_id IN [dataset=logs auth_result="failure" | return trace_id]

Enrich the alert with who by swapping the filter for a join:

status_code="OK" span_name=/.*(admin|password|token).*/
  | join trace_id [dataset=logs auth_result="failure"]

Example: data egress with a just-issued token

Exfiltration where the request's session was minted seconds earlier (stolen/forged token). Span = the large transfer; log = the suspiciously fresh credential, joined on trace_id.

# Dataset: Traces · risk_entity_field: service_name
bytes_out>10000000
trace_id IN [dataset=logs action="token_issued" | return trace_id]

Example: a service calling something it shouldn't (lateral movement)

A span path that should never exist in the mesh, whose request also touched an IOC-flagged log.

# Dataset: Traces
service_name="payments" peer.service="admin-db"
trace_id IN [dataset=logs ioc_confidence>0 | return trace_id]

Example: auth-endpoint error burst confirmed by a metrics spike

Credential stuffing that actually impacts the service — error-burst spans on /login correlated to a CPU/latency spike (spans ⋈ metrics on service_name), so you alert on impactful brute force, not scattered failures.

# Dataset: Traces
status_code="ERROR" span_name=/.*\/login.*/
service_name IN [dataset=metrics metric_name="system.cpu.utilization" value>0.9 | return service_name]

Notes

The correlation subsearch is bounded (10,000 rows by default; raise with maxout on a join) — keep the subsearch selective so the correlation set is small.
A cross-dataset correlation requires a join key; a keyless cross-dataset join is rejected.
Alerts flow into the same unified inbox as log detections, with the same lifecycle, risk scoring, and case integration.

AI Auto-Tuning System

Automated Rule Optimization

The AI auto-tuning system continuously improves detection rules:

Tuning Process:

Baseline Collection - Gather 7-14 days of alert data
Pattern Analysis - Identify false positive patterns
Threshold Optimization - Adjust numeric thresholds
Filter Enhancement - Add exclusion conditions
Validation Testing - Verify improvements against historical data

Tuning Triggers:

High false positive rates (>20%)
Excessive alert volume (>100/day)
Analyst feedback patterns
Scheduled optimization reviews

Tuning Capabilities

Threshold Adjustment:

# Original rule
failed_logins | stats count by user | where count > 5

# AI-tuned version
failed_logins | stats count by user | where count > 15

Filter Enhancement:

# Original rule
process_name="powershell.exe"

# AI-enhanced version
process_name="powershell.exe" 
| where NOT (parent_command_line="explorer.exe" AND user LIKE "admin_%")

Prevalence Integration:

# Original rule
network_connection dest_port=443

# Prevalence-enhanced version (requires domain enrichment)
network_connection dest_port=443 | prevalence domain_prevalence < 5

Tuning Dashboard

Monitor auto-tuning activities:

Metrics Tracked:

Rules under active tuning
False positive reduction rates
Alert volume changes
Tuning success rates

Tuning History:

Version timeline for each rule
Before/after performance comparisons
Rollback capabilities
Analyst approval workflows

Detection Performance & Monitoring

Key Metrics

Detection Latency:

Time from log ingestion to alert generation
Target: <5 minutes for scheduled rules, <30 seconds for materialized views
Factors: Query complexity, data volume, system resources

Rule Performance:

Execution time per rule
Resource consumption patterns
Success/failure rates
Query optimization opportunities

Alert Quality:

True positive rates by rule
False positive trends
Mean time to resolution
Analyst feedback scores

Performance Optimization

Query Optimization:

# Inefficient - scans all data
* | where process_name="powershell.exe"

# Optimized - uses indexed fields first
process_name="powershell.exe" | where command_line CONTAINS "-enc"

# Further optimized - time bounds
process_name="powershell.exe"
| where timestamp > now() - 15m
| where command_line CONTAINS "-enc"