Search Concurrency
Configure search query admission control, per-user limits, and ClickHouse resource settings
Search Concurrency
nano includes an admission controller that manages concurrent search queries. This prevents any single analyst from monopolizing ClickHouse resources and ensures detection rules always have priority over ad-hoc searches.
Overview
Without concurrency limits, a single analyst running many heavy threat-hunting queries could starve detection rules and other analysts of resources. The admission controller solves this with:
- Priority-based scheduling — Detection rules bypass the queue entirely
- Per-user limits — No single analyst can consume all available query slots
- Global limits — Total concurrent ad-hoc queries are capped
- Queue with timeout — When limits are hit, queries wait in a priority queue rather than failing immediately
- Per-query resource limits — ClickHouse settings (timeout, memory, threads) are applied per priority tier
Priority Levels
Queries are assigned a priority that determines their scheduling behavior:
| Priority | Level | Queue Behavior | Use Case |
|---|---|---|---|
| Detection | 0 (highest) | Bypasses queue, always runs | Scheduled detection rules |
| Near-Realtime | 1 | Bypasses queue, always runs | Real-time materialized view rules |
| Interactive | 2 | Subject to limits, queued if full | Analyst searches in the Search page |
| Analytics | 3 (lowest) | Subject to limits, queued if full | Long-running threat hunting |
Detection and Near-Realtime queries are never queued — they always execute immediately. This guarantees that detection rules are never delayed by analyst activity.
Concurrency Limits
These settings control how many ad-hoc queries (Interactive + Analytics) can run simultaneously:
| Setting | Default | Range | Description |
|---|---|---|---|
| Global ad-hoc limit | 30 | 1–500 | Maximum concurrent ad-hoc queries across all users |
| Per-user limit | 5 | 1–50 | Maximum concurrent queries per individual user |
| Max queue depth | 100 | 10–1,000 | Maximum number of queries waiting in the queue |
| Queue timeout | 120s | 10–600s | How long a query waits before being rejected |
When the global limit is reached, new queries enter a priority queue. If the queue is also full, the query is rejected with a QueueFull error. If a queued query waits longer than the timeout, it is rejected with a QueueTimeout error.
How the queue works
- A new search request arrives
- The admission controller checks the per-user limit — if exceeded, the request is rejected
- If the global limit has capacity, the query starts immediately
- If the global limit is full, the query enters the priority queue (ordered by priority level, then arrival time)
- When a running query completes, the next queued query (highest priority first) is started
- If the queue timeout expires before a slot opens, the query is rejected
ClickHouse Resource Settings
Each priority tier has configurable ClickHouse query settings. These are applied per-query via HTTP parameters — no ClickHouse-side profile setup is needed.
| Setting | Detection | NRT | Interactive | Analytics | Description |
|---|---|---|---|---|---|
max_execution_time | 30s | 30s | 300s | 3600s | Query timeout in ClickHouse |
max_memory_usage | 5 GB | 5 GB | 20 GB | 50 GB | Per-query memory cap |
max_threads | 8 | 8 | 16 | 32 | Parallel execution threads |
priority | 1 | 1 | 3 | 5 | ClickHouse scheduler priority (1 = highest) |
queue_max_wait_ms | 5s | 5s | 60s | 120s | ClickHouse-side queue timeout |
The Interactive, Analytics, and Realtime tiers are configurable via Settings > Search Concurrency. Detection and NRT settings use the Realtime tier values.
Configuration
Settings UI
Navigate to Settings > Search Concurrency to adjust limits. Changes take effect immediately — no restart required.
The settings page has two sections:
- Concurrency Limits — Global and per-user query limits, queue depth, timeout
- ClickHouse Resource Limits — Per-priority-tier execution time and memory caps
Environment Variables
Environment variables serve as startup defaults. Once the system boots, admin settings from the database override these:
| Variable | Default | Description |
|---|---|---|
SEARCH_GLOBAL_ADHOC_LIMIT | 30 | Initial global ad-hoc limit |
SEARCH_PER_USER_LIMIT | 5 | Initial per-user limit |
SEARCH_MAX_QUEUE_DEPTH | 100 | Initial max queue depth |
SEARCH_QUEUE_TIMEOUT_MS | 120000 | Initial queue timeout (ms) |
Queue Behavior in the UI
When a search is queued, the Search page shows:
- A clock icon with "Search queued — Position N in queue"
- An estimated wait time (based on average recent query duration)
- A cancel button to remove the search from the queue
When the search moves from queued to running, the UI transitions to the standard progress bar with rows scanned.
Active Searches Panel
The Search page includes a collapsible Active Searches panel that shows all your current and recent searches. From this panel you can:
- See the status of each search (queued, running, completed, failed)
- Cancel queued or running searches
- Click completed searches to load their results
Notifications
When an async search completes or fails, you receive a notification in the bell icon:
- Search completed — Shows result count, click to load results
- Search failed — Shows error message
Monitoring
The admission controller exposes stats via the search jobs API:
GET /api/search/jobsReturns your active and recent search jobs with status, queue position, and elapsed time.
Related
- Query Safety Limits — Per-query OOM protection settings (array aggregation caps, mvexpand limits, cost analysis enforcement)