nano v2.0.17
nano v2.0.17 fixes an ingest stall from deadlocked ClickHouse async-insert flushes, alerts on stuck flushes and extends OCSF prevalence to more fields.
v2.0.17 includes everything in v2.0.15. It is not flagged as a security release, but every v2.x install should upgrade. It fixes an ingest stall that stops all ingest with no alert.
v2.0.16 was not released. Its build was tagged, but the release step stopped before the bundle was published. v2.0.17 is the same code plus a test fix. Upgrade from v2.0.15 straight to v2.0.17.
To upgrade, follow Upgrade a self-hosted deployment. On Docker Compose, scripts/nano upgrade upgrades v2.0.15 to v2.0.17 directly. v2.0.10 through v2.0.13 also upgrade directly; there are no required stops. If ingest is stuck right now, read Recover a stuck install first.
Ingest stall fix
Under load, two ClickHouse async-insert flushes could deadlock. v2.x capped the ingest workload at 4 CPU slots. Each flush, and each materialized view inside it, takes its own slot. Two flushes could hold all 4 slots and wait on each other forever.
When that happened, ingest stopped completely. Nothing alerted. Every later write failed with ClickHouse code 202 until ClickHouse restarted.
v2.0.17 fixes it:
- The cap is removed. ClickHouse migration 198 drops the
ingestworkload's slot cap. It keeps the workload's weight, so ingest still gets CPU first. The migration also releases a stall that is in progress. - Stuck flushes are reported. A new
ingest_stallwatchdog alert fires when a flush runs longer than 5 minutes, or when the ingest slots stay full with waiters for a minute. It is logged at ERROR withalert="ingest_stall". Settings → SIEM health shows stuck flushes as a critical finding. - Migrations retry code 202. The ClickHouse migrator retries with backoff when ClickHouse is briefly saturated. If it gives up, it says how many inserts are stuck and for how long.
async_insert_threadsis 4. It was 2 on smaller sizes.
Recover a stuck install
Signs of a stall: no new events in search, and inserts failing with code 202. To release it before you upgrade, run this as the ClickHouse admin user:
CREATE OR REPLACE WORKLOAD ingest IN nano SETTINGS weight = 10On a cluster, add ON CLUSTER <your cluster> after ingest. Ingest resumes without a restart. Upgrading to v2.0.17 applies the same change.
New
- Acknowledgement durability setting.
NANO_INGEST_ACK_DURABILITYon Docker Compose, orvector.ackDurabilityon Helm, sets when an ingest2xxis sent.fsync, the default, answers after the event is synced to disk. Nothing changes from v2.0.15.page_cacheanswers once the event is in the page cache. It takes about 1.7–2.2x the request rate per Vector. A power loss or host crash can lose up to about 500 ms of acknowledged events.- Choose
page_cacheonly when senders can tolerate that loss or re-send on their own.
- Two-phase log-source activation (opt-in). With stable fan-in slots on (
VECTOR_FANIN_SLOTS=on), a new log source is staged on every ingest host first. It goes live only after every host has it. A failure in either phase rolls the deploy back, so hosts never route differently. Deploy jobs show a new activating phase. SetNANO_VECTOR_TWO_PHASE_ACTIVATION=offto turn it off. - Shodan domain lookups. The built-in Shodan provider now answers domains. It resolves the domain, then looks up the address. The resolve step costs no Shodan query credits. Private and reserved addresses are never sent.
Fixes and improvements
- Vector 0.61.0-nano.11. Every Compose file, lock and the Helm chart pin it. It fixes CrowdStrike FDR ingest.
- OCSF prevalence.
- Parent-process hashes (
actor.process.file.hashes) now get host counts. - HTTP and DNS hosts (
http_request.url.hostname,url.hostname,query.hostname) now get domain prevalence. - Every prevalence view uses one shared artifact list: search enrichment, the Rarity view, the drawer and the scatter. A domain is no longer read from
device.hostname.
- Parent-process hashes (
- OCSF field suggestions. On OCSF, pivt, playbook Where fields and case-notebook filters suggest OCSF paths, such as
src_endpoint.ipandactor.user.name. Typed UDM names still work as aliases. - Rule coverage by source. Rule-repository coverage checks each rule's fields against the fields each source actually carries. Before, a rule that named any field read as not covered on UDM, and every rule read as covered on OCSF.
- Sparse audit feeds. Google Workspace, Okta, Microsoft 365, Entra, Slack and GitHub audit sources get a 12–24 hour stale threshold. They no longer raise false stale-feed alerts overnight. A threshold you set yourself is kept.
- pivt says when answers aren't docs-grounded. Without docs search, pivt says its answer isn't grounded in the nano docs. It no longer invents doc links or settings.
- Log Source tooltips. Publish says the draft goes live for incoming events. Save VRL says incoming events aren't affected until you publish.
- Content co-signer. A bad or partial co-signing configuration no longer stops the content publisher. It logs a warning and runs without co-signing.
Upgrade notes
- All v2.x installs should upgrade. Any install under steady load can hit the ingest stall.
- Helm and
--reuse-values. An upgrade with--reuse-valueskeepsasync_insert_threadsat 2. That is harmless: removing the cap is the real fix, and migration 198 applies either way. To pick up 4, pass your values files with-finstead. See Kubernetes: Upgrades. - Don't add a thread cap back. If you manage ClickHouse workloads yourself, don't set
max_concurrent_threadson theingestworkload.
nano v2.0.18
nano v2.0.18 verifies every online license check-in, stops parser publish rollbacks on slow Vector reloads and shows OCSF fields in alerts and asset views.
nano v2.0.15
nano v2.0.15 runs custom enrichments and collectors in a WebAssembly sandbox, makes an ingest 2xx mean synced to disk and speeds up PostgreSQL HA failover.