Ingest data
How events get into nano, how an ingest token decides the source and its parser, and which integration to use
Choose an ingestion integration to send logs to nano and use them in searches and detections. Every deployment receives events with Vector, parses them with the source's parser, and stores them as OCSF events in ClickHouse.
The event arrives with a source that selects a parser. The parser turns the raw log into an OCSF event before it is written to storage.
What differs between deployments is where senders connect. See Deployment models for the full comparison.
How ingestion works
- Authenticate. Every push request carries an ingest token that you issue in nano. See Ingest tokens.
- Pick the source. The token decides the source: every event sent with it lands in the log source it is bound to, whatever
sourcetypeorX-Source-Typethe sender declares. Pull sources name their source in their configuration. See Push vs Pull Routing. - Parse. The parser whose
match_valuesclaim the source runs its VRL program and emits an OCSF event: the full OCSF object at the root, the original line inraw_data, and source fields with no OCSF home underunmapped. See OCSF schema. - Store. Vector writes the event to ClickHouse. The source name is in
metadata.log_source.
Find events from one source:
metadata.log_source="okta" | head 20See how much each source sent in the last hour:
earliest=-1h | stats count by metadata.log_sourceEvents that arrived without a source carry metadata.log_source="unknown".
Ingest endpoints
Self-run on Kubernetes
These deployments receive events on a dedicated ingest frontend with its own hostname, for example ingest.example.com, so a sender flood can't take the UI or API down. The ingest host serves only:
| Protocol | Endpoint | Auth |
|---|---|---|
| HTTP (JSON / NDJSON) | https://<ingest-host>/ingest | Authorization: Bearer <token> |
| HEC | https://<ingest-host>/services/collector (also /event, /raw, /ack) | Authorization: Splunk <token> |
| OTLP/HTTP | https://<ingest-host>/otlp/v1/logs, /v1/traces, /v1/metrics | Authorization: Bearer <token> |
The app host answers 404 on all three. HEC indexer acknowledgements are pinned to one Vector instance per channel: send every event and every /services/collector/ack poll of a channel with the same X-Splunk-Request-Channel header (or ?channel=). See HEC.
Open core and Self-run on Docker Compose
Compose installs serve ingest from the same host as the app. A dedicated ingest frontend for Self-run on Compose is coming:
| Protocol | Address | Auth |
|---|---|---|
| HTTP (JSON / NDJSON) | http(s)://<host>/ingest through nginx, or Vector on :8080 | Authorization: Bearer <token> |
| HEC | http://<host>:8088/services/collector/event (or /raw) | Authorization: Splunk <token> |
| OTLP/HTTP | http(s)://<host>/otlp/v1/logs, /v1/traces, /v1/metrics through nginx | Authorization: Bearer <token> |
| Raw OTLP | gRPC :4317, HTTP :4318 | None; bound to 127.0.0.1 by default |
| Vector native | :6000 | Mutual TLS from v2.0.30: a client certificate per sender. See Vector native |
| Fluent Forward | :24224 | None; no source by default, bound to 127.0.0.1 |
The unauthenticated listeners (raw OTLP, Fluent Forward and the optional bundled Tenzir node on :9095) bind to 127.0.0.1 unless you set NANO_OTLP_BIND_ADDRESS, NANO_FLUENT_BIND_ADDRESS or NANO_TENZIR_BIND_ADDRESS in .env. Vector native (:6000) requires a client certificate from v2.0.30 and listens on every interface; on v2.0.29 and earlier it was unauthenticated, on 127.0.0.1 by default. Expose them only on a trusted network, and firewall them to your log sources. Authenticated senders should use /ingest, HEC or /otlp.
- HTTP ingest refuses requests without a valid token, unless you set
VECTOR_ALLOW_UNAUTHENTICATED_INGEST=true. - An HTTP
200means Vector accepted the event into its buffer, not that it has been written to ClickHouse. - Syslog ships disabled. Enable Vector's syslog source, or collect it with a Vector aggregator or the
connectCLI.
Ingest tokens
Issue a named token for each sender under Ingestion → Log Sources → Credentials (/api/ingest-tokens). Every token is bound to exactly one log source when you create it, and the token decides the source: an event sent with it lands in that source whatever sourcetype or X-Source-Type it declares.
- The token value is shown once, in the create or rotate response. It is never logged or readable again.
- Rotate issues a new value with the same name and source and revokes the old value. Revoke stops a token. A binding never changes: to move a sender to another source, revoke its token and issue a new one.
- A token whose source was deleted is unbound and refused until an admin binds it.
- Vector reloads the hashed token list every 2 seconds, so issuing, rotating or revoking a token takes effect within seconds, without restarting Vector.
- One token can authenticate HTTP, HEC and OTLP/HTTP for its source.
The install token (VECTOR_AUTH_TOKEN in .env) is not an ingest credential on a fresh install: it is accepted only while it is bound to a source. Installs upgraded from before per-source tokens keep it in a deprecated legacy multiplex mode, where the sender's X-Source-Type or HEC sourcetype picks the source. Move each sender to its own token and then leave legacy mode under Credentials.
Test HTTP ingest with a token bound to your source:
curl -sS --fail-with-body "https://<ingest-host>/ingest" \
-H "Authorization: Bearer $NANO_TOKEN" \
-H "Content-Type: application/json" \
-d '{"time": "2026-01-01T12:00:00Z", "level": "INFO", "message": "User login", "user": "john"}'On open core, use http(s)://<host>/ingest instead.
Push sources
| Integration | Transport | Common log types |
|---|---|---|
| HEC | HEC | Anything from Cribl, Fluent Bit or the OpenTelemetry Collector |
| OpenTelemetry (OTLP) | OTLP/HTTP | Application logs; traces and metrics to their own tables |
| HTTP | POST /ingest with a token | Applications, webhooks, Fluentd, Filebeat |
| Cribl Stream | HEC, HTTP or OTLP | Firewalls, identity providers, endpoints, cloud audit |
| Vector Aggregator | HEC sinks, or Vector native | Syslog, files and other sources collected on your network |
| On-premise collection | Edge collectors forwarding to nano | Firewalls, switches, servers |
Pull sources
nano connects to cloud queues and SaaS APIs and pulls log data directly, with no agent to install.
| Integration | Transport | Common log types |
|---|---|---|
| AWS S3 via SQS | S3 bucket notifications → SQS queue | CloudTrail, VPC Flow Logs, GuardDuty, ALB, WAF |
| AWS SQS | SQS message bodies | Events your producer writes to a queue |
| CrowdStrike FDR | Falcon Data Replicator: SQS batch manifests → S3 event files | Falcon process, network, DNS, file and detection telemetry |
| GCP Pub/Sub | Log Router sink → Pub/Sub subscription | Cloud Audit Logs, VPC Flow Logs, GKE |
| Azure Event Hubs | Azure Monitor diagnostic setting → Event Hub | Activity Logs, Entra sign-in and audit logs |
| Kafka | Kafka consumer group | Application logs, security events, streaming pipelines |
| SaaS Collectors | Scheduled pull from a vendor's REST API | Okta, Google Workspace, Slack, Netskope |
Cloud queues and brokers are read by Vector with credentials you own; manage them under Ingestion → Log Sources → Cloud pull. See Cloud pull. Cloud credentials are encrypted at rest with AES-256-GCM. The API never returns secret values after creation, and a credential can't be deleted while a source uses it.
Events already shaped to OCSF
If your own pipeline (Tenzir, Cribl or a custom forwarder) already emits OCSF, see Direct OCSF ingestion.
Onboard a new source
- Check for a parser. nano ships parsers for common products; see Supported sources. If one claims your source, issue a token for it, send data and verify the result.
- Build a parser if needed. Open Ingestion → Log Sources, create a source, and paste three to five representative log lines. The AI parser builder proposes VRL that maps the log to OCSF, and the test runs it against your samples and validates the output against the OCSF contract.
- Publish. Publishing makes an immutable version of the parser and deploys it to Vector. Earlier versions stay in the source's history, and you can revert to one.
- Send data and verify with a
metadata.log_sourcesearch, as above.
To adjust a shipped parser for one source without forking it, use a parser extension. To filter, redact, sample or route events before they are stored, use a pipeline. The step-by-step walkthrough is in Set up your first feed.
Choosing an integration
| I want to ingest... | Use |
|---|---|
| Logs from Cribl Stream | Cribl Stream |
| Data from HEC-compatible senders | HEC |
| OpenTelemetry logs, traces and metrics | OTLP |
| Application logs over HTTP or webhooks | HTTP POST /ingest with an ingest token |
| Syslog from firewalls, switches, servers | connect CLI or Vector Aggregator |
| AWS CloudTrail, VPC Flow Logs, GuardDuty | AWS S3 via SQS |
| Google Cloud audit and security logs | GCP Pub/Sub |
| Azure Monitor and Entra logs | Azure Event Hubs |
| Logs already in Kafka | Kafka |
| Okta, Google Workspace, Slack, Netskope | SaaS Collectors |
Troubleshoot
Need help with these checks? Contact support with the deployment, request ID and time with timezone.
| Symptom | Check |
|---|---|
401 | The token and header scheme: Authorization: Bearer <token> for HTTP and OTLP; Authorization: Splunk <token> for HEC. Check that the token isn't revoked or unbound. |
| Events land in the wrong source | The token decides the source. Issue a token bound to the right source; a declared sourcetype doesn't move events. |
Events land as unknown | An install token in legacy multiplex mode with no X-Source-Type or sourcetype. Use a per-source token. |
| Events arrive but fields are missing | No parser claims the source, or the parser doesn't map them. Look under unmapped and in raw_data, then build or extend the parser. |
404 on the app host | Self-run on Kubernetes serves ingest only on the ingest host. |
429 or dropped events from one busy sender | HTTP ingest is throttled per client IP (5,000 events per second in the default Vector config). Spread load across senders or back off and retry. |
Monitor each source's volume and errors on Ingestion → Log Sources.