Push vs Pull Routing
How nano routes events to parsers, and why push and pull ingestion shapes need different routing rules
Choose routing rules that match how your events arrive so nano runs the right parser. Each event needs a source (source_type on the wire); the parser maps it to OCSF and stores the source as metadata.log_source.
Push senders and pull bindings supply that source differently.
The rest of this page is about pull sources, where routing rules split one binding into several source types.
Use the decision tree to choose a Pub/Sub configuration, then adapt the examples for your pull driver.
The Two Ingestion Shapes
Push senders authenticate with a token that decides the source. Pull routing applies when nano reads a broker binding.
Push (HTTP, HEC, OTLP)
- The token decides the source. Each sender authenticates with an ingest token bound to one log source, and every event it sends lands there. A declared
X-Source-Type, HECsourcetypeor.source_typedoesn't change it. - N source types means N tokens: a forwarder that carries several log types uses one sink (or output) per type, each with the token bound to that type's source.
- Push sources need no routing rules.
Pull (GCP Pub/Sub, Apache Kafka, AWS S3 via SQS)
- One source configuration = one binding (subscription / topic / queue). The sender cannot add a header to a Pub/Sub message attribute or a Kafka header on the consumer side; those have to be set by the publisher, before the message lands in the broker.
- No inbound
.source_type: a Pub/Sub event arriving at nano has.message,.attributes, and broker metadata, but no.source_typefield. A routing rule that matchesmatch_field=source_type, match_type=exact, match_value=foocan never fire on a pull source. It would require the field to exist on the event already, which it doesn't. - Multiplexing within one binding requires either:
- A publisher-set broker attribute (Pub/Sub message attribute, Kafka header, SQS message attribute) that the routing rule reads, or
- Content sniffing of a field inside the decoded message body, or
- Single source type for the whole feed: every event in this binding is the same source type (the mono-vendor case).
Per-Driver Routing Field Summary
The match field a routing rule reads from depends on the driver. The natural broker-native field, the recommended attribute-based field, and the content-sniff path are all listed below.
| Driver | Broker-native field | Recommended attribute path | Content-sniff example |
|---|---|---|---|
| GCP Pub/Sub | subscription (one per source config, not a routing rule) | .attributes.source_type | .message.routing.event_type |
| Apache Kafka | .topic (when fan-out within consumer group) | .headers.source_type | .message.event_type |
| AWS S3 via SQS | .key (S3 object key, e.g. AWSLogs/.../CloudTrail/) or .bucket | .attributes.source_type (SQS message attribute set by producer) | path-based on .key |
| HEC | The token's source (legacy multiplex install token: sourcetype) | n/a | n/a |
| HTTP (push) | The token's source (legacy multiplex install token: X-Source-Type) | n/a | n/a |
| Vector native (push) | vector_native (upstream .source_type only with NANO_VECTOR_NATIVE_TRUST_SOURCE_TYPE=true) | n/a | n/a |
Decision Tree: "I Have N Log Sources via Pub/Sub"
Choose the configuration based on how the publisher exports logs.
N Topics (Most Common)
If your environment has one Pub/Sub topic per log type (nano-audit-logs, nano-vpc-flowlogs, nano-scc-findings), create one Source Configuration per subscription. Each binds to one subscription and tags every event with one source type. Each topic uses its own source configuration.
One Topic, N Log Types — Publisher Sets Attributes
If you control the publisher and it can set Pub/Sub message attributes, set a source_type attribute (or any agreed key) on each message before publishing. Then in the Source Configuration, switch to Multiple source types mode and add routing rules with match_field = .attributes.source_type.
One Topic, N Log Types — Third-Party Publisher
Some SaaS exporters (LimaCharlie's outbound Pub/Sub, for example) publish all event types to one topic and don't expose a way to set per-message attributes. In that case:
- You actually need split routing (e.g. you want
vendor_edr_processandvendor_edr_networkto use different parsers): use Multiple source types mode and content-sniff a field inside the decoded message body, e.g.match_field = .message.routing.event_type. - You don't need split routing (e.g. one parser handles every LimaCharlie event): use Single source type mode. Every event in the subscription gets tagged
limacharlieand routed to that parser. Use this mode when one parser handles the entire feed.
Per-Driver Examples
Use these examples to configure each pull driver. The HEC example shows source declarations for push events.
GCP Pub/Sub — Single Source Type (LimaCharlie)
Scenario: LimaCharlie publishes all event types to one Pub/Sub topic. You don't need per-event-type routing; one parser handles every LimaCharlie event.
- Create the subscription and credential per the GCP Pub/Sub setup guide.
- Create a Source Configuration of type
gcp_pubsub:- Project ID:
your-gcp-project - Subscription:
nano-limacharlie-sub - Credential: the GCP service account credential
- Project ID:
- In the routing section, choose Single source type:
- Source type:
limacharlie - Parser: the bundled LimaCharlie OCSF parser, which claims
limacharlie
- Source type:
This produces one default routing rule and the generator emits an unconditional tag:
.source_type = "limacharlie"GCP Pub/Sub — Multiple Source Types via Message Attributes
Scenario: Your own producer publishes events from multiple internal services to one Pub/Sub topic and sets a source_type message attribute on each message.
Publisher (Python google-cloud-pubsub):
from google.cloud import pubsub_v1
publisher = pubsub_v1.PublisherClient()
topic_path = publisher.topic_path("your-gcp-project", "internal-events")
publisher.publish(
topic_path,
data=event_json.encode("utf-8"),
source_type="auth_service", # message attribute
)Source Configuration routing rules (Multiple source types mode):
| Match Field | Match Type | Match Value | Target Source Type |
|---|---|---|---|
.attributes.source_type | exact | auth_service | auth_service |
.attributes.source_type | exact | api_gateway | api_gateway |
.attributes.source_type | exact | payment_service | payment_service |
| — | default | — | unknown_internal |
The generator emits:
if .attributes.source_type == "auth_service" {
.source_type = "auth_service"
} else if .attributes.source_type == "api_gateway" {
.source_type = "api_gateway"
} else if .attributes.source_type == "payment_service" {
.source_type = "payment_service"
} else {
.source_type = "unknown_internal"
}GCP Pub/Sub — Content Sniffing (Third-Party SaaS, One Topic, N Event Types)
Scenario: A third-party exporter publishes events from multiple event categories to one topic. You can't change the publisher, but the body contains a discriminator field.
Sample event body (decoded .message):
{
"routing": {
"event_type": "NEW_PROCESS",
"hostname": "web-01"
},
"event": { "image": "powershell.exe", "cmdline": "..." }
}Source Configuration routing rules:
| Match Field | Match Type | Match Value | Target Source Type |
|---|---|---|---|
.message.routing.event_type | exact | NEW_PROCESS | vendor_edr_process |
.message.routing.event_type | exact | NEW_CONNECTION | vendor_edr_network |
.message.routing.event_type | prefix | DNS_ | vendor_edr_dns |
| — | default | — | vendor_edr_other |
Apache Kafka — One Topic per Log Type
Scenario: You already use a topic-per-log-type convention.
Create one Source Configuration per Kafka topic. Each one uses Single source type mode and tags every event with its source type. The consumer group is shared.
cloudtrail-eventstopic → Source Configurationaws_cloudtrailokta-eventstopic → Source Configurationoktaapp-logstopic → Source Configurationmy_app
Apache Kafka — One Topic, N Log Types via Headers
Scenario: A shared security-events topic carries multiple log types and the producer sets a source_type Kafka record header.
Producer (Java kafka-clients):
ProducerRecord<String, String> record =
new ProducerRecord<>("security-events", null, eventJson);
record.headers().add("source_type", "okta".getBytes(UTF_8));
producer.send(record);Source Configuration routing rules (Multiple source types mode):
| Match Field | Match Type | Match Value | Target Source Type |
|---|---|---|---|
.headers.source_type | exact | okta | okta |
.headers.source_type | exact | aws_cloudtrail | aws_cloudtrail |
.headers.source_type | prefix | app_ | application_logs |
| — | default | — | generic_kafka |
Apache Kafka — Routing on Topic Name
Scenario: A single nano consumer subscribes to multiple topics in one consumer group, and you want to route on .topic (the broker-native field) rather than headers.
| Match Field | Match Type | Match Value | Target Source Type |
|---|---|---|---|
.topic | exact | cloudtrail-events | aws_cloudtrail |
.topic | exact | okta-events | okta |
.topic | prefix | app- | application_logs |
| — | default | — | generic_kafka |
The generator emits if .topic == "cloudtrail-events" { .source_type = "aws_cloudtrail" } etc.
AWS S3 via SQS — Routing on Object Key Prefix
Scenario: A single S3 bucket holds multiple log types under different prefixes (AWSLogs/.../CloudTrail/, AWSLogs/.../vpcflowlogs/, WAFLogs/) and one shared SQS queue receives notifications for all of them.
Source Configuration routing rules (Multiple source types mode):
| Match Field | Match Type | Match Value | Target Source Type |
|---|---|---|---|
.key | prefix | AWSLogs/123456789012/CloudTrail/ | aws_cloudtrail |
.key | prefix | AWSLogs/123456789012/vpcflowlogs/ | aws_vpc_flow |
.key | prefix | WAFLogs/ | aws_waf |
| — | default | — | aws_s3_other |
The .key match field reads the S3 object key (in the routing-rule UI it's the "Object key" preset; the default preset is .bucket). Producer-controlled SQS attribute alternative: if your application code wraps writes through Lambda or another shim, you can set an SQS message attribute on publish and route on .attributes.source_type instead, exactly like the Pub/Sub attribute example.
HEC — Routing on sourcetype (Push, for Contrast)
HEC is a push protocol, included here for contrast. With per-source tokens the token decides the source and these rules aren't consulted. They apply only to the install token in the deprecated legacy multiplex mode, where the HEC client sets sourcetype on each event, and nano routes on the canonical .sourcetype field. The match shape looks similar to a pull-source attribute match, but the field is set by the sender per-event, not by a publisher attribute.
| Match Field | Match Type | Match Value | Target Source Type |
|---|---|---|---|
sourcetype | exact | WinEventLog:Security | windows_event |
sourcetype | exact | pan:traffic | palo_alto |
sourcetype | prefix | myapp: | my_app |
| — | default | — | unknown_hec |
Prefer one token per source over HEC routing rules: the token can't be spoofed by a sender's sourcetype.
When to Create a New Source Configuration vs Add a Routing Rule
Choose a new binding or a routing rule based on what changes:
| Situation | Create a new Source Configuration | Add a routing rule |
|---|---|---|
| New Pub/Sub subscription / Kafka topic / SQS queue | Yes | No |
| New credential / different cloud account | Yes | No |
| Same binding, new log type (publisher sets an attribute or there's a content discriminator) | No | Yes (Multiple source types mode) |
| Same binding, mono-vendor feed | No | Use Single source type mode (one default rule) |
| New source type from an existing push forwarder | No | No: issue a token bound to the new source |
A Source Configuration models a binding (one subscription, one topic, one queue, or one HTTP listener). Routing rules model how events inside that binding are split into source types. If you find yourself wanting to split across bindings, that's a new Source Configuration.
Source Configuration UI
Routing rules are configured per source configuration in the nano web UI: go to Ingestion → Log Sources → Cloud pull, open the pull source, and use the routing section. Pull-source configurations expose two modes, Single source type (one default rule) and Multiple source types (the routing-rules table), with per-driver match-field defaults.
Troubleshoot
Need help with these checks? Contact support with the deployment, request ID and time with timezone.
A pull routing rule never matches
Check that the match field exists on incoming events. Pull events do not arrive with push-style source_type; use a broker attribute, a decoded content field, or Single source type mode.
Push events land in the wrong source
The token decides the source. Check which source the sender's token is bound to under Ingestion → Log Sources → Credentials, and issue a token bound to the right source.
What's next
- Ingest data: endpoints, pipeline and the source requirement
- Supported Data Sources: protocol-level reference for every ingestion method
- GCP Pub/Sub: end-to-end setup, sample events, troubleshooting
- Apache Kafka: bootstrap, SASL, multiple topics, routing rules table; AWS S3 via SQS: bucket notifications, per-prefix vs shared queue; HEC: sourcetype routing, common vendor sourcetype → nano mappings