nano SIEM
Integrations

Push vs Pull Routing

Supported in:Self-hosted

How nano routes events to parsers, and why push and pull ingestion shapes need different routing rules

Choose routing rules that match how your events arrive so nano runs the right parser. Each event needs a source (source_type on the wire); the parser maps it to OCSF and stores the source as metadata.log_source.

Push senders and pull bindings supply that source differently.

The rest of this page is about pull sources, where routing rules split one binding into several source types.

Use the decision tree to choose a Pub/Sub configuration, then adapt the examples for your pull driver.

The Two Ingestion Shapes

Push senders authenticate with a token that decides the source. Pull routing applies when nano reads a broker binding.

Push (HTTP, HEC, OTLP)

  • The token decides the source. Each sender authenticates with an ingest token bound to one log source, and every event it sends lands there. A declared X-Source-Type, HEC sourcetype or .source_type doesn't change it.
  • N source types means N tokens: a forwarder that carries several log types uses one sink (or output) per type, each with the token bound to that type's source.
  • Push sources need no routing rules.

Pull (GCP Pub/Sub, Apache Kafka, AWS S3 via SQS)

  • One source configuration = one binding (subscription / topic / queue). The sender cannot add a header to a Pub/Sub message attribute or a Kafka header on the consumer side; those have to be set by the publisher, before the message lands in the broker.
  • No inbound .source_type: a Pub/Sub event arriving at nano has .message, .attributes, and broker metadata, but no .source_type field. A routing rule that matches match_field=source_type, match_type=exact, match_value=foo can never fire on a pull source. It would require the field to exist on the event already, which it doesn't.
  • Multiplexing within one binding requires either:
    1. A publisher-set broker attribute (Pub/Sub message attribute, Kafka header, SQS message attribute) that the routing rule reads, or
    2. Content sniffing of a field inside the decoded message body, or
    3. Single source type for the whole feed: every event in this binding is the same source type (the mono-vendor case).

Per-Driver Routing Field Summary

The match field a routing rule reads from depends on the driver. The natural broker-native field, the recommended attribute-based field, and the content-sniff path are all listed below.

DriverBroker-native fieldRecommended attribute pathContent-sniff example
GCP Pub/Subsubscription (one per source config, not a routing rule).attributes.source_type.message.routing.event_type
Apache Kafka.topic (when fan-out within consumer group).headers.source_type.message.event_type
AWS S3 via SQS.key (S3 object key, e.g. AWSLogs/.../CloudTrail/) or .bucket.attributes.source_type (SQS message attribute set by producer)path-based on .key
HECThe token's source (legacy multiplex install token: sourcetype)n/an/a
HTTP (push)The token's source (legacy multiplex install token: X-Source-Type)n/an/a
Vector native (push)vector_native (upstream .source_type only with NANO_VECTOR_NATIVE_TRUST_SOURCE_TYPE=true)n/an/a

Decision Tree: "I Have N Log Sources via Pub/Sub"

Choose the configuration based on how the publisher exports logs.

N Topics (Most Common)

If your environment has one Pub/Sub topic per log type (nano-audit-logs, nano-vpc-flowlogs, nano-scc-findings), create one Source Configuration per subscription. Each binds to one subscription and tags every event with one source type. Each topic uses its own source configuration.

One Topic, N Log Types — Publisher Sets Attributes

If you control the publisher and it can set Pub/Sub message attributes, set a source_type attribute (or any agreed key) on each message before publishing. Then in the Source Configuration, switch to Multiple source types mode and add routing rules with match_field = .attributes.source_type.

One Topic, N Log Types — Third-Party Publisher

Some SaaS exporters (LimaCharlie's outbound Pub/Sub, for example) publish all event types to one topic and don't expose a way to set per-message attributes. In that case:

  • You actually need split routing (e.g. you want vendor_edr_process and vendor_edr_network to use different parsers): use Multiple source types mode and content-sniff a field inside the decoded message body, e.g. match_field = .message.routing.event_type.
  • You don't need split routing (e.g. one parser handles every LimaCharlie event): use Single source type mode. Every event in the subscription gets tagged limacharlie and routed to that parser. Use this mode when one parser handles the entire feed.

Per-Driver Examples

Use these examples to configure each pull driver. The HEC example shows source declarations for push events.

GCP Pub/Sub — Single Source Type (LimaCharlie)

Scenario: LimaCharlie publishes all event types to one Pub/Sub topic. You don't need per-event-type routing; one parser handles every LimaCharlie event.

  1. Create the subscription and credential per the GCP Pub/Sub setup guide.
  2. Create a Source Configuration of type gcp_pubsub:
    • Project ID: your-gcp-project
    • Subscription: nano-limacharlie-sub
    • Credential: the GCP service account credential
  3. In the routing section, choose Single source type:
    • Source type: limacharlie
    • Parser: the bundled LimaCharlie OCSF parser, which claims limacharlie

This produces one default routing rule and the generator emits an unconditional tag:

.source_type = "limacharlie"

GCP Pub/Sub — Multiple Source Types via Message Attributes

Scenario: Your own producer publishes events from multiple internal services to one Pub/Sub topic and sets a source_type message attribute on each message.

Publisher (Python google-cloud-pubsub):

from google.cloud import pubsub_v1

publisher = pubsub_v1.PublisherClient()
topic_path = publisher.topic_path("your-gcp-project", "internal-events")

publisher.publish(
    topic_path,
    data=event_json.encode("utf-8"),
    source_type="auth_service",      # message attribute
)

Source Configuration routing rules (Multiple source types mode):

Match FieldMatch TypeMatch ValueTarget Source Type
.attributes.source_typeexactauth_serviceauth_service
.attributes.source_typeexactapi_gatewayapi_gateway
.attributes.source_typeexactpayment_servicepayment_service
—default—unknown_internal

The generator emits:

if .attributes.source_type == "auth_service" {
    .source_type = "auth_service"
} else if .attributes.source_type == "api_gateway" {
    .source_type = "api_gateway"
} else if .attributes.source_type == "payment_service" {
    .source_type = "payment_service"
} else {
    .source_type = "unknown_internal"
}

GCP Pub/Sub — Content Sniffing (Third-Party SaaS, One Topic, N Event Types)

Scenario: A third-party exporter publishes events from multiple event categories to one topic. You can't change the publisher, but the body contains a discriminator field.

Sample event body (decoded .message):

{
  "routing": {
    "event_type": "NEW_PROCESS",
    "hostname": "web-01"
  },
  "event": { "image": "powershell.exe", "cmdline": "..." }
}

Source Configuration routing rules:

Match FieldMatch TypeMatch ValueTarget Source Type
.message.routing.event_typeexactNEW_PROCESSvendor_edr_process
.message.routing.event_typeexactNEW_CONNECTIONvendor_edr_network
.message.routing.event_typeprefixDNS_vendor_edr_dns
—default—vendor_edr_other

Apache Kafka — One Topic per Log Type

Scenario: You already use a topic-per-log-type convention.

Create one Source Configuration per Kafka topic. Each one uses Single source type mode and tags every event with its source type. The consumer group is shared.

  • cloudtrail-events topic → Source Configuration aws_cloudtrail
  • okta-events topic → Source Configuration okta
  • app-logs topic → Source Configuration my_app

Apache Kafka — One Topic, N Log Types via Headers

Scenario: A shared security-events topic carries multiple log types and the producer sets a source_type Kafka record header.

Producer (Java kafka-clients):

ProducerRecord<String, String> record =
    new ProducerRecord<>("security-events", null, eventJson);
record.headers().add("source_type", "okta".getBytes(UTF_8));
producer.send(record);

Source Configuration routing rules (Multiple source types mode):

Match FieldMatch TypeMatch ValueTarget Source Type
.headers.source_typeexactoktaokta
.headers.source_typeexactaws_cloudtrailaws_cloudtrail
.headers.source_typeprefixapp_application_logs
—default—generic_kafka

Apache Kafka — Routing on Topic Name

Scenario: A single nano consumer subscribes to multiple topics in one consumer group, and you want to route on .topic (the broker-native field) rather than headers.

Match FieldMatch TypeMatch ValueTarget Source Type
.topicexactcloudtrail-eventsaws_cloudtrail
.topicexactokta-eventsokta
.topicprefixapp-application_logs
—default—generic_kafka

The generator emits if .topic == "cloudtrail-events" { .source_type = "aws_cloudtrail" } etc.

AWS S3 via SQS — Routing on Object Key Prefix

Scenario: A single S3 bucket holds multiple log types under different prefixes (AWSLogs/.../CloudTrail/, AWSLogs/.../vpcflowlogs/, WAFLogs/) and one shared SQS queue receives notifications for all of them.

Source Configuration routing rules (Multiple source types mode):

Match FieldMatch TypeMatch ValueTarget Source Type
.keyprefixAWSLogs/123456789012/CloudTrail/aws_cloudtrail
.keyprefixAWSLogs/123456789012/vpcflowlogs/aws_vpc_flow
.keyprefixWAFLogs/aws_waf
—default—aws_s3_other

The .key match field reads the S3 object key (in the routing-rule UI it's the "Object key" preset; the default preset is .bucket). Producer-controlled SQS attribute alternative: if your application code wraps writes through Lambda or another shim, you can set an SQS message attribute on publish and route on .attributes.source_type instead, exactly like the Pub/Sub attribute example.

HEC — Routing on sourcetype (Push, for Contrast)

HEC is a push protocol, included here for contrast. With per-source tokens the token decides the source and these rules aren't consulted. They apply only to the install token in the deprecated legacy multiplex mode, where the HEC client sets sourcetype on each event, and nano routes on the canonical .sourcetype field. The match shape looks similar to a pull-source attribute match, but the field is set by the sender per-event, not by a publisher attribute.

Match FieldMatch TypeMatch ValueTarget Source Type
sourcetypeexactWinEventLog:Securitywindows_event
sourcetypeexactpan:trafficpalo_alto
sourcetypeprefixmyapp:my_app
—default—unknown_hec

Prefer one token per source over HEC routing rules: the token can't be spoofed by a sender's sourcetype.

When to Create a New Source Configuration vs Add a Routing Rule

Choose a new binding or a routing rule based on what changes:

SituationCreate a new Source ConfigurationAdd a routing rule
New Pub/Sub subscription / Kafka topic / SQS queueYesNo
New credential / different cloud accountYesNo
Same binding, new log type (publisher sets an attribute or there's a content discriminator)NoYes (Multiple source types mode)
Same binding, mono-vendor feedNoUse Single source type mode (one default rule)
New source type from an existing push forwarderNoNo: issue a token bound to the new source

A Source Configuration models a binding (one subscription, one topic, one queue, or one HTTP listener). Routing rules model how events inside that binding are split into source types. If you find yourself wanting to split across bindings, that's a new Source Configuration.

Source Configuration UI

Routing rules are configured per source configuration in the nano web UI: go to Ingestion → Log Sources → Cloud pull, open the pull source, and use the routing section. Pull-source configurations expose two modes, Single source type (one default rule) and Multiple source types (the routing-rules table), with per-driver match-field defaults.

Troubleshoot

Need help with these checks? Contact support with the deployment, request ID and time with timezone.

A pull routing rule never matches

Check that the match field exists on incoming events. Pull events do not arrive with push-style source_type; use a broker attribute, a decoded content field, or Single source type mode.

Push events land in the wrong source

The token decides the source. Check which source the sender's token is bound to under Ingestion → Log Sources → Credentials, and issue a token bound to the right source.

What's next

  • Ingest data: endpoints, pipeline and the source requirement
  • Supported Data Sources: protocol-level reference for every ingestion method
  • GCP Pub/Sub: end-to-end setup, sample events, troubleshooting
  • Apache Kafka: bootstrap, SASL, multiple topics, routing rules table; AWS S3 via SQS: bucket notifications, per-prefix vs shared queue; HEC: sourcetype routing, common vendor sourcetype → nano mappings

On this page