nano SIEM
Integrations

Cloud pull

Supported in:Self-hosted

Read cloud queues, subscriptions, event hubs and Kafka topics with Vector, using credentials you own

Use Cloud pull to collect logs that your cloud delivers to a queue, subscription, event hub or Kafka topic. Vector in your nano deployment reads them with credentials you own, then routes each event to the parser for its source. Open Ingestion → Log Sources → Cloud pull (/ingestion/log-sources/cloud-pull).

Pull sourceGuide
AWS S3 + SQSAWS S3 via SQS: S3 bucket notifications on an SQS queue; Vector reads each notified object
AWS SQSAWS SQS: event message bodies read directly from a queue
CrowdStrike FDRCrowdStrike FDR: Falcon Data Replicator batch manifests on SQS, event files in S3
GCP Pub/SubGCP Pub/Sub: a Log Router sink publishing to a pull subscription
Azure Event HubsAzure Event Hubs: Azure Monitor diagnostic settings streaming to an event hub
KafkaKafka: a consumer group on one or more topics

The page has two parts:

  • Pull sources: the queues, subscriptions, hubs and topics Vector reads from (/api/source-configurations). Open one to edit its connection and routing rules.
  • Credentials: the cloud credentials the pull sources read with. You enter, rotate and roll them back in place; secret values are masked and never returned after you save them.

Before adding a pull source, enable or publish the parser for each source it delivers into under Log Sources. A pull source supplies the transport; it does not create a parser. A pull source delivers into the log sources its routing rules name: one source in Single source type mode, or several split by broker attribute or message content. See Push vs Pull Routing.

Credentials and identity

Every pull source reads with a credential you control:

  • AWS: an access key, or no key at all so Vector uses the deployment's ambient identity (EC2 instance profile, EKS IRSA or ECS task role). Either can assume a role in another account.
  • GCP: a service-account JSON key with Subscriber on the subscription.
  • Azure: an Entra app (client ID and secret) or SAS connection strings, stored as an azure_event_hubs credential. With no credential bound, the source authenticates with the deployment's managed identity.
  • Kafka: SASL username and password, or none for an unauthenticated cluster.

Credentials are encrypted at rest with AES-256-GCM, and a credential can't be deleted while a pull source uses it. Review the pull permissions matrix for the exact cloud grants.

Permissions

ActionPermission
List pull sourcessource_configs:view
Add a pull sourcesource_configs:create
Edit, enable or disablesource_configs:edit
Deploy or undeploysource_configs:deploy
Deletesource_configs:delete
View, enter or rotate credentialscredentials:view, credentials:create, credentials:use

What happens when you enable a pull source

A new pull source starts point-forward. It collects events that arrive after you create it, not the history already sitting in your queue, subscription, event hub or topic.

Why nano does not pull history

When you add a source, history may already be waiting for it:

  • An SQS queue can hold up to 14 days of messages.
  • An event hub retains 7 to 90 days of events, depending on your tier.
  • A Pub/Sub subscription can hold a week of unacknowledged messages.

If Vector read all of that when you enabled the source, it would arrive as one burst on top of your live traffic. That burst can starve every other source and delay detections on current activity. Point-forward is a protection, not a missing feature.

How each source type starts

SourceWhere it startsWhat happens to older data
AWS S3 via SQSWhen you create the sourceA notification for an object from more than five minutes before the source was created is acknowledged and skipped, never ingested. The time is the S3 notification's event time, or the object's LastModified.
AWS SQSWhen you create the sourceA message whose SQS SentTimestamp is more than five minutes before the source was created is acknowledged and skipped.
CrowdStrike FDREverything on the queue (start_from = beginning), or the first start, or the last delivered batchFDR keeps about seven days of batches; with Only batches after the first start, older batches are deleted unread on their first delivery. A batch nano already received is never skipped.
GCP Pub/SubWhen you create the sourceA message whose publish time is more than five minutes before the source was created is acknowledged and skipped.
Azure Event HubsThe newest event (latest)Events the hub already retained aren't read. The read position is checkpointed in Azure Blob Storage, so a restart neither loses nor replays events.
KafkaThe newest offset (auto_offset_reset = "latest") for a new consumer groupAn existing consumer group resumes from its committed offsets.
SaaS collectorsWhen you create the feedOlder API history isn't requested.

The five-minute grace absorbs clock differences between your cloud and nano. An event that can't be dated passes rather than being skipped.

The cutoff is fixed at creation

The cutoff is the time the source was created. Disabling and re-enabling a source does not move it: whatever queued in your cloud meanwhile is ingested when you resume, as long as your cloud still retains it. Deleting a source and adding it again creates a new source with a new cutoff.

Collect the backlog on purpose

To ingest what an S3/SQS, SQS or Pub/Sub source already holds, set include_backlog: true in its connection configuration. That turns the cutoff off for that source. Do it only for a backlog you have sized: the whole queue arrives at once.

Source lifecycle

StateWhat it means
pendingEnabled but not deployed yet. Choose Deploy to render it into Vector's configuration.
activeEnabled and deployed. Vector is reading. Check Search separately to confirm events.
pausedDisabled. Vector stops reading; events keep queuing in your cloud.

Deploying publishes a new Vector configuration generation. Vector reloads it without a restart and acknowledges it; if Vector rejects the generation, it rolls back to the last good one and the deploy reports the error. Undeploy removes the source from Vector without deleting it.

Troubleshoot

Need help with these checks? Contact support with the deployment, request ID and time with timezone.

The source is active but Search is empty

Check the cloud producer (notification, sink or diagnostic setting), the parser for each routed source, the routing rules, and the search time range. A deployed source proves Vector is configured, not that new events are being generated or parsed.

Access denied or authentication errors

Check the credential and its cloud grants against the pull permissions matrix. Provider-specific fixes are in the AWS, GCP and Azure guides.

Old messages were not ingested

That is the point-forward cutoff. Set include_backlog: true if you want the backlog, or resend the data.

What's next

On this page