Cloud pull
Read cloud queues, subscriptions, event hubs and Kafka topics with Vector, using credentials you own
Use Cloud pull to collect logs that your cloud delivers to a queue, subscription, event hub or Kafka topic. Vector in your nano deployment reads them with credentials you own, then routes each event to the parser for its source. Open Ingestion → Log Sources → Cloud pull (/ingestion/log-sources/cloud-pull).
| Pull source | Guide |
|---|---|
| AWS S3 + SQS | AWS S3 via SQS: S3 bucket notifications on an SQS queue; Vector reads each notified object |
| AWS SQS | AWS SQS: event message bodies read directly from a queue |
| CrowdStrike FDR | CrowdStrike FDR: Falcon Data Replicator batch manifests on SQS, event files in S3 |
| GCP Pub/Sub | GCP Pub/Sub: a Log Router sink publishing to a pull subscription |
| Azure Event Hubs | Azure Event Hubs: Azure Monitor diagnostic settings streaming to an event hub |
| Kafka | Kafka: a consumer group on one or more topics |
The page has two parts:
- Pull sources: the queues, subscriptions, hubs and topics Vector reads from (
/api/source-configurations). Open one to edit its connection and routing rules. - Credentials: the cloud credentials the pull sources read with. You enter, rotate and roll them back in place; secret values are masked and never returned after you save them.
Before adding a pull source, enable or publish the parser for each source it delivers into under Log Sources. A pull source supplies the transport; it does not create a parser. A pull source delivers into the log sources its routing rules name: one source in Single source type mode, or several split by broker attribute or message content. See Push vs Pull Routing.
Credentials and identity
Every pull source reads with a credential you control:
- AWS: an access key, or no key at all so Vector uses the deployment's ambient identity (EC2 instance profile, EKS IRSA or ECS task role). Either can assume a role in another account.
- GCP: a service-account JSON key with Subscriber on the subscription.
- Azure: an Entra app (client ID and secret) or SAS connection strings, stored as an
azure_event_hubscredential. With no credential bound, the source authenticates with the deployment's managed identity. - Kafka: SASL username and password, or none for an unauthenticated cluster.
Credentials are encrypted at rest with AES-256-GCM, and a credential can't be deleted while a pull source uses it. Review the pull permissions matrix for the exact cloud grants.
Permissions
| Action | Permission |
|---|---|
| List pull sources | source_configs:view |
| Add a pull source | source_configs:create |
| Edit, enable or disable | source_configs:edit |
| Deploy or undeploy | source_configs:deploy |
| Delete | source_configs:delete |
| View, enter or rotate credentials | credentials:view, credentials:create, credentials:use |
What happens when you enable a pull source
A new pull source starts point-forward. It collects events that arrive after you create it, not the history already sitting in your queue, subscription, event hub or topic.
Why nano does not pull history
When you add a source, history may already be waiting for it:
- An SQS queue can hold up to 14 days of messages.
- An event hub retains 7 to 90 days of events, depending on your tier.
- A Pub/Sub subscription can hold a week of unacknowledged messages.
If Vector read all of that when you enabled the source, it would arrive as one burst on top of your live traffic. That burst can starve every other source and delay detections on current activity. Point-forward is a protection, not a missing feature.
How each source type starts
| Source | Where it starts | What happens to older data |
|---|---|---|
| AWS S3 via SQS | When you create the source | A notification for an object from more than five minutes before the source was created is acknowledged and skipped, never ingested. The time is the S3 notification's event time, or the object's LastModified. |
| AWS SQS | When you create the source | A message whose SQS SentTimestamp is more than five minutes before the source was created is acknowledged and skipped. |
| CrowdStrike FDR | Everything on the queue (start_from = beginning), or the first start, or the last delivered batch | FDR keeps about seven days of batches; with Only batches after the first start, older batches are deleted unread on their first delivery. A batch nano already received is never skipped. |
| GCP Pub/Sub | When you create the source | A message whose publish time is more than five minutes before the source was created is acknowledged and skipped. |
| Azure Event Hubs | The newest event (latest) | Events the hub already retained aren't read. The read position is checkpointed in Azure Blob Storage, so a restart neither loses nor replays events. |
| Kafka | The newest offset (auto_offset_reset = "latest") for a new consumer group | An existing consumer group resumes from its committed offsets. |
| SaaS collectors | When you create the feed | Older API history isn't requested. |
The five-minute grace absorbs clock differences between your cloud and nano. An event that can't be dated passes rather than being skipped.
The cutoff is fixed at creation
The cutoff is the time the source was created. Disabling and re-enabling a source does not move it: whatever queued in your cloud meanwhile is ingested when you resume, as long as your cloud still retains it. Deleting a source and adding it again creates a new source with a new cutoff.
Collect the backlog on purpose
To ingest what an S3/SQS, SQS or Pub/Sub source already holds, set include_backlog: true in its connection configuration. That turns the cutoff off for that source. Do it only for a backlog you have sized: the whole queue arrives at once.
Source lifecycle
| State | What it means |
|---|---|
| pending | Enabled but not deployed yet. Choose Deploy to render it into Vector's configuration. |
| active | Enabled and deployed. Vector is reading. Check Search separately to confirm events. |
| paused | Disabled. Vector stops reading; events keep queuing in your cloud. |
Deploying publishes a new Vector configuration generation. Vector reloads it without a restart and acknowledges it; if Vector rejects the generation, it rolls back to the last good one and the deploy reports the error. Undeploy removes the source from Vector without deleting it.
Troubleshoot
Need help with these checks? Contact support with the deployment, request ID and time with timezone.
The source is active but Search is empty
Check the cloud producer (notification, sink or diagnostic setting), the parser for each routed source, the routing rules, and the search time range. A deployed source proves Vector is configured, not that new events are being generated or parsed.
Access denied or authentication errors
Check the credential and its cloud grants against the pull permissions matrix. Provider-specific fixes are in the AWS, GCP and Azure guides.
Old messages were not ingested
That is the point-forward cutoff. Set include_backlog: true if you want the backlog, or resend the data.