GCP Pub/Sub
End-to-end guide for ingesting Google Cloud logs via Pub/Sub subscriptions
Collect Google Cloud logs through Pub/Sub to search Cloud Audit Logs, VPC Flow Logs, Security Command Center findings, and GKE logs in nano. You can use this path for other log types exported through Cloud Logging.
nano connects to a Pub/Sub pull subscription and consumes messages in batches. Google Cloud's Log Router exports logs to a Pub/Sub topic, and nano reads from a subscription on that topic.
Setup
Vector in your deployment reads the subscription with a Google identity you control: a service-account JSON key stored in nano, or Application Default Credentials (Workload Identity on GKE, or the instance's service account) when you attach no credential. Manage the credential and the pull source under Ingestion → Log Sources → Cloud pull. A new source starts point-forward; see Cloud pull.
Prerequisites
You need credentials:create to store a credential and credentials:view to view its metadata. Creating a source configuration requires source_configs:create and credentials:use when you attach a credential. Editing it requires source_configs:edit and credentials:use when the request or saved configuration references a credential; publishing the parser requires log_sources:deploy.
- A Google Cloud project with billing enabled
gcloudCLI installed and authenticated (gcloud auth login)- Permissions to create service accounts, Pub/Sub resources, and log sinks (typically
roles/owneror a combination ofroles/iam.serviceAccountAdmin,roles/pubsub.admin,roles/logging.admin) - A running nano instance
Step 1: Create a Pub/Sub Topic and Subscription
Create a dedicated topic for the log type you want to ingest, and a pull subscription that nano will read from.
# Set your project
export PROJECT_ID="your-gcp-project-id"
gcloud config set project $PROJECT_ID
# Create the topic
gcloud pubsub topics create nanosiem-audit-logs
# Create a pull subscription
gcloud pubsub subscriptions create nanosiem-audit-logs-sub \
--topic=nanosiem-audit-logs \
--ack-deadline=600 \
--message-retention-duration=1d \
--expiration-period=neverNaming Convention
Use descriptive names that match your nano log source names:
| Log Type | Topic Name | Subscription Name |
|---|---|---|
| Cloud Audit Logs | nanosiem-audit-logs | nanosiem-audit-logs-sub |
| VPC Flow Logs | nanosiem-vpc-flowlogs | nanosiem-vpc-flowlogs-sub |
| SCC Findings | nanosiem-scc-findings | nanosiem-scc-findings-sub |
| GKE Logs | nanosiem-gke-logs | nanosiem-gke-logs-sub |
Step 2: Create a Log Router Sink
A Log Router sink exports matching log entries from Cloud Logging to your Pub/Sub topic. The sink filter determines which logs are exported.
Cloud Audit Logs (Admin Activity + Data Access)
gcloud logging sinks create nanosiem-audit-sink \
pubsub.googleapis.com/projects/$PROJECT_ID/topics/nanosiem-audit-logs \
--log-filter='logName:"cloudaudit.googleapis.com"' \
--description="Export audit logs to nano"VPC Flow Logs
gcloud logging sinks create nanosiem-vpcflow-sink \
pubsub.googleapis.com/projects/$PROJECT_ID/topics/nanosiem-vpc-flowlogs \
--log-filter='resource.type="gce_subnetwork" AND logName:"compute.googleapis.com%2Fvpc_flows"' \
--description="Export VPC Flow Logs to nano"Security Command Center Findings
gcloud logging sinks create nanosiem-scc-sink \
pubsub.googleapis.com/projects/$PROJECT_ID/topics/nanosiem-scc-findings \
--log-filter='resource.type="threat_detector" OR resource.type="security_command_center"' \
--description="Export SCC findings to nano"GKE Audit Logs
gcloud logging sinks create nanosiem-gke-sink \
pubsub.googleapis.com/projects/$PROJECT_ID/topics/nanosiem-gke-logs \
--log-filter='resource.type="k8s_cluster" AND logName:"cloudaudit.googleapis.com"' \
--description="Export GKE audit logs to nano"Custom Filter (All Security-Relevant Logs)
To export multiple log types to a single topic:
gcloud logging sinks create nanosiem-all-security \
pubsub.googleapis.com/projects/$PROJECT_ID/topics/nanosiem-security-logs \
--log-filter='
logName:"cloudaudit.googleapis.com"
OR (resource.type="gce_subnetwork" AND logName:"compute.googleapis.com%2Fvpc_flows")
OR resource.type="threat_detector"
' \
--description="Export all security-relevant logs to nano"Grant the Sink's Service Account Publish Access
When you create a log sink, GCP creates a dedicated writer service account. This account needs permission to publish to your Pub/Sub topic.
# Get the sink's writer identity
SINK_SA=$(gcloud logging sinks describe nanosiem-audit-sink \
--format='value(writerIdentity)')
echo "Sink service account: $SINK_SA"
# Grant it the Pub/Sub Publisher role on the topic
gcloud pubsub topics add-iam-policy-binding nanosiem-audit-logs \
--member="$SINK_SA" \
--role="roles/pubsub.publisher"Verify the Sink Is Working
# Check sink status (should show no errors)
gcloud logging sinks describe nanosiem-audit-sink
# Wait a minute, then check if messages are arriving
gcloud pubsub subscriptions pull nanosiem-audit-logs-sub \
--limit=1 \
--auto-ack=falseIf no messages appear after a few minutes, generate some activity (e.g., list IAM policies: gcloud iam roles list --limit=1) and check again. Admin Activity audit logs are generated immediately; Data Access logs may have a slight delay.
Step 3: Create a Service Account for nano
Create a dedicated service account with minimal permissions for nano to pull from the subscription.
# Create the service account
gcloud iam service-accounts create nanosiem-reader \
--display-name="nano Log Reader" \
--description="Service account for nano to pull logs from Pub/Sub"
# Get the full service account email
SA_EMAIL="nanosiem-reader@${PROJECT_ID}.iam.gserviceaccount.com"
echo "Service account: $SA_EMAIL"Grant Permissions
nano needs pubsub.subscriptions.consume on the subscription. This permission covers both pulling and acknowledging messages.
# Grant Pub/Sub Subscriber role on the subscription
gcloud pubsub subscriptions add-iam-policy-binding nanosiem-audit-logs-sub \
--member="serviceAccount:${SA_EMAIL}" \
--role="roles/pubsub.subscriber"The roles/pubsub.subscriber role grants pubsub.subscriptions.consume.
Create and Download the Key
# Create a JSON key file
gcloud iam service-accounts keys create /tmp/nanosiem-reader-key.json \
--iam-account="$SA_EMAIL"
echo "Key saved to /tmp/nanosiem-reader-key.json"The key file will look like this (you'll paste the entire contents into nano):
{
"type": "service_account",
"project_id": "your-project-id",
"private_key_id": "key-id",
"private_key": "-----BEGIN RSA PRIVATE KEY-----\n...\n-----END RSA PRIVATE KEY-----\n",
"client_email": "nanosiem-reader@your-project.iam.gserviceaccount.com",
"client_id": "123456789",
"auth_uri": "https://accounts.google.com/o/oauth2/auth",
"token_uri": "https://oauth2.googleapis.com/token",
"auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
"client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/..."
}Step 4: Store Credentials in nano
- Go to Ingestion → Log Sources → Cloud pull
- Under Credentials, add a credential
- Fill in the form:
| Field | Value |
|---|---|
| Provider | GCP Pub/Sub |
| Name | A descriptive name, e.g. GCP Production - Audit Logs |
| Service Account JSON | Paste the entire contents of the JSON key file |
- Click Save
The credential is encrypted at rest. The private key will never be displayed again.
# After uploading, delete the local key file
rm /tmp/nanosiem-reader-key.jsonStep 5: Create a Log Source and a Pull Source
- Go to Ingestion → Log Sources → Add data and set up the log source. If a bundled OCSF parser claims your source (see Supported sources), use it. Otherwise paste a representative log entry (see examples below) and build a parser from it. Publish it.
- Go to Ingestion → Log Sources → Cloud pull and choose Add cloud pull
- Configure the connection:
| Field | Value |
|---|---|
| Type | GCP Pub/Sub |
| GCP Project ID | Your project ID, e.g. my-gcp-project-123 |
| Subscription Name | The subscription name (not the full path), e.g. nanosiem-audit-logs-sub |
| ACK Deadline (seconds) | 600 (default, matches the subscription setting) |
| Credential | Select the credential you created in Step 4, or none for Application Default Credentials |
- Route it to the log source from step 1 (Single source type), or add routing rules for a shared subscription
- Deploy the pull source to start reading
Sample Log Events by GCP Service
Cloud Audit Log (Admin Activity)
{
"protoPayload": {
"@type": "type.googleapis.com/google.cloud.audit.AuditLog",
"serviceName": "iam.googleapis.com",
"methodName": "google.iam.admin.v1.CreateServiceAccount",
"authenticationInfo": {
"principalEmail": "alice@example.com"
},
"requestMetadata": {
"callerIp": "203.0.113.50",
"callerSuppliedUserAgent": "google-cloud-sdk gcloud/456.0.0"
},
"resourceName": "projects/my-project/serviceAccounts/new-sa@my-project.iam.gserviceaccount.com",
"request": {
"account_id": "new-sa",
"service_account": {
"display_name": "New Service Account"
}
}
},
"insertId": "abc123def456",
"resource": {
"type": "service_account",
"labels": {
"project_id": "my-project",
"email_id": "new-sa@my-project.iam.gserviceaccount.com"
}
},
"timestamp": "2025-01-15T14:23:45.678Z",
"severity": "NOTICE",
"logName": "projects/my-project/logs/cloudaudit.googleapis.com%2Factivity"
}Security Command Center Finding
{
"finding": {
"name": "organizations/123/sources/456/findings/abc123",
"parent": "organizations/123/sources/456",
"state": "ACTIVE",
"category": "PUBLIC_BUCKET_ACL",
"resourceName": "//storage.googleapis.com/my-public-bucket",
"severity": "HIGH",
"eventTime": "2025-01-15T14:23:45Z",
"createTime": "2025-01-15T14:23:45Z",
"sourceProperties": {
"ResourcePath": "my-public-bucket",
"AllowedPolicy": "allUsers"
}
},
"resource": {
"type": "google.cloud.storage.Bucket",
"name": "//storage.googleapis.com/my-public-bucket",
"project": "my-project"
}
}Step 6: Verify Ingestion
After publishing, allow a few minutes for Vector to connect to the subscription and start pulling messages.
Check log source health
- Go to Ingestion → Log Sources and open your new log source
- Check:
- Status: Should show "Healthy"
- Event Volume chart: Should show events arriving
- Last Event: Should show a recent timestamp
Search Your Data
Navigate to Search and run a query for your source type:
metadata.log_source="gcp_audit" | stats count by api.service.nameOr search for specific activity:
metadata.log_source="gcp_audit" api.operation="google.iam.admin.v1.CreateServiceAccount"
| table time, actor.user.name, src_endpoint.ip, api.operationCheck for Errors
If no data appears:
-
Check the subscription backlog. Are messages accumulating?
gcloud pubsub subscriptions describe nanosiem-audit-logs-sub \ --format='value(numUndeliveredMessages)'- Messages accumulating: nano isn't pulling. Check credentials and the log source deployment status.
- Zero messages: The Log Router sink isn't exporting, or no matching logs are being generated. Check the sink configuration and generate some test activity.
-
Verify the sink is delivering:
# Check for sink errors gcloud logging sinks describe nanosiem-audit-sink \ --format='yaml(writerIdentity,destination,filter,outputVersionFormat)' -
Check Vector logs for connection errors:
docker logs nano-vector 2>&1 | grep -i "gcp\|pubsub\|error"
Multi-Project Setup
For organizations with multiple GCP projects, you have two approaches:
Option A: Organization-Level Sink (Recommended)
Create a single sink at the organization or folder level that captures logs from all projects:
# Organization-level sink (requires Org Admin)
gcloud logging sinks create nanosiem-org-audit-sink \
pubsub.googleapis.com/projects/$PROJECT_ID/topics/nanosiem-audit-logs \
--organization=ORG_ID \
--log-filter='logName:"cloudaudit.googleapis.com"' \
--include-children \
--description="Export all org audit logs to nano"
# Grant the sink's writer identity publish access
SINK_SA=$(gcloud logging sinks describe nanosiem-org-audit-sink \
--organization=ORG_ID \
--format='value(writerIdentity)')
gcloud pubsub topics add-iam-policy-binding nanosiem-audit-logs \
--member="$SINK_SA" \
--role="roles/pubsub.publisher"Option B: Per-Project Sinks
Create individual sinks in each project, all publishing to the same topic. More granular control but more to manage.
Troubleshoot
Need help with these checks? Contact support with the deployment, request ID and time with timezone.
"Permission denied" in Vector logs
- Verify the service account has
roles/pubsub.subscriberon the subscription (not the topic) - Check the service account JSON was pasted correctly (it should start with
{and end with})
Messages in subscription but no events in nano
- Check the log source is deployed (published); look for the "Unpublished changes" banner
- Verify the Subscription Name in the log source config is the short name (e.g.,
nanosiem-audit-logs-sub), not the full resource path - Verify the Project ID matches the project where the subscription lives
- Check whether events are landing unparsed: a parser that can't read an event emits it as Base Event (
class_uid0)
Sink shows errors in Cloud Logging
- Check the writer identity has
roles/pubsub.publisheron the topic - Verify the topic exists and hasn't been deleted
- Check the Pub/Sub API is enabled:
gcloud services enable pubsub.googleapis.com
High latency or message redelivery
- Increase the ACK deadline on both the subscription and in nano's log source config (both should match)
- Check Vector resource allocation; Pub/Sub processing may need more CPU/memory for high-volume subscriptions
- Consider multiple subscriptions to the same topic for parallel consumption
Subscription expiration
By default, Pub/Sub subscriptions expire after 31 days of inactivity. Use --expiration-period=never when creating subscriptions (as shown in Step 1) to prevent this. For existing subscriptions:
gcloud pubsub subscriptions update nanosiem-audit-logs-sub \
--expiration-period=neverRecommended Log Sources for Security
If you're starting from scratch, prioritize these GCP log sources:
| Priority | Source | Log Filter | Why |
|---|---|---|---|
| 1 | Admin Activity Audit Logs | logName:"activity" | All administrative actions: IAM changes, resource creation, config modifications |
| 2 | Security Command Center | resource.type="threat_detector" | GCP-native threat detection and misconfiguration findings |
| 3 | Data Access Audit Logs | logName:"data_access" | Who read/listed/exported what data (high volume; filter carefully) |
| 4 | VPC Flow Logs | resource.type="gce_subnetwork" | Network traffic for lateral movement and exfiltration detection |
| 5 | GKE Audit Logs | resource.type="k8s_cluster" | Kubernetes API activity for container security monitoring |
| 6 | Cloud DNS Logs | resource.type="dns_query" | DNS queries for C2 and exfiltration detection |
Routing Within a Subscription
A Pub/Sub pull source binds to one subscription. To handle multiple log types inside a single subscription (e.g. a SaaS exporter that publishes everything to one topic), see Push vs Pull Routing for the attribute-based and content-sniff routing patterns.
What's next
- Compare pull permissions
- Create detection rules for your GCP logs
- Configure enrichment to add GeoIP and threat intel to GCP source IPs
- Set up the AWS integration if you also use AWS