AWS S3 via SQS
End-to-end guide for ingesting AWS logs from S3 buckets using SQS notifications
Collect AWS logs from S3 through SQS notifications to search them in nano. Use this path for CloudTrail, VPC Flow Logs, GuardDuty findings, ALB/NLB access logs, WAF logs, and other AWS services that write JSON or gzipped logs to S3.
nano uses the S3 → SQS notification pattern. When new objects land in your S3 bucket, S3 sends a notification to an SQS queue, and nano's Vector pipeline polls that queue, downloads the objects, and processes them.
Setup
Vector in your deployment reads the queue and downloads each notified object with an AWS identity you control: an access key stored in nano, or the deployment's ambient identity (EC2 instance profile, EKS IRSA or ECS task role) when you attach no credential. Either can assume a role in another account. Manage the credential and the pull source under Ingestion → Log Sources → Cloud pull. A new source starts point-forward; see Cloud pull.
Prerequisites
You need credentials:create to store a credential and credentials:view to view its metadata. Creating a source configuration requires source_configs:create and credentials:use when you attach a credential. Editing it requires source_configs:edit and credentials:use when the request or saved configuration references a credential; publishing the parser requires log_sources:deploy.
- An AWS account with permissions to create IAM users/roles, SQS queues, and S3 bucket notifications
- AWS CLI installed and configured (
aws configure) - A running nano instance
Step 1: Create the SQS Queue
Create an SQS queue that will receive S3 object notifications. Use a standard queue, not FIFO (S3 notifications don't support FIFO queues).
# Create the queue
aws sqs create-queue \
--queue-name nanosiem-cloudtrail-logs \
--attributes '{
"MessageRetentionPeriod": "86400",
"VisibilityTimeout": "300"
}'Save the queue URL and ARN. You'll need both:
# Get queue URL
QUEUE_URL=$(aws sqs get-queue-url \
--queue-name nanosiem-cloudtrail-logs \
--query 'QueueUrl' --output text)
echo "Queue URL: $QUEUE_URL"
# Get queue ARN
QUEUE_ARN=$(aws sqs get-queue-attributes \
--queue-url "$QUEUE_URL" \
--attribute-names QueueArn \
--query 'Attributes.QueueArn' --output text)
echo "Queue ARN: $QUEUE_ARN"Set the SQS Queue Policy
S3 needs permission to send messages to your SQS queue. Without this policy, S3 notifications will silently fail.
Replace ACCOUNT_ID, BUCKET_NAME, and the queue ARN with your values:
cat > /tmp/sqs-policy.json << 'EOF'
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "AllowS3Notifications",
"Effect": "Allow",
"Principal": {
"Service": "s3.amazonaws.com"
},
"Action": "sqs:SendMessage",
"Resource": "arn:aws:sqs:us-east-1:ACCOUNT_ID:nanosiem-cloudtrail-logs",
"Condition": {
"ArnLike": {
"aws:SourceArn": "arn:aws:s3:::BUCKET_NAME"
}
}
}
]
}
EOF
# Apply the policy (edit the file first to replace placeholders)
aws sqs set-queue-attributes \
--queue-url "$QUEUE_URL" \
--attributes '{"Policy": "'"$(cat /tmp/sqs-policy.json | jq -c .)"'"}'Step 2: Configure S3 Bucket Notifications
Tell your S3 bucket to send notifications to the SQS queue when new objects are created.
cat > /tmp/notification.json << 'EOF'
{
"QueueConfigurations": [
{
"Id": "nanosiem-log-notification",
"QueueArn": "arn:aws:sqs:us-east-1:ACCOUNT_ID:nanosiem-cloudtrail-logs",
"Events": ["s3:ObjectCreated:*"],
"Filter": {
"Key": {
"FilterRules": [
{"Name": "suffix", "Value": ".json.gz"}
]
}
}
}
]
}
EOF
# Apply notification configuration (edit the file first to replace placeholders)
aws s3api put-bucket-notification-configuration \
--bucket YOUR_BUCKET_NAME \
--notification-configuration file:///tmp/notification.jsonSuffix Filters by Log Type
Different AWS services use different file extensions. Adjust the FilterRules accordingly:
| AWS Service | Typical Suffix | Prefix Pattern |
|---|---|---|
| CloudTrail | .json.gz | AWSLogs/ACCOUNT_ID/CloudTrail/ |
| VPC Flow Logs | .log.gz | AWSLogs/ACCOUNT_ID/vpcflowlogs/ |
| GuardDuty | .jsonl.gz | AWSLogs/ACCOUNT_ID/GuardDuty/ |
| ALB Access Logs | .log.gz | AWSLogs/ACCOUNT_ID/elasticloadbalancing/ |
| NLB Access Logs | .log.gz | AWSLogs/ACCOUNT_ID/elasticloadbalancing/ |
| WAF Logs | .log.gz | aws-waf-logs-* |
| S3 Access Logs | none | depends on your config |
You can also filter by prefix to narrow which objects trigger notifications:
{
"FilterRules": [
{"Name": "prefix", "Value": "AWSLogs/123456789012/CloudTrail/"},
{"Name": "suffix", "Value": ".json.gz"}
]
}Verify Notifications Are Working
Upload a test file and check the queue:
# Upload a test object that matches your filter
echo '{"test": true}' | gzip | \
aws s3 cp - s3://YOUR_BUCKET_NAME/test-notification.json.gz
# Check the queue for messages (don't delete them)
aws sqs receive-message \
--queue-url "$QUEUE_URL" \
--max-number-of-messages 1 \
--wait-time-seconds 5
# Clean up the test file
aws s3 rm s3://YOUR_BUCKET_NAME/test-notification.json.gzYou should see an SQS message with an s3:ObjectCreated:Put event. If the response is empty, double-check your queue policy and notification configuration.
Step 3: Create an IAM User for nano
Create a dedicated IAM user with minimal permissions. nano needs to read from the SQS queue and download objects from the S3 bucket.
# Create the IAM user
aws iam create-user --user-name nanosiem-log-reader
# Create the access key
aws iam create-access-key --user-name nanosiem-log-readerSave the AccessKeyId and SecretAccessKey from the output. You'll enter these in nano.
IAM Policy
Attach a policy with the minimum required permissions:
cat > /tmp/nanosiem-policy.json << 'EOF'
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "SQSRead",
"Effect": "Allow",
"Action": [
"sqs:ReceiveMessage",
"sqs:DeleteMessage",
"sqs:GetQueueAttributes"
],
"Resource": "arn:aws:sqs:us-east-1:ACCOUNT_ID:nanosiem-*"
},
{
"Sid": "S3Read",
"Effect": "Allow",
"Action": [
"s3:GetObject"
],
"Resource": "arn:aws:s3:::YOUR_BUCKET_NAME/*"
}
]
}
EOF
aws iam put-user-policy \
--user-name nanosiem-log-reader \
--policy-name nanosiem-s3-sqs-read \
--policy-document file:///tmp/nanosiem-policy.jsonCross-Account Access (Optional)
If your S3 bucket is in a different AWS account than nano, use an IAM role with sts:AssumeRole instead of direct access keys:
- In the log account (where S3 lives), create an IAM role with the S3/SQS permissions above and a trust policy allowing the nano account to assume it
- In the nano account, give the nano IAM user
sts:AssumeRolepermission for that role - In nano, set the Assume Role ARN field when creating the credential (e.g.,
arn:aws:iam::LOG_ACCOUNT:role/nanosiem-reader)
Step 4: Store Credentials in nano
Skip this step if Vector runs with an ambient AWS identity (instance profile, IRSA or task role) that has the policy above.
- Go to Ingestion → Log Sources → Cloud pull
- Under Credentials, add a credential
- Fill in the form:
| Field | Value |
|---|---|
| Provider | AWS S3 |
| Name | A descriptive name, e.g. AWS Production - CloudTrail |
| Region | The region of your SQS queue, e.g. us-east-1 |
| Access Key ID | The AccessKeyId from Step 3 |
| Secret Access Key | The SecretAccessKey from Step 3 |
- Click Save
The credential is encrypted at rest and the secret key will never be displayed again.
Step 5: Create a Log Source and a Pull Source
- Go to Ingestion → Log Sources → Add data and set up the log source. If a bundled OCSF parser claims your source (see Supported sources), use it. Otherwise paste a representative log entry for your source (see examples below) and build a parser from it. Publish it.
- Go to Ingestion → Log Sources → Cloud pull and choose Add cloud pull
- Configure the connection:
| Field | Value |
|---|---|
| Type | AWS S3 + SQS |
| SQS Queue URL | The full queue URL, e.g. https://sqs.us-east-1.amazonaws.com/123456789012/nanosiem-cloudtrail-logs |
| Region | Must match the SQS queue region, e.g. us-east-1 |
| Compression | Auto (recommended); nano detects gzip/zstd automatically |
| Credential | Select the credential you created in Step 4, or none for the ambient identity |
- Route it to the log source from step 1 (Single source type), or add routing rules for a shared queue
- Deploy the pull source to start reading
Sample Log Events by AWS Service
CloudTrail
{
"eventVersion": "1.09",
"userIdentity": {
"type": "IAMUser",
"principalId": "AIDACKCEVSQ6C2EXAMPLE",
"arn": "arn:aws:iam::123456789012:user/alice",
"accountId": "123456789012",
"userName": "alice"
},
"eventTime": "2025-01-15T14:23:45Z",
"eventSource": "iam.amazonaws.com",
"eventName": "CreateUser",
"awsRegion": "us-east-1",
"sourceIPAddress": "203.0.113.50",
"userAgent": "aws-cli/2.15.0",
"requestParameters": {
"userName": "new-service-account"
},
"responseElements": {
"user": {
"userName": "new-service-account",
"userId": "AIDACKCEVSQ6C3EXAMPLE",
"arn": "arn:aws:iam::123456789012:user/new-service-account"
}
}
}VPC Flow Logs (JSON format)
{
"version": 2,
"account-id": "123456789012",
"interface-id": "eni-0a1b2c3d4e5f6g7h8",
"srcaddr": "10.0.1.5",
"dstaddr": "203.0.113.50",
"srcport": 49152,
"dstport": 443,
"protocol": 6,
"packets": 25,
"bytes": 5000,
"start": 1705312800,
"end": 1705312860,
"action": "ACCEPT",
"log-status": "OK"
}GuardDuty
{
"schemaVersion": "2.0",
"accountId": "123456789012",
"region": "us-east-1",
"type": "Recon:EC2/PortProbeUnprotectedPort",
"severity": 2,
"createdAt": "2025-01-15T14:23:45.000Z",
"updatedAt": "2025-01-15T14:23:45.000Z",
"title": "Unprotected port on EC2 instance is being probed",
"description": "EC2 instance i-0abc123 has an unprotected port which is being probed by a known malicious host.",
"resource": {
"resourceType": "Instance",
"instanceDetails": {
"instanceId": "i-0abc123def456",
"instanceType": "t3.medium"
}
},
"service": {
"action": {
"portProbeAction": {
"portProbeDetails": [
{
"localPortDetails": {"port": 22, "portName": "SSH"},
"remoteIpDetails": {"ipAddressV4": "198.51.100.0"}
}
]
}
}
}
}Step 6: Verify Ingestion
After publishing, allow a few minutes for Vector to start polling the SQS queue.
Check log source health
- Go to Ingestion → Log Sources and open your new log source
- Check:
- Status: Should show "Healthy"
- Event Volume chart: Should show events arriving
- Last Event: Should show a recent timestamp
Search Your Data
Go to Search and query the source. Events are stored as OCSF with the source in metadata.log_source:
metadata.log_source="aws_cloudtrail" | stats count by api.operationOr search for specific activity. The CloudTrail parser maps eventName to api.operation, the caller to actor.user.name and sourceIPAddress to src_endpoint.ip:
metadata.log_source="aws_cloudtrail" api.operation="CreateUser"
| table time, actor.user.name, src_endpoint.ip, api.operationCheck for Errors
If no data appears:
-
Check SQS queue depth. Are messages accumulating?
aws sqs get-queue-attributes \ --queue-url "$QUEUE_URL" \ --attribute-names ApproximateNumberOfMessages \ --query 'Attributes.ApproximateNumberOfMessages' --output text- Messages accumulating: nano isn't polling. Check credentials and the log source deployment status.
- Zero messages: Either no new S3 objects are being written, or the S3 notification isn't configured correctly. Go back to Step 2 and verify.
-
Check Vector logs for connection errors:
docker logs nano-vector 2>&1 | grep -i "aws\|s3\|sqs\|error"
AWS Service-Specific Setup Notes
CloudTrail
If you don't already have CloudTrail logging to S3:
# Create a trail (if you don't have one)
aws cloudtrail create-trail \
--name nanosiem-trail \
--s3-bucket-name YOUR_BUCKET_NAME \
--is-multi-region-trail \
--enable-log-file-validation
# Start logging
aws cloudtrail start-logging --name nanosiem-trailCloudTrail delivers logs as gzipped JSON files with a Records array. Each file contains multiple events. nano's parser handles unwrapping the Records array automatically with the CloudTrail parser from the catalog.
VPC Flow Logs
To deliver VPC Flow Logs to S3 in JSON format:
aws ec2 create-flow-logs \
--resource-type VPC \
--resource-ids vpc-0abc123def456 \
--traffic-type ALL \
--log-destination-type s3 \
--log-destination arn:aws:s3:::YOUR_BUCKET_NAME/vpcflowlogs/ \
--log-format '${version} ${account-id} ${interface-id} ${srcaddr} ${dstaddr} ${srcport} ${dstport} ${protocol} ${packets} ${bytes} ${start} ${end} ${action} ${log-status}' \
--max-aggregation-interval 60GuardDuty
GuardDuty findings can be exported to S3 via an export configuration:
- In the GuardDuty console, go to Settings → Findings export options
- Set the S3 bucket and a KMS key for encryption
- Choose export frequency (every 15 minutes for updated findings)
Alternatively, use EventBridge to route GuardDuty findings to S3 for real-time delivery:
# Create an EventBridge rule for GuardDuty findings
aws events put-rule \
--name nanosiem-guardduty-findings \
--event-pattern '{"source": ["aws.guardduty"], "detail-type": ["GuardDuty Finding"]}'
# Set S3 as the target (via Firehose for batching)
# Or send directly to nano's HTTP endpoint for real-time ingestionALB/NLB Access Logs
Enable access logging on your load balancer:
aws elbv2 modify-load-balancer-attributes \
--load-balancer-arn arn:aws:elasticloadbalancing:us-east-1:ACCOUNT_ID:loadbalancer/app/my-alb/abc123 \
--attributes Key=access_logs.s3.enabled,Value=true \
Key=access_logs.s3.bucket,Value=YOUR_BUCKET_NAME \
Key=access_logs.s3.prefix,Value=alb-logsMultiple Log Types from One Bucket
If multiple AWS services write to the same S3 bucket (common with centralized logging), you have two options:
Option A: One Queue per Log Type (Recommended)
Create separate SQS queues with S3 prefix filters:
{
"QueueConfigurations": [
{
"Id": "cloudtrail-notifications",
"QueueArn": "arn:aws:sqs:us-east-1:ACCOUNT_ID:nanosiem-cloudtrail",
"Events": ["s3:ObjectCreated:*"],
"Filter": {
"Key": {
"FilterRules": [
{"Name": "prefix", "Value": "AWSLogs/123456789012/CloudTrail/"}
]
}
}
},
{
"Id": "vpcflow-notifications",
"QueueArn": "arn:aws:sqs:us-east-1:ACCOUNT_ID:nanosiem-vpcflow",
"Events": ["s3:ObjectCreated:*"],
"Filter": {
"Key": {
"FilterRules": [
{"Name": "prefix", "Value": "AWSLogs/123456789012/vpcflowlogs/"}
]
}
}
}
]
}Then create one nano log source per queue, each with its own parser.
Option B: One Shared Queue
Use a single SQS queue for all S3 notifications and let nano route based on the S3 object key prefix. This reduces queue setup but requires routing rules for each prefix. See Push vs Pull Routing for the per-prefix routing-rules pattern.
Troubleshoot
Need help with these checks? Contact support with the deployment, request ID and time with timezone.
"Access Denied" in Vector logs
- Verify the IAM user has
sqs:ReceiveMessage,sqs:DeleteMessage, ands3:GetObjectpermissions - Check the resource ARNs in the policy match your actual queue and bucket
- If using cross-account access, verify the trust policy on the assumed role
Messages in SQS but no events in nano
- Check the pull source is deployed and the log source's parser is published; look for the "Unpublished changes" banner
- Check the object is newer than the source: a new source skips notifications from before it was created (see Cloud pull)
- Verify the SQS Queue URL in the log source config matches exactly (no trailing slash)
- Check the region matches between the credential, log source config, and actual SQS queue
- Check whether events are landing unparsed: a parser that can't read an event emits it as Base Event (
class_uid0)
S3 notifications not arriving in SQS
- Verify the SQS queue policy allows
s3.amazonaws.comto send messages (Step 1) - Check the S3 bucket notification configuration is applied:
aws s3api get-bucket-notification-configuration --bucket YOUR_BUCKET_NAME - Ensure the suffix/prefix filter matches the actual object keys
- S3 notifications can take a few seconds to propagate after configuration
"KMS Decrypt" errors
If your S3 objects are encrypted with a KMS key, add kms:Decrypt permission to the IAM policy:
{
"Sid": "KMSDecrypt",
"Effect": "Allow",
"Action": "kms:Decrypt",
"Resource": "arn:aws:kms:us-east-1:ACCOUNT_ID:key/KEY_ID"
}Recommended Log Sources for Security
If you're starting from scratch, prioritize these AWS log sources:
| Priority | Source | Why |
|---|---|---|
| 1 | CloudTrail | All API activity: IAM changes, resource creation, authentication events |
| 2 | GuardDuty | AWS-native threat detection findings |
| 3 | VPC Flow Logs | Network traffic visibility for lateral movement and exfiltration detection |
| 4 | ALB/NLB Access Logs | Web application activity and potential attack patterns |
| 5 | WAF Logs | Web attack attempts blocked/allowed by AWS WAF |
| 6 | S3 Access Logs | Data access auditing for sensitive buckets |
What's next
- Compare pull permissions
- Create detection rules for your AWS logs
- Configure enrichment to add GeoIP and threat intel to AWS source IPs
- Set up the GCP integration if you also use Google Cloud