Skip to main content

​
When to Use

The S3 Log Forwarder is a good fit when:
  • AWS services deliver log files directly to an S3 bucket (CloudFront, ALB, NLB, VPC Flow Logs, and others)
  • You want a low-complexity setup with no CloudWatch ingestion fees
  • You prefer Terraform or CloudFormation for infrastructure deployment
For application logs or traces from running workloads, see ECS FireLens, ADOT, or the overview for a full comparison.

​
Supported AWS Services

The S3 Log Forwarder is the recommended path for AWS services that deliver logs directly to S3:
ServiceLog type
Amazon CloudFrontStandard access logs
Application Load Balancer (ALB)Access logs
Network Load Balancer (NLB)Access logs
VPC Flow LogsNetwork flow records (S3 delivery)
AWS WAFWeb ACL traffic logs
AWS CloudTrailManagement and data event log files
AWS GuardDutyFindings export
Amazon S3Server access logs
Any other AWS service that delivers logs to an S3 bucket can use this forwarder. See the AWS documentation on services that publish to S3 for the full list.

​
How it Works

The Bronto AWS Log Forwarder is built around an AWS Lambda function that processes and forwards logs to Bronto. The function is triggered through S3 or EventBridge event notifications when new objects are written to your bucket. Supported AWS log sources include AWS CloudFront, Application Load Balancer (ALB), Network Load Balancer (NLB), VPC Flow Logs, and others. See the AWS documentation for the full list of services that deliver to S3.

​
Setup Instructions

1

Deploy the forwarder

Choose your preferred deployment method:Terraform (recommended) — Deploys the Lambda forwarder along with the necessary IAM roles and policies. Supports routing different log types to different destinations within Bronto. Full details and usage instructions are in the Terraform module repository.CloudFormation — Use the CloudFormation template in the AWS Console to create the stack by filling out a form.Manual — The Lambda function code and a pre-packaged deployment artifact are available in the GitHub repository.
2

Configure S3 event notifications

Configure your S3 bucket to trigger the Lambda function when new objects are created. Refer to the AWS documentation on S3 notifications for setup steps.

​
Data Organization

The forwarder Lambda controls how data lands in Bronto through its configuration, not via HTTP headers — it composes the x-bronto-* headers internally based on the per-source mapping you provide. See Data Organization for how datasets, collections, and tags work in Bronto. Map each S3 source (bucket name or CloudFront distribution ID) to its target dataset, collection, and log type using the destination_config variable on the Terraform module: 
module "bronto_aws_log_forwarding" {
  source = "git::https://github.com/brontoio/brontobytes-aws-ingestion-terraform.git//aws_log_forwarder"
  # ... bronto_api_key, artifact_bucket, logging_bucket, etc.

  destination_config = {
    "<S3_BUCKET_NAME>" = {
      dataset    = "<YOUR_DATASET>"
      collection = "<YOUR_COLLECTION>"
      log_type   = "s3_access_log"
    }
    "<CLOUDFRONT_DISTRIBUTION_ID>" = {
      dataset    = "<YOUR_DATASET>"
      collection = "<YOUR_COLLECTION>"
      log_type   = "cf_standard_access_log"
    }
  }

  bronto_tags = { environment = "production", region = "eu-west-1" }
}
  • dataset / collection route each source’s logs to a specific Bronto destination.
  • log_type tells the forwarder which parser to apply — supported values include s3_access_log, alb_access_log, nlb_access_log, clb_access_log, vpc_flow_log, cloudtrail, cf_standard_access_log, cf_realtime_access_log, bedrock_s3, and default (no parsing, for already-structured JSON).
  • bronto_tags apply globally to all datasets produced by this forwarder. A per-source tags map inside a destination_config entry adds or overrides tags for that source only.
  • For sources whose S3 object keys don’t match the default AWS naming patterns (for example, logs moved between buckets), use paths_regex with a dest_config_id capture group to map them to destination_config entries.
For the full variable reference, see the Terraform module README.

​
Configure direct Lambda deployments

If you deploy the Python Lambda directly (CloudFormation or manual), the same configuration is provided as Lambda environment variables:
Environment variablePurpose
bronto_api_keyBronto API key
bronto_endpointRegional ingestion endpoint, e.g. https://ingestion.eu.bronto.io/
destination_configBase64-encoded JSON mapping S3 resource names to dataset, collection, log_type, and optional tags
tagsComma-separated key=value pairs applied to all datasets — e.g. environment=production,region=eu-west-1
paths_regexBase64-encoded list of regex patterns for non-standard S3 key layouts
See the Python forwarder README for a sample destination_config payload and the full attribute reference.

​
Cost Notes

  • No CloudWatch ingestion fees — logs are read directly from S3 by the Lambda function.
  • You pay for S3 storage and Lambda invocations, both of which are minimal for typical log volumes.
For assistance, contact support@bronto.io.