Skip to main content

​
Overview

Bronto’s Kinesis Firehose integration lets you stream logs directly from an Amazon Data Firehose delivery stream into Bronto over HTTP. It is well suited to high-throughput pipelines where data already flows through Kinesis, services that deliver natively to Firehose, and CloudWatch-native services routed to Firehose through a subscription filter. The integration requires no Lambda function. Firehose delivers data directly to Bronto’s HTTP endpoint, handling batching, compression, and retries natively. To export CloudWatch metrics rather than logs, use AWS Kinesis Firehose for CloudWatch Metrics.

​
When to Use Firehose

For low or moderate volume sources, the S3 Log Forwarder or CloudWatch Log Forwarder are simpler to operate.

​
Bronto Ingestion Endpoint

Firehose delivers to Bronto’s dedicated AWS HTTP endpoint: Configure the delivery stream’s HTTP endpoint destination as follows: See API Keys for how to generate a key.
No log_format parameter is needed. Bronto auto-detects the payload shape — it unwraps CloudWatch Logs subscription envelopes and parses JSON record bodies automatically. See Supported record formats below.
A successful delivery returns 200 with Content-Type: application/json and a body of the form {"requestId":"<echoed-id>","timestamp":<epoch_ms>}. Firehose may send up to 500 records per request.

​
Data Organization

Set Bronto’s routing headers as the HTTP endpoint Common attributes on the delivery stream. Firehose delivers them in the X-Amz-Firehose-Common-Attributes JSON header. See Data Organization for how datasets, collections, and tags work. To route different sources to different datasets, deploy a separate Firehose delivery stream per dataset and set its common attributes accordingly.

​
Supported record formats

Bronto detects field extraction from the record body, not the service that produced it. Two tiers of record body are decoded automatically; delimited and positional text formats are stored raw and require a custom parser.

​
CloudWatch Logs subscription format

Services that reach Firehose through CloudWatch Logs → subscription filter → Firehose deliver a gzip-compressed CloudWatch Logs subscription envelope (DATA_MESSAGE containing one or more logEvents). Bronto unwraps it automatically:
  • One stored event per logEvent.
  • CloudWatch Logs metadata is preserved under $logGroup, $logStream, $messageType, $owner, and related fields.
  • Firehose delivery metadata is preserved under $firehose.requestId, $firehose.sourceArn, and $firehose.timestamp.
  • Each logEvent message is then parsed by its own format — JSON bodies are flattened (below), delimited text is stored raw.
This covers every service that exports via a CloudWatch Logs subscription filter, including CloudTrail, RDS/Aurora, API Gateway, and Route 53 public query logging.

​
JSON record bodies

Records delivered directly to Firehose as JSON objects are parsed and flattened into searchable fields, with nested keys expressed in dot notation (for example mail.destination.0, answers.0.Rdata, httpRequest.clientIp). No configuration is required. Verified for Amazon SES events, Route 53 Resolver query logs, AWS WAF logs (native JSON), and ElastiCache logs when the JSON log format is selected on the AWS side.
AWS requires a WAF logging delivery stream’s name to start with aws-waf-logs- before WAF will accept it as a destination.

​
Formats requiring a custom parser

Delimited and positional text records are accepted and stored raw (in @raw) but are not field-extracted automatically. Each requires a Bronto custom parser:
For positional formats (CloudFront real-time logs, VPC Flow Logs), the custom parser must be pinned to the exact field list and order configured on the AWS side — the order is fixed, but the selected subset varies per configuration, so map columns 1:1. Where a service offers a JSON log format (ElastiCache, API Gateway), prefer it to avoid writing a parser.

​
Setup

​
Step 1 — Create the Firehose delivery stream

In the AWS Console, go to Amazon Data Firehose → Create Firehose stream. Choose the source that matches your service:
  • Direct PUT — for services that publish directly to Firehose (SES, Route 53 Resolver, ElastiCache).
  • Amazon Kinesis Data Streams — for CloudFront real-time logs, which deliver to a Kinesis Data Stream first.
Set the destination to HTTP Endpoint.
Create Firehose stream with source Direct PUT and destination HTTP Endpoint

​
Step 2 — Configure the HTTP endpoint destination

Destination settings: Bronto HTTP endpoint URL, access key, and GZIP content encoding
In the destination’s Parameters section (Firehose delivers these as the X-Amz-Firehose-Common-Attributes header), add x-bronto-collection, x-bronto-dataset, and optionally x-bronto-tags. No log_format parameter is needed — Bronto auto-detects the payload shape.
Parameters section with x-bronto-collection, x-bronto-dataset, and x-bronto-tags key-value pairs

​
Step 3 — Connect the source (CloudWatch-native services)

For services that have no native Firehose export (CloudTrail, RDS/Aurora, API Gateway, Route 53 public query logging), first publish the service’s logs to a CloudWatch log group, then add a subscription filter on that log group targeting the Firehose stream. The subscription filter needs an IAM role trusting logs.amazonaws.com with permission to firehose:PutRecord* on the delivery stream.
CloudWatch Logs already compresses subscription data with gzip. Do not add a second compression layer in the delivery stream’s data transformation — set only the destination content encoding to GZIP as above.
For native-Firehose services, configure the service’s own event/log destination to point at the delivery stream (see the per-service pages: SES, CloudFront, Route 53, ElastiCache).

​
Cost

Firehose pricing is based on the volume of data processed through the delivery stream (per GB). There are no Lambda compute costs. Services routed through CloudWatch Logs incur standard CloudWatch Logs ingestion charges in addition to Firehose costs.
For assistance, contact support@bronto.io.