> ## Documentation Index
> Fetch the complete documentation index at: https://docs.bronto.io/llms.txt
> Use this file to discover all available pages before exploring further.

# AWS Kinesis Firehose (Preview)

> Stream high-volume AWS log data into Bronto with a native Amazon Kinesis Data Firehose delivery stream that batches and retries HTTP delivery automatically.

## Overview

Bronto's Kinesis Firehose integration lets you stream data directly from an [Amazon Data Firehose](https://docs.aws.amazon.com/firehose/latest/dev/what-is-this-service.html) delivery stream into Bronto over HTTP. It is well suited to high-throughput pipelines where data already flows through Kinesis, to services that deliver natively to Firehose (CloudFront real-time logs, Route 53 Resolver query logs, SES events, ElastiCache logs), and to CloudWatch-native services routed to Firehose via a subscription filter (CloudTrail, RDS/Aurora, API Gateway, Route 53 public query logging).

The integration requires no Lambda function. Firehose delivers data directly to Bronto's HTTP endpoint, handling batching, compression, and retries natively.

## When to Use Firehose

| Situation                                       | Recommendation                                                                            |
| ----------------------------------------------- | ----------------------------------------------------------------------------------------- |
| Data already in a Kinesis Data Stream           | Firehose is the natural choice                                                            |
| A service that delivers natively to Firehose    | Firehose captures the full-fidelity record                                                |
| Very high event volume (GB/hour+)               | Firehose scales without additional Lambda cost                                            |
| You want managed delivery with built-in retries | Firehose handles this natively                                                            |
| Application or container logs                   | Use [ADOT](./aws-adot) or the [CloudWatch Log Forwarder](./aws-client-cloudwatch) instead |

For low or moderate volume sources, the [S3 Log Forwarder](./aws-client-s3) or [CloudWatch Log Forwarder](./aws-client-cloudwatch) are simpler to operate.

***

## Bronto Ingestion Endpoint

Firehose delivers to Bronto's dedicated AWS HTTP endpoint:

| Region | Endpoint URL                               |
| ------ | ------------------------------------------ |
| EU     | `https://ingestion.eu.bronto.io/aws/logs/` |
| US     | `https://ingestion.us.bronto.io/aws/logs/` |

Configure the delivery stream's HTTP endpoint destination as follows:

| Setting           | Value                                                                            |
| ----------------- | -------------------------------------------------------------------------------- |
| HTTP endpoint URL | `https://ingestion.<REGION>.bronto.io/aws/logs/`                                 |
| Access key        | Your Bronto API key — sent by Firehose in the `X-Amz-Firehose-Access-Key` header |
| Content encoding  | **GZIP**                                                                         |

See [API Keys](/Account-Management/API-Keys) for how to generate a key.

<Note>
  No `log_format` parameter is needed. Bronto auto-detects the payload shape — it unwraps CloudWatch Logs subscription envelopes and parses JSON record bodies automatically. See [Supported record formats](#supported-record-formats) below.
</Note>

A successful delivery returns `200` with `Content-Type: application/json` and a body of the form `{"requestId":"<echoed-id>","timestamp":<epoch_ms>}`. Firehose may send up to 500 records per request.

***

## Data Organization

Set Bronto's routing headers as the HTTP endpoint **Common attributes** on the delivery stream. Firehose delivers them in the `X-Amz-Firehose-Common-Attributes` JSON header. See [Data Organization](/Search-and-Visualize/Partitions) for how datasets, collections, and tags work.

| Common attribute      | Description                              |
| --------------------- | ---------------------------------------- |
| `x-bronto-dataset`    | Dataset to ingest into                   |
| `x-bronto-collection` | Collection name                          |
| `x-bronto-tags`       | Comma-separated tags to attach to events |

To route different sources to different datasets, deploy a separate Firehose delivery stream per dataset and set its common attributes accordingly.

***

## Supported record formats

Bronto detects field extraction from the **record body**, not the service that produced it. Two tiers of record body are decoded automatically; delimited and positional text formats are stored raw and require a [custom parser](/core-features/custom-parser).

### CloudWatch Logs subscription format

Services that reach Firehose through **CloudWatch Logs → subscription filter → Firehose** deliver a gzip-compressed CloudWatch Logs subscription envelope (`DATA_MESSAGE` containing one or more `logEvents`). Bronto unwraps it automatically:

* One stored event per `logEvent`.
* CloudWatch Logs metadata is preserved under `$logGroup`, `$logStream`, `$messageType`, `$owner`, and related fields.
* Firehose delivery metadata is preserved under `$firehose.requestId`, `$firehose.sourceArn`, and `$firehose.timestamp`.
* Each `logEvent` message is then parsed by its own format — JSON bodies are flattened (below), delimited text is stored raw.

This covers every service that exports via a CloudWatch Logs subscription filter, including CloudTrail, RDS/Aurora, API Gateway, and Route 53 public query logging.

### JSON record bodies

Records delivered directly to Firehose as JSON objects are parsed and flattened into searchable fields, with nested keys expressed in dot notation (for example `mail.destination.0`, `answers.0.Rdata`, `httpRequest.clientIp`). No configuration is required. Verified for Amazon SES events, Route 53 Resolver query logs, AWS WAF logs (native JSON), and ElastiCache logs **when the JSON log format is selected** on the AWS side.

<Note>
  AWS requires a WAF logging delivery stream's name to start with `aws-waf-logs-` before WAF will accept it as a destination.
</Note>

### Formats requiring a custom parser

Delimited and positional text records are accepted and stored raw (in `@raw`) but are **not** field-extracted automatically. Each requires a Bronto [custom parser](/core-features/custom-parser):

| Source                        | Record shape                                                                             |
| ----------------------------- | ---------------------------------------------------------------------------------------- |
| CloudFront real-time logs     | TAB-delimited, fixed field order, unselected fields rendered as `-`                      |
| Route 53 public query logging | Space-delimited line                                                                     |
| RDS / Aurora plain-text logs  | PostgreSQL `stderr`, MySQL slow-query blocks, Aurora audit CSV (heterogeneous)           |
| ElastiCache TEXT format       | Comma-delimited                                                                          |
| VPC Flow Logs                 | Space-delimited v2 positional line (`version account-id interface-id srcaddr dstaddr …`) |

<Warning>
  For positional formats (CloudFront real-time logs, VPC Flow Logs), the custom parser must be pinned to the exact field list and order configured on the AWS side — the order is fixed, but the selected subset varies per configuration, so map columns 1:1. Where a service offers a JSON log format (ElastiCache, API Gateway), prefer it to avoid writing a parser.
</Warning>

***

## Setup

### Step 1 — Create the Firehose delivery stream

In the AWS Console, go to **Amazon Data Firehose → Create Firehose stream**. Choose the **source** that matches your service:

* **Direct PUT** — for services that publish directly to Firehose (SES, Route 53 Resolver, ElastiCache).
* **Amazon Kinesis Data Streams** — for CloudFront real-time logs, which deliver to a Kinesis Data Stream first.

Set the **destination** to **HTTP Endpoint**.

<Frame>
  <img src="https://mintcdn.com/bronto/7L-HCrBNhuCImWiB/images/integrations/assets/images/firehose-create-stream.png?fit=max&auto=format&n=7L-HCrBNhuCImWiB&q=85&s=adbfa14301fcc414576a31fab7ff0e53" alt="Create Firehose stream with source Direct PUT and destination HTTP Endpoint" width="1928" height="1134" data-path="images/integrations/assets/images/firehose-create-stream.png" />
</Frame>

### Step 2 — Configure the HTTP endpoint destination

| Field             | Value                                            |
| ----------------- | ------------------------------------------------ |
| HTTP endpoint URL | `https://ingestion.<REGION>.bronto.io/aws/logs/` |
| Access key        | `<YOUR_API_KEY>`                                 |
| Content encoding  | `GZIP`                                           |

<Frame>
  <img src="https://mintcdn.com/bronto/7L-HCrBNhuCImWiB/images/integrations/assets/images/firehose-destination-settings.png?fit=max&auto=format&n=7L-HCrBNhuCImWiB&q=85&s=a3ebca098ef9a3ac68f33a342295c913" alt="Destination settings: Bronto HTTP endpoint URL, access key, and GZIP content encoding" width="1846" height="1224" data-path="images/integrations/assets/images/firehose-destination-settings.png" />
</Frame>

In the destination's **Parameters** section (Firehose delivers these as the `X-Amz-Firehose-Common-Attributes` header), add `x-bronto-collection`, `x-bronto-dataset`, and optionally `x-bronto-tags`. No `log_format` parameter is needed — Bronto auto-detects the payload shape.

<Frame>
  <img src="https://mintcdn.com/bronto/7L-HCrBNhuCImWiB/images/integrations/assets/images/firehose-parameters.png?fit=max&auto=format&n=7L-HCrBNhuCImWiB&q=85&s=c33fe451662936c8798f31f9c18d68ca" alt="Parameters section with x-bronto-collection, x-bronto-dataset, and x-bronto-tags key-value pairs" width="1730" height="568" data-path="images/integrations/assets/images/firehose-parameters.png" />
</Frame>

### Step 3 — Connect the source (CloudWatch-native services)

For services that have no native Firehose export (CloudTrail, RDS/Aurora, API Gateway, Route 53 public query logging), first publish the service's logs to a CloudWatch log group, then add a **subscription filter** on that log group targeting the Firehose stream. The subscription filter needs an IAM role trusting `logs.amazonaws.com` with permission to `firehose:PutRecord*` on the delivery stream.

<Note>
  CloudWatch Logs already compresses subscription data with gzip. Do not add a second compression layer in the delivery stream's data transformation — set only the destination content encoding to GZIP as above.
</Note>

For native-Firehose services, configure the service's own event/log destination to point at the delivery stream (see the per-service pages: [SES](./aws-ses), [CloudFront](./aws-cloudfront), [Route 53](./aws-route53), [ElastiCache](./aws-elasticache)).

***

## Cost

Firehose pricing is based on the volume of data processed through the delivery stream (per GB). There are no Lambda compute costs. Services routed through CloudWatch Logs incur standard CloudWatch Logs ingestion charges in addition to Firehose costs.

***

For assistance, contact [support@bronto.io](mailto:support@bronto.io).