> ## Documentation Index
> Fetch the complete documentation index at: https://docs.bronto.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Apache Kafka broker logs with OpenTelemetry

> Collect Apache Kafka broker, controller, and request logs with the OpenTelemetry Collector and forward them to Bronto to monitor cluster health and partition activity.

Collect Apache Kafka broker logs — the Log4j application logs such as `server.log` — with the OpenTelemetry Collector. The same Collector can also forward traces and metrics from your apps — see [Connect OpenTelemetry Collector to Bronto](/agent-setup/open-telemetry).

This covers the broker's operational logs, not the commit-log data Kafka stores for topics.

## Prerequisites

* A Bronto account and API key ([how to create one](/Account-Management/API-Keys#create-a-new-api-key))
* Kafka brokers writing Log4j logs to disk. The default location is `$KAFKA_HOME/logs/` (often `/var/log/kafka/`), with `server.log` as the main broker log
* An [OpenTelemetry Collector](https://opentelemetry.io/docs/collector/installation/) (recommended) or [Fluent Bit](https://docs.fluentbit.io/manual/installation/getting-started-with-fluent-bit) installed on the same host

## Configure the Collector

Kafka stack traces span multiple lines. Use `multiline` keyed on the Log4j `[timestamp]` prefix:

```yaml /etc/otel/config.yaml theme={"dark"}
receivers:
  filelog/kafka:
    include:
      - /var/log/kafka/server.log
      - /var/log/kafka/controller.log
    multiline:
      line_start_pattern: '^\[\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d{3}\]'
    resource:
      service.name: kafka
      service.namespace: <YOUR_COLLECTION_NAME>

processors:
  batch:

exporters:
  otlphttp/brontologs:
    logs_endpoint: "https://ingestion.<REGION>.bronto.io/v1/logs"
    compression: gzip
    headers:
      x-bronto-api-key: <YOUR_API_KEY>

service:
  pipelines:
    logs:
      receivers: [filelog/kafka]
      processors: [batch]
      exporters: [otlphttp/brontologs]
```

Set `<REGION>` to `eu` or `us`.

## What you will see in Bronto

Open [Search](https://app.bronto.io/search) and filter by `service.name = kafka`. Entries follow the Log4j pattern — timestamp, level (`INFO`, `WARN`, `ERROR`), logger, and message — covering broker lifecycle, controller and partition events, ISR changes, and exceptions with stack traces (kept intact by the multiline setting). Bronto's [Custom Parser](/core-features/custom-parser) extracts these fields server-side.

## Distributed tracing

The Kafka broker does not emit traces — Apache Kafka has no native tracing, so the entries above are the broker's operational logs only. To trace messages end to end, instrument your **producer and consumer applications** with the OpenTelemetry SDK: it creates send/receive spans and propagates trace context through Kafka message headers. Point those apps at the same Collector and traces flow to Bronto alongside these logs — see [Send Traces to Bronto](/tracing/send-traces) and the [OpenTelemetry SDK guides](/opentelemetry/overview). On Kubernetes, [Strimzi](https://strimzi.io/) can additionally enable OpenTelemetry tracing for Kafka Connect, MirrorMaker, and the Kafka Bridge.

## Troubleshooting

* **Where are the logs?** Check the `kafka.logs.dir` / `LOG_DIR` setting or the appender file path in your `log4j.properties`, and adjust `include`.
* **Want structured fields at the source?** Configure a Log4j JSON layout for direct field extraction.
* For general issues, see [OTel Collector troubleshooting](https://opentelemetry.io/docs/collector/troubleshooting/).

## Alternative: Fluent Bit

If you already run Fluent Bit, tail the broker log files and forward them over HTTP. The built-in `java` multiline parser keeps stack traces intact:

```ini fluent-bit.conf theme={"dark"}
[INPUT]
    name              tail
    path              /var/log/kafka/server.log,/var/log/kafka/controller.log
    multiline.parser  java
    tag               kafka

[OUTPUT]
    name        http
    match       kafka
    host        ingestion.<REGION>.bronto.io
    port        443
    tls         on
    format      json_lines
    compress    gzip
    header      x-bronto-api-key    <YOUR_API_KEY>
    header      x-bronto-dataset    kafka
    header      x-bronto-collection <YOUR_COLLECTION_NAME>
```

Set **Format** to `json_lines`, not `json`. See [Connect Fluent Bit to Bronto](/agent-setup/fluent-bit) for installation.