Skip to main content
Collect Apache Kafka broker logs — the Log4j application logs such as server.log — with the OpenTelemetry Collector. The same Collector can also forward traces and metrics from your apps — see Connect OpenTelemetry Collector to Bronto. This covers the broker’s operational logs, not the commit-log data Kafka stores for topics.

​
Prerequisites

  • A Bronto account and API key (how to create one)
  • Kafka brokers writing Log4j logs to disk. The default location is $KAFKA_HOME/logs/ (often /var/log/kafka/), with server.log as the main broker log
  • An OpenTelemetry Collector (recommended) or Fluent Bit installed on the same host

​
Configure the Collector

Kafka stack traces span multiple lines. Use multiline keyed on the Log4j [timestamp] prefix:
/etc/otel/config.yaml
receivers:
  filelog/kafka:
    include:
      - /var/log/kafka/server.log
      - /var/log/kafka/controller.log
    multiline:
      line_start_pattern: '^\[\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d{3}\]'
    resource:
      service.name: kafka
      service.namespace: <YOUR_COLLECTION_NAME>

processors:
  batch:

exporters:
  otlphttp/brontologs:
    logs_endpoint: "https://ingestion.<REGION>.bronto.io/v1/logs"
    compression: gzip
    headers:
      x-bronto-api-key: <YOUR_API_KEY>

service:
  pipelines:
    logs:
      receivers: [filelog/kafka]
      processors: [batch]
      exporters: [otlphttp/brontologs]
Set <REGION> to eu or us.

​
What you will see in Bronto

Open Search and filter by service.name = kafka. Entries follow the Log4j pattern — timestamp, level (INFO, WARN, ERROR), logger, and message — covering broker lifecycle, controller and partition events, ISR changes, and exceptions with stack traces (kept intact by the multiline setting). Bronto’s Custom Parser extracts these fields server-side.

​
Distributed tracing

The Kafka broker does not emit traces — Apache Kafka has no native tracing, so the entries above are the broker’s operational logs only. To trace messages end to end, instrument your producer and consumer applications with the OpenTelemetry SDK: it creates send/receive spans and propagates trace context through Kafka message headers. Point those apps at the same Collector and traces flow to Bronto alongside these logs — see Send Traces to Bronto and the OpenTelemetry SDK guides. On Kubernetes, Strimzi can additionally enable OpenTelemetry tracing for Kafka Connect, MirrorMaker, and the Kafka Bridge.

​
Troubleshooting

  • Where are the logs? Check the kafka.logs.dir / LOG_DIR setting or the appender file path in your log4j.properties, and adjust include.
  • Want structured fields at the source? Configure a Log4j JSON layout for direct field extraction.
  • For general issues, see OTel Collector troubleshooting.

​
Alternative: Fluent Bit

If you already run Fluent Bit, tail the broker log files and forward them over HTTP. The built-in java multiline parser keeps stack traces intact:
fluent-bit.conf
[INPUT]
    name              tail
    path              /var/log/kafka/server.log,/var/log/kafka/controller.log
    multiline.parser  java
    tag               kafka

[OUTPUT]
    name        http
    match       kafka
    host        ingestion.<REGION>.bronto.io
    port        443
    tls         on
    format      json_lines
    compress    gzip
    header      x-bronto-api-key    <YOUR_API_KEY>
    header      x-bronto-dataset    kafka
    header      x-bronto-collection <YOUR_COLLECTION_NAME>
Set Format to json_lines, not json. See Connect Fluent Bit to Bronto for installation.