Kubernetes logs and metrics with OpenTelemetry

Collect pod logs and kubelet metrics across a Kubernetes cluster by running the OpenTelemetry Collector as a DaemonSet. The same Collector can receive application logs, metrics, and traces over OTLP.

Prerequisites

A Bronto account and API key (how to create one)
A Kubernetes cluster with kubectl and Helm access
Pods writing to stdout/stderr (collected from /var/log/pods on each node)

Deploy the Collector DaemonSet

Install the Collector with the OpenTelemetry Helm chart. Its logsCollection preset wires up the filelog receiver and node log mounts; the kubernetesAttributes preset adds pod, namespace, and node metadata and the RBAC it needs:

values.yaml

mode: daemonset

presets:
  logsCollection:
    enabled: true
  kubernetesAttributes:
    enabled: true
  kubeletMetrics:
    enabled: true

config:
  exporters:
    otlphttp/brontologs:
      logs_endpoint: "https://ingestion.<REGION>.bronto.io/v1/logs"
      compression: gzip
      headers:
        x-bronto-api-key: <YOUR_API_KEY>
        x-bronto-collection: <CLUSTER_NAME>
    otlphttp/brontometrics:
      metrics_endpoint: "https://ingestion.<REGION>.bronto.io/v1/metrics"
      compression: gzip
      headers:
        x-bronto-api-key: <YOUR_API_KEY>
  service:
    pipelines:
      logs:
        exporters: [otlphttp/brontologs]
      metrics:
        exporters: [otlphttp/brontometrics]

helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts
helm install otel-collector open-telemetry/opentelemetry-collector -f values.yaml

Set <REGION> to eu or us. For logs, use x-bronto-collection to identify the cluster; Bronto infers per-workload log datasets from the Kubernetes metadata. Metrics appear in the Metric Explorer and do not belong to that collection or those datasets.

Volume metrics, extra pod metadata, and request/limit utilization require the Collector service account to read the kubelet /pods endpoint through the nodes/proxy subresource. Add that RBAC permission if you enable those measurements. Use the Kubernetes or contrib Collector distribution because the core image does not include every Kubernetes receiver.

For receiver settings, authentication, and available measurements, see the Collector Contrib Kubelet Stats receiver.

Cluster events

The DaemonSet above collects pod logs only. To also capture Kubernetes events (scheduling, image pulls, OOMKills, evictions), run a second Collector as a single-replica Deployment with the kubernetesEvents preset — it adds the k8sobjects receiver and the RBAC to watch events. Use a Deployment, not the DaemonSet, so events are not duplicated across nodes:

events-values.yaml

mode: deployment
replicaCount: 1

presets:
  kubernetesEvents:
    enabled: true

config:
  exporters:
    otlphttp/brontologs:
      logs_endpoint: "https://ingestion.<REGION>.bronto.io/v1/logs"
      compression: gzip
      headers:
        x-bronto-api-key: <YOUR_API_KEY>
        x-bronto-collection: <CLUSTER_NAME>
  service:
    pipelines:
      logs:
        exporters: [otlphttp/brontologs]

helm install otel-events open-telemetry/opentelemetry-collector -f events-values.yaml

This needs a Collector image that bundles the k8sobjects receiver (the k8s or contrib distribution). Events arrive in Bronto as log records carrying the event reason, message, involved object, and namespace.

What you will see in Bronto

Open Search for pod logs and the Metric Explorer for kubelet metrics. Kubernetes resource attributes identify the pod, namespace, node, and container.

Exponential histograms are not supported. For Sum, Summary, and Histogram metrics, the Metric Explorer provides per-second rate functions.

This setup collects pod logs and kubelet metrics. To add distributed traces and RED metrics for your workloads without instrumenting them, deploy OpenTelemetry eBPF Instrumentation as a DaemonSet alongside this Collector.

Troubleshooting

No logs? Confirm the DaemonSet is running on every node (kubectl get pods -l app.kubernetes.io/name=opentelemetry-collector) and that nodes expose logs under /var/log/pods.
Missing pod metadata? The kubernetesAttributes preset creates the required RBAC by default — confirm it is enabled.
Kubelet returns 403 for /pods? Grant get on the nodes/proxy subresource.
For general issues, see OTel Collector troubleshooting.

Alternative: Fluent Bit

Fluent Bit is a long-established way to collect Kubernetes pod logs and a fine choice if your organization already runs it. Deploy it as a DaemonSet via the official Helm chart:

fluent-bit.conf

[INPUT]
    name              tail
    path              /var/log/containers/*.log
    multiline.parser  docker, cri
    tag               kube.*

[FILTER]
    name                kubernetes
    match               kube.*
    k8s-logging.parser  on
    k8s-logging.exclude on

[OUTPUT]
    name                http
    match               kube.*
    host                ingestion.<REGION>.bronto.io
    port                443
    tls                 on
    format              json_lines
    compress            gzip
    header              x-bronto-api-key    <YOUR_API_KEY>
    header              x-bronto-collection <CLUSTER_NAME>

Set Format to json_lines, not json. The kubernetes filter enriches each log with pod metadata, mirroring the kubernetesAttributes preset. Fluent Bit does not collect cluster events — use the OpenTelemetry Collector for those. See Connect Fluent Bit to Bronto for details.

Overview

Ingest with agents

Ingest from AWS

Ingest from Azure

Ingest from CDN and edge

Ingest from apps and platforms

Ingest from infrastructure and databases

Ingest OpenTelemetry data

Kubernetes logs and metrics with OpenTelemetry

Prerequisites

Deploy the Collector DaemonSet

Cluster events

What you will see in Bronto

Troubleshooting

Alternative: Fluent Bit

​Prerequisites

​Deploy the Collector DaemonSet

​Cluster events

​What you will see in Bronto

​Troubleshooting

​Alternative: Fluent Bit

Prerequisites

Deploy the Collector DaemonSet

Cluster events

What you will see in Bronto

Troubleshooting

Alternative: Fluent Bit