> ## Documentation Index
> Fetch the complete documentation index at: https://docs.bronto.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Zero-code instrumentation with eBPF

> Instrument applications with OpenTelemetry eBPF Instrumentation (OBI) to send distributed traces and RED metrics to Bronto with no code changes, no SDK, and no redeploy.

[OpenTelemetry eBPF Instrumentation](https://opentelemetry.io/docs/zero-code/obi/) (OBI) produces distributed traces and RED metrics for services that contain no instrumentation at all — no SDK dependency, no initialisation code, and no redeploy. It attaches eBPF programs to already-running processes and reads HTTP and gRPC activity as it crosses the kernel boundary.

This makes it the fastest way to get coverage across an estate, and the only practical option for services whose build you do not control: vendored binaries, legacy applications, and third-party components.

<Note>
  OBI is the OpenTelemetry project donated from Grafana Beyla. If you have used Beyla, the configuration keys are the same with an `OTEL_EBPF_` prefix instead of `BEYLA_`.
</Note>

## When to use eBPF instead of an SDK

eBPF observes the process boundary, so it sees requests between services rather than logic inside them. The two approaches complement each other and land in Bronto through the same OTLP pipeline, with matching trace IDs.

|                      | eBPF (OBI)                                       | OpenTelemetry SDK              |
| -------------------- | ------------------------------------------------ | ------------------------------ |
| Code changes         | None                                             | Dependency plus initialisation |
| Redeploy needed      | No                                               | Yes                            |
| Coverage             | HTTP and gRPC at the process boundary            | Anything you instrument        |
| Custom attributes    | No                                               | Yes                            |
| Business-logic spans | No                                               | Yes                            |
| Logs                 | No — [ship separately](/agent-setup/agent-intro) | Yes, via the log bridge        |

**Use eBPF** for immediate breadth, for services you cannot rebuild, and to find out where latency and errors actually are. **Add an SDK** in the services that then warrant domain detail. See [Choose your language](/opentelemetry/overview) for SDK setup.

## Requirements

| Requirement  | Detail                                                                                                                          |
| ------------ | ------------------------------------------------------------------------------------------------------------------------------- |
| Kernel       | Linux 5.8+; **5.17+** for distributed traces across services                                                                    |
| BTF          | `/sys/kernel/btf/vmlinux` must be present                                                                                       |
| Architecture | amd64 or arm64                                                                                                                  |
| Privileges   | Root, or the [documented capability set](https://opentelemetry.io/docs/zero-code/obi/security/), plus host PID namespace access |

Check a host before you start:

```bash theme={"dark"}
uname -r && ls -l /sys/kernel/btf/vmlinux
```

<Tip>
  This also works on Docker Desktop for macOS, which runs a LinuxKit VM with BTF enabled — useful for evaluating OBI before deploying it to a cluster.
</Tip>

## Prerequisites

* A Bronto account and API key ([how to create one](/Account-Management/API-Keys#create-a-new-api-key))
* An [OpenTelemetry Collector](https://opentelemetry.io/docs/collector/installation/) — recommended, and required if you want delta metrics
* A host or cluster meeting the requirements above

## Run OBI with Docker Compose

OBI runs as one container alongside your services. It needs the host PID namespace to see other containers' processes, and elevated privileges to load eBPF programs.

```yaml docker-compose.yml theme={"dark"}
services:
  obi:
    image: otel/ebpf-instrument:main
    pid: "host"
    privileged: true
    environment:
      OTEL_EBPF_CONFIG_PATH: /etc/obi/obi.yml
    volumes:
      - ./obi.yml:/etc/obi/obi.yml:ro
    depends_on: [otel-collector]
```

OBI discovers what to instrument by the ports your services listen on — the applications themselves are not modified or restarted:

```yaml obi.yml theme={"dark"}
discovery:
  instrument:
    # The ports your services listen on.
    - open_ports: 8080,3000,5000
      containers_only: true

  exclude_instrument:
    # Publishing a port makes the Docker daemon listen on it too, so a
    # port selector matches dockerd unless you exclude it explicitly.
    - exe_path: "*dockerd*"
    - exe_path: "*docker-proxy*"
    - exe_path: "*containerd*"
    - exe_path: "*otelcol*"

ebpf:
  # Inject W3C trace context into outgoing requests so spans from different
  # services join into one distributed trace. Requires kernel 5.17+.
  context_propagation: all

otel_traces_export:
  endpoint: http://otel-collector:4318

otel_metrics_export:
  endpoint: http://otel-collector:4318
  interval: 15s
  features:
    - application
    - application_span
```

## Run OBI on Kubernetes

Deploy OBI as a DaemonSet so every node instruments its own pods. The container needs `hostPID: true` and the eBPF capabilities:

```yaml obi-daemonset.yaml theme={"dark"}
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: obi
spec:
  selector:
    matchLabels: { app: obi }
  template:
    metadata:
      labels: { app: obi }
    spec:
      hostPID: true
      containers:
        - name: obi
          image: otel/ebpf-instrument:main
          securityContext:
            privileged: true
          env:
            - name: OTEL_EBPF_CONFIG_PATH
              value: /etc/obi/obi.yml
            - name: OTEL_EBPF_BPF_CONTEXT_PROPAGATION
              value: all
          volumeMounts:
            - { name: config, mountPath: /etc/obi }
      volumes:
        - name: config
          configMap: { name: obi-config }
```

On Kubernetes you can select workloads by namespace, labels, or annotations instead of ports:

```yaml obi.yml theme={"dark"}
discovery:
  instrument:
    - k8s_namespace: production
      k8s_pod_labels:
        instrument: obi
```

See [Kubernetes logs and metrics with OpenTelemetry](/integrations/kubernetes) for collecting pod logs alongside this.

## Forward to Bronto

Point the Collector's OTLP receiver at Bronto's trace and metric endpoints. Traces and metrics use separate endpoints and the same API key header:

```yaml otel-config.yaml theme={"dark"}
receivers:
  otlp:
    protocols:
      grpc: { endpoint: 0.0.0.0:4317 }
      http: { endpoint: 0.0.0.0:4318 }

processors:
  batch:
  # OBI exports cumulative metrics; Bronto strongly prefers delta.
  cumulativetodelta:
    initial_value: drop

exporters:
  otlphttp/brontotraces:
    traces_endpoint: "https://ingestion.<REGION>.bronto.io/v1/traces"
    compression: gzip
    headers:
      x-bronto-api-key: <YOUR_API_KEY>
  otlphttp/brontometrics:
    metrics_endpoint: "https://ingestion.<REGION>.bronto.io/v1/metrics"
    compression: gzip
    headers:
      x-bronto-api-key: <YOUR_API_KEY>

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlphttp/brontotraces]
    metrics:
      receivers: [otlp]
      processors: [cumulativetodelta, batch]
      exporters: [otlphttp/brontometrics]
```

Replace `<REGION>` with `eu` or `us` to match your account. A mismatched region returns `401`, which looks identical to an invalid key.

<Warning>
  Do not set `histogram_aggregation: base2_exponential_bucket_histogram`. Bronto does not currently ingest exponential histograms. OBI's default, `explicit_bucket_histogram`, is the correct setting.
</Warning>

## Name your services

By default OBI names a service after its executable. That is fine for a compiled binary, but interpreted runtimes all end up named after the interpreter — `node`, `python3.13`, `java`.

OBI reads `OTEL_SERVICE_NAME` and `OTEL_RESOURCE_ATTRIBUTES` from each target process's **own environment**, so you can name services without adding an SDK or touching application code:

```yaml docker-compose.yml theme={"dark"}
services:
  checkout-api:
    image: your-app:latest
    environment:
      OTEL_SERVICE_NAME: checkout-api
      OTEL_RESOURCE_ATTRIBUTES: service.namespace=production
```

Nothing inside the container reads these variables — OBI does, from the outside.

<Note>
  Traces are stored in Bronto's `.traces` collection, with one dataset per `service.name`. `service.namespace` is carried on every span and is the attribute to filter on to isolate an environment, but it does not determine where traces are stored.
</Note>

## What you will see in Bronto

OBI emits standard OpenTelemetry semantic conventions, so no Bronto-specific configuration is required.

**Traces.** Each request produces a server span, client spans for outgoing calls, and — for Go services — an `in queue` and `processing` breakdown that separates time spent waiting to be handled from time spent being handled:

```
checkout-api    SERVER    GET /checkout    50.65ms
checkout-api    INTERNAL  in queue          0.20ms
checkout-api    INTERNAL  processing       50.45ms
checkout-api    CLIENT    GET /orders      49.93ms
  orders-api    SERVER    GET /orders      48.28ms
  orders-api    CLIENT    GET /inventory   46.54ms
    inventory   SERVER    GET /inventory   46.35ms
```

**Metrics.** RED metrics per route and status code:

```
http.server.request.duration
http.server.request.body.size
http.server.response.body.size
http.client.request.duration
traces_spanmetrics_calls_total
traces_spanmetrics_latency
```

**Searchable span fields.** `span.trace_id`, `span.parent_span_id`, `span.kind`, `span.duration_nano`, `http.route`, `http.request.method`, `http.response.status_code`, `service.name`, `service.namespace`, and `url.path`.

Spans produced by OBI carry `telemetry.distro.name = opentelemetry-ebpf-instrumentation`, which is a convenient way to separate eBPF-derived data from SDK-derived data in queries.

## Verify

<Steps>
  <Step title="Confirm OBI attached to your processes">
    OBI logs one line per instrumented process, including the runtime it detected:

    ```
    instrumenting process cmd=/usr/local/bin/checkout-api type=go
    instrumenting process cmd=/usr/local/bin/node          type=nodejs
    instrumenting process cmd=/usr/local/bin/python3.13    type=python
    ```

    If nothing appears, your selector matched no processes — check `open_ports` against the ports your services actually listen on.
  </Step>

  <Step title="Print spans without leaving the terminal">
    Set `trace_printer: text` in `obi.yml` to have OBI print every captured span to stdout as well as exporting it. Send a request through your service and confirm spans appear, then set it back to `disabled`.
  </Step>

  <Step title="Check delivery to Bronto">
    Look for export errors in the Collector's logs:

    ```bash theme={"dark"}
    docker compose logs otel-collector | grep -i "exporting failed"
    ```

    A `401` means the API key or the region is wrong.
  </Step>

  <Step title="Find the data in Bronto">
    Open [Explore Traces](/tracing/explore-traces) and filter on your `service.name`, or query `service.namespace` to see the whole environment. Metrics appear in the Metric Explorer under `http.server.request.duration`.
  </Step>
</Steps>

## Troubleshooting

* **Spans appear but never join into a trace.** Context propagation is off by default. Set `context_propagation: all` and confirm the kernel is 5.17+.
* **A service you did not expect is being instrumented.** Publishing a container port makes the Docker daemon listen on it too. Add `dockerd`, `docker-proxy`, and `containerd` to `exclude_instrument`.
* **Go services produce fewer details than expected.** OBI attaches uprobes to Go runtime symbols. Building with `-ldflags="-s -w"` strips the symbol table and silently degrades OBI to generic syscall tracing — leave Go binaries unstripped.
* **`creating OTEL namespace in bpffs failed`.** `/sys/fs/bpf` is not mounted. Core tracing still works; features that rely on pinned maps, such as the log enricher, are disabled. Mount bpffs to restore them.
* **`FIONREAD compensation is ineffective`.** Applications that size reads via `FIONREAD` — nginx, Java, and .NET among them — may stall or truncate transfers while context propagation is enabled. Validate against those runtimes, or set `context_propagation: disabled` for them.
* **Metrics rejected or missing.** Confirm you are not exporting exponential histograms, and that `cumulativetodelta` is in the metrics pipeline.

## Security considerations

Running with `privileged: true` is the documented starting point and the simplest way to prove the setup works, but it grants more than OBI needs. For production, use the [least-privilege capability set](https://opentelemetry.io/docs/zero-code/obi/security/) — `CAP_BPF`, `CAP_PERFMON`, `CAP_NET_RAW`, and `CAP_DAC_READ_SEARCH` among others, depending on the features you enable.

Set `OTEL_EBPF_ENFORCE_SYS_CAPS=true` so OBI fails loudly when a required capability is missing instead of degrading quietly.

Host PID namespace access is not optional: OBI must see other processes in order to attach to them.

<Note>
  Go library-level context propagation relies on `bpf_probe_write_user`, which is blocked by Secure Boot and kernel lockdown mode. Network-level propagation still works in those environments.
</Note>

## Next steps

* [Explore Traces](/tracing/explore-traces) — investigate services, latency, and errors
* [Send Metrics to Bronto](/metrics/send-metrics) — metric support and limitations
* [Choose your language](/opentelemetry/overview) — add SDK instrumentation where you need business context
* [Agent Setup](/agent-setup/agent-intro) — ship logs, which OBI does not produce
* [OBI documentation](https://opentelemetry.io/docs/zero-code/obi/) — full configuration reference