> ## Documentation Index > Fetch the complete documentation index at: https://docs.bronto.io/llms.txt > Use this file to discover all available pages before exploring further. # Kong Gateway traces, logs, and metrics with OpenTelemetry > Use Kong's OpenTelemetry plugin to send traces, access logs, runtime logs, and metrics — including AI Gateway token usage — straight to Bronto over OTLP, with no Collector. Kong Gateway's `opentelemetry` plugin exports four streams over OTLP/HTTP: request traces, access logs, the gateway's runtime log, and metrics. Point all four at Bronto and the gateway becomes the only thing emitting telemetry — no Collector, no sidecar, no application instrumentation. It works the same for a self-managed gateway and a Konnect-managed data plane. ## Prerequisites * A Bronto account and API key ([how to create one](https://docs.bronto.io/Account-Management/API-Keys.md#create-a-new-api-key)) * Kong Gateway 3.13 or later for access logs and metrics (verified with 3.16). On 3.13, use the flat `access_logs_endpoint` field in place of `access_logs.endpoint` * A way to apply plugin configuration — the examples use [decK](https://developer.konghq.com/deck/), but the Admin API and Konnect UI take the same fields ## Enable tracing and runtime logs on the data plane Set these in the data plane's environment and restart it: ```bash theme={"dark"} KONG_TRACING_INSTRUMENTATIONS=all KONG_TRACING_SAMPLING_RATE=1.0 KONG_LOG_LEVEL=info ``` `KONG_TRACING_INSTRUMENTATIONS` is off by default, and without it the plugin has no spans to export. `KONG_LOG_LEVEL=info` is optional: at the default `notice` the runtime log channel is almost silent. Lower `KONG_TRACING_SAMPLING_RATE` for high-traffic gateways. ## Configure the OpenTelemetry plugin Add the plugin globally so it covers every route: ```yaml kong.yaml theme={"dark"} _format_version: "3.0" plugins: - name: opentelemetry config: traces_endpoint: https://ingestion..bronto.io/v1/traces logs_endpoint: https://ingestion..bronto.io/v1/logs access_logs: endpoint: https://ingestion..bronto.io/v1/logs metrics: endpoint: https://ingestion..bronto.io/v1/metrics enable_request_metrics: true enable_latency_metrics: true enable_bandwidth_metrics: true enable_upstream_health_metrics: true enable_consumer_attribute: true push_interval: 15 headers: x-bronto-api-key: ${{ env "DECK_BRONTO_API_KEY" }} resource_attributes: service.name: service.namespace: sampling_rate: 1.0 http_response_header_for_traceid: X-Trace-Id ``` Set `` to `eu` or `us`, export `DECK_BRONTO_API_KEY`, then apply it: ```bash theme={"dark"} deck gateway validate kong.yaml deck gateway sync kong.yaml ``` Things to know about this block: * **Each signal takes its full URL.** Kong does not append `/v1/...` to a base host. Runtime logs (`logs_endpoint`) and access logs (`access_logs.endpoint`) are separate streams, so both need setting. * **Use the nested `access_logs.endpoint`.** The flat `access_logs_endpoint` field is deprecated from 3.14. * **An endpoint alone exports no metrics.** Every `enable_*` flag defaults to `false`. Two more exist: `enable_ai_metrics` adds AI Gateway metrics, and `enable_principal_attribute` (3.16+) tags metrics with the authenticated principal. `push_interval` defaults to `60` seconds. * **`service.name` and `service.namespace`** select the log dataset and collection. Traces and metrics don't belong to a dataset; they appear in [Explore Traces](https://docs.bronto.io/tracing/explore-traces.md) and [Metric Explorer](https://docs.bronto.io/metrics/explore-metrics.md) under the same `service.name`. * **`http_response_header_for_traceid`** returns the trace ID to the client on every response, so a caller can find its own request in Bronto. * `${{ env "DECK_..." }}` is resolved by decK at sync time, which keeps the key out of `kong.yaml`. decK only exposes variables with the `DECK_` prefix. The resolved key is still stored in the gateway's configuration; to keep it out of that too, use a [vault reference](#store-the-api-key-in-a-vault) as the header value. ## Configure in the Konnect UI The same settings are available in Konnect without decK. Open **API Gateway → Control planes**, select your control plane, then **Plugins → New plugin** and choose **OpenTelemetry**. 1. **Plugin scope:** choose **Global** to cover every service and route. 2. **Plugin configuration:** fill in **Traces endpoint**, **Logs endpoint**, **Access logs → Endpoint** and **Metrics → Endpoint** with the four URLs above, then switch on the **Enable … metrics** toggles under **Metrics**. 3. Select **Show additional settings** under Plugin configuration. This reveals the fields the main form hides: * **Headers:** key `x-bronto-api-key`, with your API key — or a vault reference — as the value * **Resource attributes:** `service.name` and `service.namespace` * **HTTP response header for TraceID:** `X-Trace-Id` * **Sampling rate:** `1` 4. Select **View configuration** to check the result against the YAML above, then **Save**. The data plane environment variables in the [first step](#enable-tracing-and-runtime-logs-on-the-data-plane) are still set where the data plane runs; the plugin form does not replace them. ### Store the API key in a vault Every header value and endpoint field has a **Look up key in vault** link. It lists the vaults configured on the control plane, so create one first under **Vaults → New vault**: * **Konnect** keeps the secret in Konnect itself, with nothing to set up on the data plane. * **Environment variables** reads it from the data plane's environment, alongside the `KONG_*` variables above. Store the Bronto API key as the secret, then use **Look up key in vault** on the header value. The field becomes a reference of the form `{vault:///}`, resolved by the data plane at runtime. A reference must be the whole field value. Store the bare API key as the secret, not `x-bronto-api-key:` plus the key — the header name stays in the **Headers** key field. With decK, the equivalent is a `vaults` entry and a quoted reference as the header value: ```yaml kong.yaml theme={"dark"} vaults: - name: konnect prefix: bronto config: config_store_id: plugins: - name: opentelemetry config: headers: x-bronto-api-key: "{vault://bronto/bronto-api-key}" # ...rest of the configuration above ``` `deck gateway sync` deletes entities missing from the file, so keep the `vaults` entry in the same file as the plugin. ## What you will see in Bronto ### Traces Open [Tracing](https://app.bronto.io/tracing) and filter by your `service.name`. Each request is a trace rooted at a `kong` server span, with one child per phase and plugin, named `kong..plugin.` — for example `kong.access.plugin.key-auth`. `kong.router`, `kong.dns` and `kong.balancer` cover routing and the upstream connection. Only proxied requests reach `kong.dns` and `kong.balancer`, which makes those two spans a reliable "did this request reach the upstream?" marker. ### Access logs Filter Search by `$log.type = 'access'`. Each request is one log record with route, service, consumer, latency and status fields. Two are particularly useful: | Field | Meaning | | - | - | | `$kong.response.source` | `upstream` if the service produced the response, `kong` if Kong did (a plugin rejection, for example) | | `$kong.request.id` | Present on every access log and every span — the reliable join key between them | `$kong.response.source` separates a 401 from your upstream from a 401 from `key-auth`, which the status code alone cannot. ### Runtime logs Filter by `$log.type = 'error'`. These are Kong's own log lines, with `$introspection.source` naming the Lua file that wrote each one. Lines written during a request carry its trace and span IDs. ### Metrics With every family enabled, Kong emits 14 metrics, including `http.server.request.count`, `http.server.request.size`, `http.server.response.size`, `kong.latency.total`, `kong.latency.upstream` and `kong.latency.internal`, plus gateway health under `kong.nginx.*`, `kong.shared_dict.*` and `kong.memory.workers.lua_vm`. `http.server.request.count` carries `kong.response.source`, `http.response.status_code`, `kong.route.name` and `kong.service.name` as attributes, so rejections can be counted by route and status without querying logs. Kong exports metrics as **cumulative**. Bronto accepts cumulative and [recommends delta](https://docs.bronto.io/metrics/send-metrics.md#supported-metric-types); to convert, send `metrics.endpoint` to an OpenTelemetry Collector running the `cumulativetodelta` processor and forward from there to Bronto. ## AI Gateway On routes that use `ai-proxy`, Kong also records token usage, model, cost and LLM latency for every call. Turn on statistics in the plugin: ```yaml kong.yaml theme={"dark"} plugins: - name: ai-proxy route: config: route_type: llm/v1/chat logging: log_statistics: true log_payloads: false model: provider: openai name: gpt-4o-mini options: input_cost: 0.15 # per 1M tokens output_cost: 0.60 ``` `log_statistics: true` is what adds the AI fields to the access log — without it there is no token telemetry. `log_payloads: false` keeps prompt and completion text out of Bronto. `input_cost` and `output_cost` are your model's price per million tokens; until they are set, `$ai.proxy.usage.cost` reports `0`. The fields arrive on the same access log records: | Field | Use | | - | - | | `$ai.proxy.usage.total_tokens` | What token-based rate limits meter | | `$ai.proxy.usage.prompt_tokens`, `$ai.proxy.usage.completion_tokens` | Context size versus generation length | | `$ai.proxy.usage.cost` | Estimated spend per call | | `$ai.proxy.meta.llm_latency` | Time spent in the model provider | | `$ai.proxy.meta.request_model`, `$ai.proxy.meta.response_model` | Which model served the call | | `$gen_ai.usage.input_tokens`, `$gen_ai.usage.output_tokens` | OpenTelemetry GenAI semantic-convention equivalents | | `$kong.auth.consumer.username` | Which consumer made the call | Group by `$kong.auth.consumer.username` to see token use and spend per tenant. ## Troubleshooting * **No metrics?** Check the `metrics.enable_*` flags. With all of them `false` — the default — the metrics endpoint receives nothing. * **No spans?** `KONG_TRACING_INSTRUMENTATIONS` must be set on the data plane, not only the plugin. The plugin's `sampling_rate` and `KONG_TRACING_SAMPLING_RATE` both apply. * **Error-rate panels on spans read zero.** Kong leaves `$span.status_code` as `STATUS_CODE_UNSET` on rejected requests, including 401s and 429s. Build rejection panels on `$http.response.status_code` or `$kong.response.source` instead. * **`kong.access.plugin.opentelemetry` span missing on some requests.** Expected: the plugin has a low access-phase priority, so its own span is absent whenever an earlier plugin rejects the request. * **AI fields missing from access logs?** Set `logging.log_statistics: true` on `ai-proxy`. * **Need to see exactly what Kong sends?** Point the four endpoints at an OpenTelemetry Collector with the [`debug` exporter](https://github.com/open-telemetry/opentelemetry-collector/tree/main/exporter/debugexporter) at `verbosity: detailed`, then read the Collector's output for span names, attribute keys and metric temporality. *** For assistance or questions, contact [support@bronto.io](mailto:support@bronto.io). This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.