> ## Documentation Index
> Fetch the complete documentation index at: https://docs.bronto.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Keycloak traces and logs with OpenTelemetry

> Turn on Keycloak's built-in OpenTelemetry exporter to send login, token and JDBC traces — plus trace-correlated server logs — to Bronto over OTLP.

Keycloak has shipped an OpenTelemetry exporter since 26.0. Seven lines in `conf/keycloak.conf` turn it on and point it at Bronto over OTLP/HTTP — no Collector, no Java agent, nothing installed. Every login, token grant and JDBC statement becomes a span, and the server's own log stream can travel over the same connection, already correlated by trace ID.

## Prerequisites

* A Bronto account and API key ([how to create one](/Account-Management/API-Keys#create-a-new-api-key))
* Keycloak 26.0 or later (verified with 26.7.1, embedded Quarkus 3.33.2.1 and OpenTelemetry Java SDK 1.57.0)
* Tracing is gated behind the `opentelemetry` preview feature, which the configuration below enables

## Configure Keycloak

Add to `conf/keycloak.conf`:

```ini conf/keycloak.conf theme={"dark"}
features=opentelemetry,opentelemetry-logs
tracing-enabled=true
telemetry-logs-enabled=true
telemetry-protocol=http/protobuf
telemetry-endpoint=https://ingestion.<REGION>.bronto.io
telemetry-header-x-bronto-api-key=${BRONTO_API_KEY}
telemetry-service-name=<YOUR_DATASET_NAME>
```

Set `<REGION>` to `eu` or `us`, export `BRONTO_API_KEY` in the server's environment, and restart Keycloak. `features` is a build-time option, so `start` re-augments the server on the next boot.

`telemetry-endpoint` takes the **host only**. Keycloak appends `/v1/traces` and `/v1/logs` itself — pass a full signal URL and it will POST to `/v1/traces/v1/traces`.

`telemetry-protocol=http/protobuf` is not optional. The exporter defaults to gRPC, which Bronto's ingestion endpoint does not accept.

`${BRONTO_API_KEY}` is resolved by Keycloak at boot, so the key stays out of the config file and out of your image. The single `telemetry-header-*` line is inherited by the logs exporter as well as the tracer.

`telemetry-service-name` sets `service.name`, which selects the dataset. Keycloak is one process and emits everything itself, so this one line is the whole routing configuration.

Two signals means two datasets: traces arrive in the `.traces` collection and logs in `default`, both under the name you set here. Drop `opentelemetry-logs` from `features` and `telemetry-logs-enabled` if you only want traces.

### Option naming

On 26.7 the endpoint, protocol, service-name and resource-attribute options were generalised from the `tracing-*` family to `telemetry-*`, which is shared with logs. `tracing-service-name` and `tracing-resource-attributes` are marked deprecated in `--help-all`. The older `tracing-*` spellings still work, but `tracing-enabled` was **not** renamed — there is no `telemetry-enabled`, which is why the block above mixes the two families.

### Environment variables instead

Every option has a `KC_*` twin, which suits a Kubernetes Deployment or a Compose file:

```bash theme={"dark"}
KC_FEATURES=opentelemetry,opentelemetry-logs
KC_TRACING_ENABLED=true
KC_TELEMETRY_LOGS_ENABLED=true
KC_TELEMETRY_PROTOCOL=http/protobuf
KC_TELEMETRY_ENDPOINT=https://ingestion.<REGION>.bronto.io
KC_TELEMETRY_HEADER_X_BRONTO_API_KEY=<YOUR_API_KEY>
KC_TELEMETRY_SERVICE_NAME=<YOUR_DATASET_NAME>
```

## Metrics

Keycloak also exposes Prometheus metrics, on its management port and separately from the OTLP exporter above. Add to `conf/keycloak.conf`:

```ini conf/keycloak.conf theme={"dark"}
metrics-enabled=true
event-metrics-user-enabled=true
```

and add `user-event-metrics` to the `features` line:

```ini conf/keycloak.conf theme={"dark"}
features=opentelemetry,opentelemetry-logs,user-event-metrics
```

`metrics-enabled=true` on its own gives JVM, HTTP, Infinispan cache and JDBC pool metrics — 260 families. The `user-event-metrics` feature adds the counters specific to an identity provider:

| Metric                                                                                                              | Labels                                              | What it answers                                                                    |
| ------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------- | ---------------------------------------------------------------------------------- |
| `keycloak_user_events_total`                                                                                        | `realm`, `event`, `error`                           | Logins, client logins and token refreshes per realm, and the reason when one fails |
| `keycloak_credentials_password_hashing_validations_total`                                                           | `realm`, `algorithm`, `hashing_strength`, `outcome` | Which hash algorithm and cost factor each realm is really running                  |
| `agroal_active_count`, `agroal_available_count`, `agroal_awaiting_count`, `agroal_blocking_time_total_milliseconds` | `datasource`                                        | JDBC pool depth, and time spent waiting for a connection                           |
| `http_server_requests_seconds`                                                                                      | `method`, `uri`, `status`, `outcome`                | Request latency histogram, with `uri` templated the same way span names are        |

Scrape the endpoint with the OpenTelemetry Collector:

```yaml theme={"dark"}
receivers:
  prometheus:
    config:
      scrape_configs:
        - job_name: keycloak
          scrape_interval: 15s
          scheme: https
          static_configs:
            - targets: [<KEYCLOAK_HOST>:9000]
```

Add this receiver to the [Bronto metrics pipeline](/metrics/send-metrics). Prometheus counters are cumulative, so include the `cumulativetodelta` processor — Bronto accepts both, and recommends delta.

The management interface inherits the server's TLS configuration: once `https-certificate-file` is set, port `9000` serves HTTPS, and the scrape config needs `scheme: https` as above.

### Exporting metrics over OTLP instead

Keycloak can also export metrics over OTLP directly, reusing the endpoint and header already configured for traces — add `opentelemetry-metrics` to `features`, and set `telemetry-metrics-enabled=true` alongside `metrics-enabled=true`. No Collector is involved.

Two things to weigh against the scrape above. The feature is Experimental rather than Preview, and Keycloak escalates its startup banner accordingly. And the series arrive cumulative, under OpenTelemetry's dotted names (`keycloak.credentials.password.hashing`) rather than the Prometheus spellings in the table above, so the two routes do not produce interchangeable metric names.

## What you will see in Bronto

Open [Tracing](https://app.bronto.io/tracing) and filter by your service name.

A browser login is one trace of about 28 spans, rooted at `POST /realms/{realm}/login-actions/authenticate`. Token requests root at `TokenEndpoint.processGrantRequest`. JDBC is instrumented by default, so every statement appears with `$db.statement`, `$db.sql.table` and `$db.operation` attached.

Span names are templated, so the realm is not in the name — `$url.path` carries it. Grouping login spans by `$url.path` shows latency per realm, and grouping token spans by grant type shows what each kind of token costs.

Grant type, realm, client and outcome arrive as a **span event** rather than span attributes: `$span.events.0.name` is `LOGIN`, `REFRESH_TOKEN`, `CODE_TO_TOKEN` or `LOGIN_ERROR`, with `$span.events.0.kc.details.grant_type`, `.kc.realmName` and `.kc.clientId` alongside it. Grouping by an event key needs it quoted inside the query string — `"groups": ["\"$span.events.0.kc.details.grant_type\""]` — though the same key needs no quoting in a filter.

A failed login is not a span error. It is an HTTP 200 re-render or a 400, and its status is `STATUS_CODE_UNSET` just like a success, so filter on `$http.response.status_code` or on the `LOGIN_ERROR` span event rather than on span status.

## Troubleshooting

* **No traces, and no error in the log?** `telemetry-endpoint` must be the host only. A full `/v1/traces` URL becomes `/v1/traces/v1/traces`, which 404s, and the exporter treats the 404 as a successful delivery. Logins keep working and nothing is logged.
* **`First received frame was not SETTINGS` in the Keycloak log?** `telemetry-protocol` is unset, so the exporter is sending gRPC. Set it to `http/protobuf`. Note this is a `WARNING` on `io.quarkus`, so a production log level of `ERROR` hides it.
* **`Installed features: [… opentelemetry …]` at startup does not mean tracing is on.** Quarkus lists its extension either way. The reliable local check is the log line format: with tracing enabled every line gains a `traceId=, parentId=, spanId=, sampled=` prefix.
* **Everything disappeared after tuning the sampler.** `tracing-sampler-ratio=0.0` drops 100% of spans silently. So does `tracing-sampler-type=parentbased_traceidratio` when anything upstream — a proxy sampling at 10%, a frontend SDK — marks the request unsampled, because Keycloak honours incoming W3C trace context. The default `traceidratio` at ratio `1.0` ignores the upstream flag and is the safe setting.
* **Span volume higher than expected?** One browser login is 48 server spans, 35 of which are login-theme CSS, fonts and images. There is no per-endpoint filter; `tracing-jdbc-enabled=false` will not help, since JDBC is only 9 of the 48.
* **No span naming the hashing cost?** Only the Argon2 provider is instrumented. Under PBKDF2 you see `PasswordCredentialProvider.isValid` with an unexplained gap and no children.

***

For assistance or questions, contact [support@bronto.io](mailto:support@bronto.io).
