Keycloak traces and logs with OpenTelemetry

Keycloak has shipped an OpenTelemetry exporter since 26.0. Seven lines in conf/keycloak.conf turn it on and point it at Bronto over OTLP/HTTP — no Collector, no Java agent, nothing installed. Every login, token grant and JDBC statement becomes a span, and the server’s own log stream can travel over the same connection, already correlated by trace ID.

Prerequisites

A Bronto account and API key (how to create one)
Keycloak 26.0 or later (verified with 26.7.1, embedded Quarkus 3.33.2.1 and OpenTelemetry Java SDK 1.57.0)
Tracing is gated behind the opentelemetry preview feature, which the configuration below enables

Configure Keycloak

Add to conf/keycloak.conf:

conf/keycloak.conf

features=opentelemetry,opentelemetry-logs
tracing-enabled=true
telemetry-logs-enabled=true
telemetry-protocol=http/protobuf
telemetry-endpoint=https://ingestion.<REGION>.bronto.io
telemetry-header-x-bronto-api-key=${BRONTO_API_KEY}
telemetry-service-name=<YOUR_DATASET_NAME>

Set <REGION> to eu or us, export BRONTO_API_KEY in the server’s environment, and restart Keycloak. features is a build-time option, so start re-augments the server on the next boot. telemetry-endpoint takes the host only. Keycloak appends /v1/traces and /v1/logs itself — pass a full signal URL and it will POST to /v1/traces/v1/traces. telemetry-protocol=http/protobuf is not optional. The exporter defaults to gRPC, which Bronto’s ingestion endpoint does not accept. ${BRONTO_API_KEY} is resolved by Keycloak at boot, so the key stays out of the config file and out of your image. The single telemetry-header-* line is inherited by the logs exporter as well as the tracer. telemetry-service-name sets service.name, which selects the dataset. Keycloak is one process and emits everything itself, so this one line is the whole routing configuration. Two signals means two datasets: traces arrive in the .traces collection and logs in default, both under the name you set here. Drop opentelemetry-logs from features and telemetry-logs-enabled if you only want traces.

Option naming

On 26.7 the endpoint, protocol, service-name and resource-attribute options were generalised from the tracing-* family to telemetry-*, which is shared with logs. tracing-service-name and tracing-resource-attributes are marked deprecated in --help-all. The older tracing-* spellings still work, but tracing-enabled was not renamed — there is no telemetry-enabled, which is why the block above mixes the two families.

Environment variables instead

Every option has a KC_* twin, which suits a Kubernetes Deployment or a Compose file:

KC_FEATURES=opentelemetry,opentelemetry-logs
KC_TRACING_ENABLED=true
KC_TELEMETRY_LOGS_ENABLED=true
KC_TELEMETRY_PROTOCOL=http/protobuf
KC_TELEMETRY_ENDPOINT=https://ingestion.<REGION>.bronto.io
KC_TELEMETRY_HEADER_X_BRONTO_API_KEY=<YOUR_API_KEY>
KC_TELEMETRY_SERVICE_NAME=<YOUR_DATASET_NAME>

Metrics

Keycloak also exposes Prometheus metrics, on its management port and separately from the OTLP exporter above. Add to conf/keycloak.conf:

conf/keycloak.conf

metrics-enabled=true
event-metrics-user-enabled=true

and add user-event-metrics to the features line:

conf/keycloak.conf

features=opentelemetry,opentelemetry-logs,user-event-metrics

metrics-enabled=true on its own gives JVM, HTTP, Infinispan cache and JDBC pool metrics — 260 families. The user-event-metrics feature adds the counters specific to an identity provider:

Metric	Labels	What it answers
`keycloak_user_events_total`	`realm`, `event`, `error`	Logins, client logins and token refreshes per realm, and the reason when one fails
`keycloak_credentials_password_hashing_validations_total`	`realm`, `algorithm`, `hashing_strength`, `outcome`	Which hash algorithm and cost factor each realm is really running
`agroal_active_count`, `agroal_available_count`, `agroal_awaiting_count`, `agroal_blocking_time_total_milliseconds`	`datasource`	JDBC pool depth, and time spent waiting for a connection
`http_server_requests_seconds`	`method`, `uri`, `status`, `outcome`	Request latency histogram, with `uri` templated the same way span names are

Scrape the endpoint with the OpenTelemetry Collector:

receivers:
  prometheus:
    config:
      scrape_configs:
        - job_name: keycloak
          scrape_interval: 15s
          scheme: https
          static_configs:
            - targets: [<KEYCLOAK_HOST>:9000]

Add this receiver to the Bronto metrics pipeline. Prometheus counters are cumulative, so include the cumulativetodelta processor — Bronto accepts both, and recommends delta. The management interface inherits the server’s TLS configuration: once https-certificate-file is set, port 9000 serves HTTPS, and the scrape config needs scheme: https as above.

Exporting metrics over OTLP instead

Keycloak can also export metrics over OTLP directly, reusing the endpoint and header already configured for traces — add opentelemetry-metrics to features, and set telemetry-metrics-enabled=true alongside metrics-enabled=true. No Collector is involved. Two things to weigh against the scrape above. The feature is Experimental rather than Preview, and Keycloak escalates its startup banner accordingly. And the series arrive cumulative, under OpenTelemetry’s dotted names (keycloak.credentials.password.hashing) rather than the Prometheus spellings in the table above, so the two routes do not produce interchangeable metric names.

What you will see in Bronto

Open Tracing and filter by your service name. A browser login is one trace of about 28 spans, rooted at POST /realms/{realm}/login-actions/authenticate. Token requests root at TokenEndpoint.processGrantRequest. JDBC is instrumented by default, so every statement appears with $db.statement, $db.sql.table and $db.operation attached. Span names are templated, so the realm is not in the name — $url.path carries it. Grouping login spans by $url.path shows latency per realm, and grouping token spans by grant type shows what each kind of token costs. Grant type, realm, client and outcome arrive as a span event rather than span attributes: $span.events.0.name is LOGIN, REFRESH_TOKEN, CODE_TO_TOKEN or LOGIN_ERROR, with $span.events.0.kc.details.grant_type, .kc.realmName and .kc.clientId alongside it. Grouping by an event key needs it quoted inside the query string — "groups": ["\"$span.events.0.kc.details.grant_type\""] — though the same key needs no quoting in a filter. A failed login is not a span error. It is an HTTP 200 re-render or a 400, and its status is STATUS_CODE_UNSET just like a success, so filter on $http.response.status_code or on the LOGIN_ERROR span event rather than on span status.

Troubleshooting

No traces, and no error in the log? telemetry-endpoint must be the host only. A full /v1/traces URL becomes /v1/traces/v1/traces, which 404s, and the exporter treats the 404 as a successful delivery. Logins keep working and nothing is logged.
First received frame was not SETTINGS in the Keycloak log? telemetry-protocol is unset, so the exporter is sending gRPC. Set it to http/protobuf. Note this is a WARNING on io.quarkus, so a production log level of ERROR hides it.
Installed features: [… opentelemetry …] at startup does not mean tracing is on. Quarkus lists its extension either way. The reliable local check is the log line format: with tracing enabled every line gains a traceId=, parentId=, spanId=, sampled= prefix.
Everything disappeared after tuning the sampler. tracing-sampler-ratio=0.0 drops 100% of spans silently. So does tracing-sampler-type=parentbased_traceidratio when anything upstream — a proxy sampling at 10%, a frontend SDK — marks the request unsampled, because Keycloak honours incoming W3C trace context. The default traceidratio at ratio 1.0 ignores the upstream flag and is the safe setting.
Span volume higher than expected? One browser login is 48 server spans, 35 of which are login-theme CSS, fonts and images. There is no per-endpoint filter; tracing-jdbc-enabled=false will not help, since JDBC is only 9 of the 48.
No span naming the hashing cost? Only the Argon2 provider is instrumented. Under PBKDF2 you see PasswordCredentialProvider.isValid with an unexplained gap and no children.

For assistance or questions, contact support@bronto.io.

Overview

Ingest with agents

Ingest from AWS

Ingest from Azure

Ingest from CDN and edge

Ingest from apps and platforms

Ingest from infrastructure and databases

Ingest OpenTelemetry data

Keycloak traces and logs with OpenTelemetry

Prerequisites

Configure Keycloak

Option naming

Environment variables instead

Metrics

Exporting metrics over OTLP instead

What you will see in Bronto

Troubleshooting

​Prerequisites

​Configure Keycloak

​Option naming

​Environment variables instead

​Metrics

​Exporting metrics over OTLP instead

​What you will see in Bronto

​Troubleshooting

Prerequisites

Configure Keycloak

Option naming

Environment variables instead

Metrics

Exporting metrics over OTLP instead

What you will see in Bronto

Troubleshooting