ADR-0014: Application meters are scraped; auto-instrumentation is pushed

ADR-0011 established OpenTelemetry as the instrumentation boundary: services export OTLP to one collector, and the collector decides destinations. That rule was written about where telemetry goes. It did not settle how a metric registered by application code gets there, and the omission produced a silent gap.

Signal & Sweep leader election was instrumented with six Micrometer meters registered on the injected Spring MeterRegistry. None of them reached Prometheus, and nothing reported an error. The dashboards and alerts that were supposed to consume them could not be built.

The cause is specific and worth recording, because it looks like configuration but is not. MetricsConfiguration.otelRegistry() searches Metrics.globalRegistry for an OpenTelemetryMeterRegistry, removes it, and republishes it as a Spring bean — or returns null when it is not there. That registry only exists when the OpenTelemetry Java agent’s Micrometer bridge is running, and that bridge is off by default. The flag otel.instrumentation.micrometer.enabled is set nowhere in the estate. So the bean resolves to null, no OTel registry joins the Spring composite, and every application meter goes only to the Prometheus registry — which nothing was scraping.

Confirmed empirically on the production cluster: no import_*, executor_* or spring_security_* series exist in Prometheus, though all of them are present in the service’s own /management/prometheus output. Every application metric that does reach Prometheus carries a semantic-convention name — jvm_memory_used, http_server_request_duration, db_client_connections_* — which is agent auto-instrumentation, not Micrometer.

Two routes could close the gap, and the forces differ sharply.

Enabling the agent’s Micrometer bridge is a single flag and keeps one transport. But OTLP metric series carry only job and instance, and instance is a UUID rather than a pod name; pod identity and environment live on target_info and must be recovered with a group_left join in every panel and every alert. It also exports the entire Micrometer surface, so jvm_memory_used and jvm_memory_used_bytes would both land — two differently-named measurements of the same thing in one Prometheus. Suppressing that means adding a filter processor to a collector pipeline that also carries traces, logs and span metrics for five services.

Scraping the actuator endpoint needs no application change — /management/prometheus is already served and already permitAll() — and yields pod and namespace as native target labels, plus up as free scrape health. Against that, the endpoint exposes 16,237 series per pod against a Prometheus holding roughly 256,000, so it cannot be ingested unfiltered.

The deciding constraint came from the instrumentation itself. Lease metrics deliberately carry no pod identifier — holderId appears only in logs and on /management/health — so the question "which pod holds this role", which the alerting exists to answer, can only be answered from scrape target labels.

Metrics registered by application code on the Spring MeterRegistry reach Prometheus by scrape, from /management/prometheus via a ServiceMonitor, under Micrometer names.

Auto-instrumentation telemetry — JVM, HTTP, database, traces and logs — reaches the backend by OTLP push, under OpenTelemetry semantic-convention names, exactly as ADR-0011 requires.

Every ServiceMonitor added under this rule MUST carry a keep-only metricRelabelings block naming the metric families it exists to collect. Scraping an actuator endpoint without one admits the service’s entire Micrometer surface.

Do not enable the OpenTelemetry agent’s Micrometer bridge. It is the mechanism that would collapse these two routes into one, and doing so reintroduces the duplicate naming this ADR exists to prevent.