OpenTelemetry Configuration

1. Overview

EMS services emit traces, metrics, and (via Logback appender) correlated logs to a central OpenTelemetry Collector running in the cluster’s observability namespace. The collector forwards to the in-cluster Grafana stack: Tempo for traces and Loki for logs, queried through Grafana in the same namespace. For how to query them — sign-in, datasources, LogQL, dashboards and alerts — see Observability Access.

Pod logs reach Loki by a second, independent path: a Grafana Alloy DaemonSet tails /var/log/pods on each node and writes to Loki directly, without passing through the collector. This is worth knowing because it collects stdout and stderr only — a container that writes its real log to a file inside its own filesystem is invisible to it. See Dependency Update Checks for how that bit us on the SSH bastion.

The previous ELK/Kibana backend is deprecated. Loki retains for 720h (30 days). Any document still describing Elasticsearch or Kibana as the logging destination is out of date.

Two instrumentation paths:

  1. Java agent (opentelemetry-javaagent) — auto-instruments HTTP servers, JDBC, Hazelcast, Kafka, etc. at bytecode level. Baked into the service’s Docker image via Jib’s extraDirectories.

  2. SDK / starter (opentelemetry-spring-boot-starter) — enables manual @WithSpan annotations, @Counted, @Timed method-level instrumentation, and custom span creation via Tracer.

Both are active simultaneously in admin-service. Spring Web auto-instrumentation is disabled in admin-service’s application-otlp.yml — the agent would double-count if Spring Web also reported. Manual spans and JDBC spans cover the rest.

See also: SpringApplication Bootstrap, Hazelcast Configuration.

2. Topology

otel-topology

Collector endpoint in-cluster: http://opentelemetry-collector.observability.svc.cluster.local:4318. The collector also listens on :4317 for gRPC, but EMS services must use 4318 — see OTLP transport below. Deployed via ~/dev/idl-xnl-jhb-rc01/argocd/opentelemetry-collector.yml.

3. Admin-Service Configuration

Full detail — this is the reference implementation.

3.1. Profile-gated

Active only when otlp profile is enabled. ArgoCD prod manifest sets config.profiles: "prod,kubernetes,otlp" (or similar — see ArgoCD Deployment Patterns).

3.2. POM configuration

The agent is always baked into the image (regardless of which Spring profile is active); the otlp profile only adds the Spring-side dependencies that activate application-otlp.yml.

In the main <build> section (always runs):

  • Maven plugin: maven-dependency-plugin copy execution, bound to the package phase. Downloads io.opentelemetry.javaagent:opentelemetry-javaagent:<version> into ${agent-extraction-root} (= ${project.build.directory}/jib-agents) as ${opentelemetry-javaagent-filename} (= opentelemetry-javaagent.jar).

  • Jib <extraDirectories> then copies that directory into the image at ${agent-install-location} (= /javaagent). The image always has /javaagent/opentelemetry-javaagent.jar regardless of profile.

  • Jib <jvmFlags> always include -javaagent:/javaagent/opentelemetry-javaagent.jar plus -Dotel.{logs,traces,metrics}.exporter=otlp.

  • Jib <environment> sets OTEL_SERVICE_NAME=${project.artifactId} so traces are tagged with the service name.

Under the otlp profile (Spring-side only):

  • BOM: io.opentelemetry.instrumentation:opentelemetry-instrumentation-bom:<version>

  • Dependency: io.opentelemetry.instrumentation:opentelemetry-spring-boot-starter

  • Dependency: io.opentelemetry.instrumentation:opentelemetry-instrumentation-annotations

  • Dependency: io.opentelemetry.instrumentation:opentelemetry-logback-appender-1.0:<version>-alpha

The agent operates at bytecode level and works without any Spring-side dependency. The Spring starter adds @WithSpan annotation support, the logback appender for trace correlation, and SDK-level autoconfiguration. With the otlp profile inactive, the agent is still loaded but it falls back to its own auto-instrumentation only — no Spring @WithSpan, no logback trace correlation.

Anti-pattern (do not copy from older internal scaffolds): downloading the agent into src/main/jib/opt/otel/ with Jib <extraDirectories> pointing at a different path. The path mismatch silently drops the agent from the image; the JVM emits -javaagent: file not found on startup and OTel emits nothing. Source-tree pollution is also wrong — agent jars belong in target/.

See Jib Docker Build § OTel javaagent for the full downloader + Jib XML.

3.3. Runtime config (application-otlp.yml)

spring:
  jpa:
    properties:
      hibernate.generate_statistics: true            # admin-service only — drop for non-JPA services

management:
  metrics:
    export:
      otlp:
        enabled: true

otel:
  java:
    global-autoconfigure:
      enabled: true
  exporter:
    otlp:
      endpoint: 'http://opentelemetry-collector.observability.svc.cluster.local:4318'
    jaeger:
      enabled: false
    zipkin:
      enabled: false
  springboot:
    resource:
      enable: true
  resource:
    attributes:
      'service.version': '${OTEL_SERVICE_VERSION:unknown}'
      'deployment.environment': '${OTEL_DEPLOYMENT_ENVIRONMENT:production}'
  instrumentation:
    annotations:
      enabled: true
    # logback-appender.enabled is deliberately NOT set — see Two-stream logging below
    spring-web.enabled: false
    spring-webmvc.enabled: false
    spring-webflux.enabled: false

Key decisions:

  • global-autoconfigure: true — picks up Spring Boot auto-config

  • Jaeger and Zipkin exporters disabled — we only export OTLP to the collector

  • Resource attributes here apply to the Spring starter’s SDK only — they do not reach log records. See Resource attributes must go in OTEL_RESOURCE_ATTRIBUTES` below. Never hard-code `deployment.environment: a literal production labels dev and stage telemetry as production, and does so precisely when the agent has failed to attach and the fallback is in use

  • Spring Web/WebMVC/WebFlux auto-instrumentation disabled — the javaagent already instruments these at bytecode level; keeping the Spring-SDK version enabled produces duplicate spans

  • logback-appender.enabled deliberately unset — the Jib flag -Dotel.instrumentation.logback-appender.enabled=false disables the agent’s duplicate capture, and Spring resolves system properties ahead of this file. The appender is installed from OpenTelemetryLoggingConfiguration; see Two-stream logging

3.4. Resource attributes must go in OTEL_RESOURCE_ATTRIBUTES

deployment.environment, service.version and any other resource attribute that must appear on log records belongs in the OTEL_RESOURCE_ATTRIBUTES environment variable, set by the Helm chart. Declaring it under otel.resource.attributes in application-otlp.yml is not equivalent and will not work.

The reason is a direct consequence of the two-stream standard. OpenTelemetryLoggingConfiguration installs the Logback appender with GlobalOpenTelemetry.get(), which is the agent’s SDK. The agent builds its resource from OTEL_RESOURCE_ATTRIBUTES and OTEL_SERVICE_NAME. Attributes declared as Spring properties belong to the starter’s SDK — a different object — and never reach a log record.

This regressed silently in both services that adopted the standard first:

Service Version deployment_environment on log records

admin-portal

0.1.3

Present — logs still went through the starter’s SDK

admin-portal

0.1.4, 0.1.5

Absent — the install component moved log export to the agent’s SDK

ems-mcp-server

dev, pre-0.1.2

Present — the chart set OTEL_RESOURCE_ATTRIBUTES

ems-mcp-server

stage 0.1.2

Absent — the variable was removed while dropping service.namespace

It matters more once OTEL_SERVICE_NAME is environment-independent, which this standard requires: with no deployment.environment, dev, stage and prod records carry nothing that distinguishes them.

Do not set service.namespace. The collector’s Loki exporter composes the service_name label as <service.namespace>/<service.name> whenever the namespace attribute is present. Setting it puts an environment prefix back into the service name — ems-mcp-server became event-dev/dev-ems-mcp-server, which is why a query on the bare service name returned nothing at all and the service appeared to export no logs.

Keep the Spring-side declaration as well, sourced from the same environment variables. It governs metrics, and it is the resource in use if the agent ever fails to attach — the failure this standard exists to make visible.

# Helm chart — read by the agent, reaches log records
- name: OTEL_RESOURCE_ATTRIBUTES
  value: "deployment.environment={{ include "chart.otelDeploymentEnvironment" . }}"
- name: OTEL_DEPLOYMENT_ENVIRONMENT      # read by application-otlp.yml, Spring-side SDK
  value: {{ include "chart.otelDeploymentEnvironment" . | quote }}

Verify with a Loki query rather than by reading the chart — the failure is silent:

{service_name="<name>", exporter="OTLP"}    # stream labels must include deployment_environment

3.5. Javaagent at runtime

Jib bakes the javaagent at /javaagent/opentelemetry-javaagent.jar (the path is parameterised by the Maven properties agent-install-location and opentelemetry-javaagent-filename so it stays consistent across services). The container’s jvmFlags include:

<jvmFlag>-javaagent:${agent-install-location}/${opentelemetry-javaagent-filename}</jvmFlag>
<jvmFlag>-Dotel.logs.exporter=otlp</jvmFlag>
<jvmFlag>-Dotel.traces.exporter=otlp</jvmFlag>
<jvmFlag>-Dotel.metrics.exporter=otlp</jvmFlag>

Plus environment:

<environment>
    <OTEL_SERVICE_NAME>${project.artifactId}</OTEL_SERVICE_NAME>
</environment>

3.6. Exporter selection and OTEL_LOGS_EXPORTER

The exporter selectors (-Dotel.{logs,traces,metrics}.exporter=otlp) are set explicitly, and the Helm charts additionally set OTEL_LOGS_EXPORTER, OTEL_TRACES_EXPORTER and OTEL_METRICS_EXPORTER.

Neither is strictly required. Measured on agent 2.10.0: otlp is already the default for every signal — an agent with no exporter setting at all still constructs an OTLP log exporter and attempts to export. They are set anyway because the default is invisible. An operator reading a Helm chart cannot tell from silence whether logs ship, and that ambiguity is exactly how four of six services drifted into three different broken states without anyone noticing.

OTEL_LOGS_EXPORTER being absent is not by itself an explanation for a service producing no logs. That inference was drawn during the 2026-08-24 health check and later disproved. If a service is not logging, check in this order:

  1. Is -javaagent actually on /proc/1/cmdline? Environment variables prove nothing — two services carried a full set of OTEL_* variables with no agent attached.

  2. Is the endpoint on port 4318? See below.

  3. Does the service log anything at all? An idle service is not a broken one.

3.7. OTLP transport: HTTP/protobuf on port 4318

The EMS standard is OTLP over HTTP/protobuf on port 4318. Every service points at:

http://opentelemetry-collector.observability.svc.cluster.local:4318

Port 4317 is OTLP gRPC. It requires OTEL_EXPORTER_OTLP_PROTOCOL=grpc in addition to the endpoint; without it the export fails with Connection reset and nothing ships at all — no traces, no logs, no metrics. That failure is silent from the application’s point of view and easy to misread as a missing exporter setting.

4318 is preferred over 4317 for three reasons:

  1. It is the javaagent’s own default protocol in 2.x, so there is no second setting to remember or forget.

  2. It is debuggable with ordinary HTTP tooling and legible through any proxy or ingress.

  3. gRPC’s advantages — multiplexing, compression over long links — do not apply to an in-cluster collector at EMS volumes.

Revisit if the collector moves off-cluster or telemetry volume grows by an order of magnitude.

ems-mcp-server was left on 4317 in stage and prod after dev was corrected, and consequently exported nothing from either environment. Corrected 2026-08-23. No service should set OTEL_EXPORTER_OTLP_PROTOCOL; if you find yourself needing it, the endpoint is wrong.

Customisation (sampling rate, instrumentation toggles, custom resource attributes) is done via env vars (OTEL_TRACES_SAMPLER=parentbased_traceidratio, OTEL_TRACES_SAMPLER_ARG=0.1, etc.) injected by the Helm chart from the ArgoCD valuesObject — not via an agent.properties file. The properties-file approach is supported by the agent but adds an extra config artefact for no benefit.

4. Per-service state

Measured 2026-08-23 against /proc/1/cmdline on each pod and Loki’s /series API. This supersedes the 2026-08-24 health-check table, which was wrong in two places.

Service Agent attached OTLP logs Notes

event-admin-service

yes

yes

Reference implementation of the two-stream standard (ADO-1009)

admin-portal

yes

yes

Was believed broken; in fact exporting correctly. Adopted the standard in ADO-1012

ems-mcp-server

dev only

no

stage and prod ran 0.1.1, which has no javaagent at all. Also pointed at 4317. Both corrected 2026-08-23 (ADO-1013)

registration-portal

no

no

Carries OTEL_* environment variables with nothing to consume them. ADO-1010

membership-ui

no

no

Same shape as registration-portal. ADO-1011

admin-ui

n/a

n/a

Retired 2026-08-23 (ADO-1014). Deployment and ArgoCD application removed

registration-portal and membership-ui have the environment variables but no agent, so they emit nothing over OTLP. Their console output still reaches Loki through the Alloy path described in the Overview.

4.1. Adding OTel to a service

Clone admin-service’s otlp profile config:

  • Same POM dependencies

  • Same application-otlp.yml — except service.version adjusted per-service

  • Same Jib extraDirectory for the javaagent

  • Same jvmFlag -javaagent:${agent-install-location}/${opentelemetry-javaagent-filename}, resolving to /javaagent/opentelemetry-javaagent.jar

An earlier revision of this page told you to write -javaagent:/opt/otel/javaagent.jar here, contradicting the correct path given under Javaagent at runtime above. ems-mcp-server followed it, and its 0.1.1 image shipped a startup check for a file that is never created — printing WARNING: otlp profile active but /opt/otel/javaagent.jar missing while running with no instrumentation at all, in stage and prod, for months.

Always use the Maven properties rather than a literal path. A hard-coded agent path that does not match the Jib extraDirectory fails silently: the JVM starts, the application serves traffic, and no telemetry is produced.

Gateway-specific spans worth adding manually:

  • POST /api/session/tenant — wrap the token-exchange call in a span with attributes user.sub, tenant.requested, tenant.current-before

  • AdminServiceJwtRelayFilter — wrap the proxy call in a span with admin-service.endpoint attribute

  • TenantResolutionFilter — a short span identifying the resolution source (domain / header / session)

These make debugging multi-tenant auth issues tractable in trace view.

5. Metrics

Micrometer + OTel bridge emit the standard JVM + HTTP + Hazelcast metrics. Additional EMS-custom metrics live in admin-service/src/main/java/…​/config/MetricsConfiguration.java — OtlpMetricsNamingConvention keeps names dot-separated (OTel style) rather than underscore-separated (Prometheus style).

Selected metrics:

  • http.server.requests — rate, p95/p99 latency, status-code distribution per URI (automatic)

  • jdbc.connections.active / jdbc.connections.max — HikariCP pool state

  • hazelcast.partition.is-migrating — cluster rebalance indicator

  • jvm.memory.used / jvm.gc.pause — standard JVM

  • Custom: ems.import.duration / ems.import.rows — import-specific timers, see ImportAsyncConfiguration

Dashboards live in the observability backend; owner: Solution Architect / Ops.

6. Tracing Patterns

6.1. Business-flow spans

Group multiple API calls that belong to the same user journey under a "business flow" span. See design-journal/2026-03/end-to-end-distributed-tracing.adoc for the design. Pattern:

@WithSpan("membership-registration")
public void registerMembership(...) {
    // child spans from auto-instrumented Spring Web + JDBC roll up under this
}

Useful for showing "registration took 3.4s" with breakdown across the participant, payment, and email sub-operations.

6.2. W3C traceparent propagation

Frontend → gateway → admin-service all propagate traceparent header. registration-portal’s interceptor (when OTel lands for it) should extract any existing trace from the browser’s performance-navigation entries and attach; otherwise generate a new root.

Cross-cluster propagation (e.g. admin-service → WordPress → RunSignup) honours the same convention where supported.

7. Two-stream logging

Services adopting the standard (ADO-1008) split logging by audience:

Stream Audience Content

Console (stdout)

A Kubernetes operator with no domain knowledge

Everything at WARN+, plus INFO from an allowlist of infrastructure loggers — lifecycle, port binding, connection pool, schema migration, readiness

OTLP

A developer debugging behaviour

Everything, with trace context and MDC as queryable attributes

Console keeps WARN+ permanently. OTLP records are lost outright when the collector is unreachable, and a service that dies before the appender is installed never flushes its buffer — console is the fallback of last resort in exactly the cases that matter most.

The classification test for a reviewer: can a Kubernetes operator with no domain knowledge act on this line? If yes, it belongs on console.

Traces and metrics come from the agent; logs come from an explicitly declared OpenTelemetryAppender, because agent log capture happens at Logger.callAppenders, before appender routing, and therefore cannot be shaped by Logback filters or per-logger `appender-ref`s.

With both the agent’s Logback instrumentation and an explicit appender active, every statement is exported twice. The agent’s capture must be disabled with -Dotel.instrumentation.logback-appender.enabled=false.

That same property name also gates the OpenTelemetry Spring Boot starter’s OpenTelemetryAppender.install() call, which Spring resolves from system properties — so the flag disables both, leaving an appender that never receives an SDK. It then buffers 1000 records and discards everything after that, silently. Services therefore install the appender from application code (OpenTelemetryLoggingConfiguration) rather than relying on the starter.

Do not set scan="true" in logback-spring.xml on a service with an OTLP appender: reconfiguration rebuilds the appenders without an SDK and produces the same silent loss.

7.1. Conformance checklist

A logback-spring.xml adopting this standard must have all of the following. Each line is here because its absence has cost time in this estate.

Requirement Why

Explicit CONSOLE appender with a ThresholdFilter

Threshold is WARN under the otlp profile, TRACE otherwise — without the profile split, local development runs near-silent

Explicit CONSOLE_INFRA appender with a LevelFilter accepting INFO only

onMismatch=DENY is what stops an allowlisted WARN printing twice, once via CONSOLE_INFRA and once via the root logger’s CONSOLE

OTLP appender with captureMdcAttributes set to *

Business context must arrive as queryable attributes, not baked into the message string

Root logger referencing both CONSOLE and OTLP

An appender that is configured but never referenced is silently inert

Per-logger allowlist entries referencing CONSOLE_INFRA, additivity left on

Additivity is what keeps allowlisted records flowing to OTLP as well as console

No scan attribute

See above — a rescan produces SDK-less appenders

No file appender

Container logs are collected from stdout; a file appender writes to a volume nothing reads

<springProfile> never nested inside <root>, <logger> or <appender>

Spring Boot rejects the nested form outright — SpringProfileIfNestedWithinSecondPhaseElementSanityChecker emits "`<springProfile>` elements cannot be nested within an <appender>, <logger> or <root> element". Invert the nesting: declare <root> once per profile inside <springProfile>

Every appender declared in the same profile scope that references it

A declared-but-unreferenced appender produces "Appender named [X] not referenced. Skipping further processing." An appender used only by otlp-scoped loggers must itself sit under <springProfile name="otlp">, or every non-otlp run — every local development run — warns

No NopStatusListener

Tempting, and wrong. Logback’s status output (lines prefixed |-INFO in ch.qos.logback…​) is written by its StatusManager, not through the appenders, so no filter reaches it — but the volume is a symptom: StatusPrinter dumps the entire status list, some 70 lines, as soon as a single WARN appears. Silencing it hides the cause along with the noise and permanently suppresses genuine Logback errors, a misspelled appender class among them. Fix the warnings instead; a warning-free configuration prints nothing

The two rules above were found together on 2026-08-23. Every EMS service had <springProfile> nested inside <root>; admin-service had carried a NopStatusListener since its original configuration, which is exactly why the invalid nesting went unnoticed for so long. Measured on ems-mcp-server, same boot command either side of the fix: 72 status lines before, 0 after.

Assert these in a unit test rather than trusting review. Every item fails silently in production: the application starts, serves traffic, and either stops exporting or exports twice, with no error and no failed probe. LogbackTwoStreamConfigurationTest in admin-service, admin-portal and ems-mcp-server parses the shipped file and asserts each rule.

Parsing the XML is necessary but not sufficient — it cannot distinguish a <springProfile> Spring Boot honours from one it declines. LogbackProfileWiringTest (ems-mcp-server) is the stronger check: it drives the shipped file through Spring Boot’s own SpringBootJoranConfigurator with profiles active and asserts what Logback actually built — [CONSOLE, OTLP] under otlp, [CONSOLE] without it, and no warnings under either. It lives in package org.springframework.boot.logging.logback because that configurator is package-private; a Spring Boot upgrade that moves it breaks compilation, which is the failure mode to prefer. Copy it into each service.

8. Logs-to-Traces Correlation

opentelemetry-logback-appender-1.0 emits every log record with the current trace and span IDs as structured fields. In Grafana, clicking a trace shows the correlated log lines.

Correlation works on the OTLP path only. An earlier revision of this page claimed the IDs also reach kubectl logs via %X{trace_id} / %X{span_id} in the console pattern. No EMS console pattern contained those tokens, so that was never true.

If you do want trace IDs on console, the agent’s Logback MDC instrumentation supplies them — but the keys are snake_case. Measured on agent 2.10.0 with an active span:

MDC = [trace_id=ceaf42e3…, trace_flags=01, span_id=16fee16e…]

ems-mcp-server used %X{traceId:-} — camelCase — which matched nothing and rendered an empty column on every line it ever logged, failing silently because :- substitutes an empty string rather than raising. Corrected to %X{trace_id:-} in ADO-1013. Use trace_id and span_id.

9. Sampling

Current default: 100% (all traces exported). Low volume; backend handles it. When volume grows:

  • Head-based sampling at the collector — drop 90% of low-interest traces (health checks, readiness probes), keep 100% of error traces, keep high-percentage of slow traces.

  • Tail-based sampling — collector decides after collecting the full trace, based on total duration + error status.

Tune at the collector, not at the service. Services always emit; collector filters.

10. Known Gaps

Post-migration position, 2026-08-23.

  • registration-portal and membership-ui have no javaagent attached — both carry a complete set of OTEL_* environment variables that nothing consumes, so neither produces traces, metrics or OTLP logs. ADO-1010 and ADO-1011. Their console output still reaches Loki via Alloy.

  • ems-mcp-server is not yet verified end-to-end — stage and prod ran an image with no agent and were pointed at the gRPC port; both are corrected but unverified. The service is also silent when idle (56 stdout lines in 8 days), so verification requires generating traffic rather than merely deploying.

  • Frontend instrumentation — browser-originated OTLP proxied via gateway is designed (see design-journal/2026-03/end-to-end-distributed-tracing.adoc) but not implemented. Adds browser-to-admin-service trace root.

  • No SLO tracking — metrics exist but service-level objectives are not formalised. Future work.

Closed since the last revision: the ELK/Kibana backend is gone, replaced by in-cluster Loki, Tempo and Grafana; admin-ui is retired; the collector now has resource limits and terminates every signal inside the cluster.

A recurring lesson across all of the above: every failure mode in this page is silent. A missing agent, a hard-coded agent path, a gRPC port without a protocol variable, an appender with no SDK, a camelCase MDC key — none of them raise an error, fail a probe, or increment a counter. The service starts, serves traffic, and reports nothing.

When verifying, never treat absence of errors as evidence. Check /proc/1/cmdline for the agent, and query Loki for records carrying exporter="OTLP" — a service can appear in Loki purely through the Alloy stdout path while exporting nothing itself.

11. Reference

File Role

admin-service/src/main/resources/config/application-otlp.yml

OTel runtime config for admin-service

admin-service/src/main/java/…​/config/MetricsConfiguration.java

Custom metrics registry + OTel bridge

admin-service/src/main/java/…​/config/apidoc/OtlpMetricsNamingConvension.java

OTel-style naming convention

admin-service/pom.xml (otlp profile)

Dependencies + javaagent download

~/dev/idl-xnl-jhb-rc01/argocd/opentelemetry-collector.yml

OTel Collector ArgoCD Application

design-journal/2026-03/end-to-end-distributed-tracing.adoc

Cross-cutting tracing design including frontend

12. Related Documentation

13. Change History

Date Change

2026-04-24

Initial draft. Grounded in application-otlp.yml and admin-service otlp Maven profile.

2026-08-23

2026-08-23

Added the conformance checklist for logback-spring.xml.

2026-08-24

2026-08-24

Documented that resource attributes must travel in OTEL_RESOURCE_ATTRIBUTES, not as Spring properties. Installing the Logback appender with GlobalOpenTelemetry.get() moves log export onto the agent’s SDK, so otel.resource.attributes in application-otlp.yml stops reaching log records — deployment.environment disappeared silently from admin-portal at 0.1.4 and from ems-mcp-server at 0.1.2. Added the service.namespace warning. Corrected the sample config, which hard-coded deployment.environment: production.

2026-08-23

2026-08-23

Reversed the NopStatusListener guidance added earlier the same day. The ~70 lines of Logback status on the operator stream are a symptom: StatusPrinter dumps the whole status list once any WARN appears, and every EMS service had two real faults producing one — a <springProfile> nested inside <root>, which Spring Boot rejects, and an appender declared outside the profile that references it. Suppressing the listener would have hidden both, and silenced genuine Logback errors permanently. Checklist now requires fixing the warnings and forbids the listener. Added LogbackProfileWiringTest, which asserts the appenders Logback actually built rather than the XML that describes them.

2026-08-23