ADR-0011: Self-host the observability backend instead of paying a managed vendor

Status

Accepted
Date

2026-08-17
Deciders

Solution Architect
Related

design-journal/2026-08/self-hosted-observability-platform.adoc

1. Context

EMS services were instrumented with OpenTelemetry and exported to a managed Elastic Cloud backend. The arrangement had three problems, and only the first was visible from the invoice.

Delivery was a small fraction of what was billed. Measured over a 45-day window, Elastic was receiving roughly 6–7% of the telemetry sent to it, with more than 38 million records dropped before reaching the wire. The bill did not reflect that; it reflected the plan. Paying in full for a backend that discards most of what it is given is not a tuning problem, it is a reason to leave.

The metrics pipeline was failing outright. Whatever was decided about logs and traces, metrics needed fixing, and the fix was in-cluster in every candidate design.

Retention was a billing tier rather than a capacity decision. How much history the team could keep was set by what had been purchased, not by what the estate could hold.

Against that, the on-premises workload cluster had 4.29 TiB of unused Longhorn capacity, already ran Grafana, Prometheus and Alertmanager, and is where the telemetry originates in the first place.

The forces pulling the other way were real. Self-hosting means operating the platform: upgrades, storage, and the uncomfortable property that a monitoring system living inside the thing it monitors cannot report its own death.

2. Decision

Terminate Elastic Cloud. Run the observability backend on the workload cluster: Loki for logs, Tempo for traces, the existing Rancher Prometheus for metrics, and one Grafana over all of them.

Keep OpenTelemetry as the instrumentation boundary. No service may reference a backend directly. Services export OTLP to a single collector; the collector decides destinations.

Do not pay Grafana Cloud as a replacement vendor. Use its free tier for exactly two purposes: an external dead-man’s switch, and metrics from the Azure management cluster.

3. Consequences

3.1. Positive

  • No per-GB ingest billing, and no reason to ration what is sent.

  • Retention is a capacity decision. 30 days for both logs and traces, set by disk.

  • The backend became replaceable. Swapping Elastic for the Grafana stack changed collector configuration and Helm values and no application code. The next such change will be equally cheap.

  • Metrics work again, by enabling remote-write on a Prometheus that was already running.

  • Telemetry stays in the cluster in normal operation.

3.2. Negative

  • We operate the platform now — upgrades, storage growth, and the failure modes of five more components.

  • Shared failure domain. The stack runs on the cluster it observes. If that cluster is gone, so is the evidence of why. This is mitigated, not eliminated, by the dead-man’s switch below.

  • Grafana OSS has no fine-grained RBAC — three org roles. Giving a developer Explore without write access to shared dashboards required viewers_can_edit, which is a least-privilege workaround rather than a designed permission.

3.3. Neutral

  • Two signals now reach Grafana Cloud’s free tier (the heartbeat), so the vendor relationship is reduced rather than ended.

  • Loki runs single-binary with the distributed components scaled to zero. Correct for a single-tenant estate; it is not a horizontally scalable deployment and would need rethinking if it became one.

4. Alternatives Considered

4.1. Stay on Elastic Cloud and tune the pipeline

The delivery shortfall might have been an export or mapping fault rather than a product limit. Rejected because the cost of investigation was open-ended against a backend that would still bill per GB afterwards, and because the metrics pipeline needed in-cluster work regardless. It would become the right choice again if a managed vendor’s cost model changed to something that did not penalise volume.

4.2. Move to paid Grafana Cloud

The most direct swap, and it would have avoided operating anything. Rejected because traces and logs were already landing at roughly 100% on the free tier, so the only broken signal was metrics — whose fix was in-cluster in every design. Paying to solve a problem that was already solved for free was not defensible. It becomes the right choice if the team grows past the point where operating the platform is cheaper than buying it.

4.3. Host the platform on the Azure management cluster

This was the earlier design, and it buys genuine absence-detection by putting the observer outside the observed. Rejected on measured capacity: three arm64 nodes totalling 24 GB, already at 147–175% memory overcommit, on per-GB billed Azure Disk — the exact cost being escaped. The architectural benefit was real, so it was preserved another way: see the dead-man’s switch.

4.4. Deploy Mimir alongside Loki and Tempo

Completes the Grafana stack and gives long-term metric storage. Rejected because the Rancher Prometheus already held ~251,000 active series with working retention, and enabling its remote-write receiver closed the metrics gap with no new component. It becomes the right choice when Prometheus retention or cardinality becomes the binding constraint.

5. Notes

The shared failure domain is the one consequence with no clean answer, so it is mitigated explicitly rather than accepted silently. A dedicated collector pipeline federates two stable series — ALERTS{alertname="Watchdog"} and alertmanager_build_info — to Grafana Cloud, where two alert rules fire on their absence. Detection lives outside the failure domain, and costs no infrastructure.

The choice of an absence detector over a heartbeat SaaS is deliberate: it covers the whole chain from rule evaluation to notification, rather than one endpoint responding, and it supplies a Grafana that still works when the cluster does not.

6. References