Observability Platform

Services are instrumented with OpenTelemetry and nothing else. No service knows that Loki, Tempo or Prometheus exist — each one exports OTLP to a single collector endpoint, and the collector decides where signals go.

This is the property that made the Elastic migration survivable. Replacing the entire backend changed collector configuration and Helm values; it changed no application code, and no service redeployed to make it happen. A backend the applications knew about would have made the same migration a coordinated release across every service.

All three signals converge on one OpenTelemetry Collector before they are routed. That gives a single place to apply sampling, filtering, noise suppression and label normalisation, and a single place to change when a destination changes.

It also concentrates risk, which is why the collector’s own health is one of the two things federated out of the cluster — see Watching the watchmen.

The stack runs on the on-premises workload cluster, on Longhorn storage that was already paid for and largely unused. Telemetry does not leave the cluster in normal operation, retention is bounded by disk rather than by a billing tier, and there is no per-GB ingest charge to ration against.

The alternative — hosting the platform on the Azure management cluster — was measured and rejected: 24 GB of arm64 capacity already running at 147–175% memory overcommit, on per-GB billed Azure Disk. See ADR-0011.

Every component is deployed from a chart mirrored into an OCI registry under our own namespace, rather than pulled from the publisher’s repository:

Upstream chart repositories withdraw versions and occasionally disappear. Two ArgoCD applications elsewhere in the estate were found unmanageable in August 2026 for exactly those reasons — one pinned to a version the publisher had removed, one pointing at a repository that had begun returning 404. Mirroring means a rebuild does not depend on a third party still publishing what it published last year.

Services export OTLP traces to the collector. The trace pipeline filters known noise, applies resource attributes, derives RED metrics through the spanmetrics connector, and exports to Tempo.

The spanmetrics connector reads the span stream and emits RED metrics — the three service-level measures that answer "is this service healthy?" without knowing anything about what it does:

RED is the request-driven counterpart to USE (Utilisation, Saturation, Errors), which describes resources — CPU, memory, disk. Both are collected here: RED from the trace stream for services, USE from kube-state-metrics and node exporters for the cluster.

spanmetrics is also the reason no EMS service has a ServiceMonitor. All three measures are derived from the trace stream the services already emit; scraping Actuator as well would land the same measurements a second time under a different label set, at additional cardinality cost.

Logs reach Loki two ways, and the distinction matters when a query comes back empty:

The second path is stdout and stderr only. A container writing its real log to a file inside its own filesystem is invisible to it regardless of how correct the query is — a failure mode that has already cost debugging time on the SSH bastion.

A second Alloy instance ships Kubernetes events to Loki, labelled to distinguish them from pod logs.

The collector remote-writes OTLP metrics into the Prometheus that Rancher monitoring already operates, rather than introducing Mimir or a second Prometheus. That instance already held a quarter of a million active series with working retention; enabling its remote-write receiver restored the metrics pipeline without adding a component.

Blackbox Exporter probes the estate’s public endpoints from inside the cluster, with the probe list held as Probe custom resources under GitOps.

Blackbox Exporter runs inside the cluster it probes. It resolves the public hostname and follows it back to the ingress, so it verifies that the ingress, the service and the application behind it are answering — but the request never leaves the estate’s network.

That leaves a specific blind spot: the cluster is healthy and reporting, and the site is still unreachable from the internet. DNS, the upstream network, or a certificate as seen by an outside client can all fail while every in-cluster probe stays green.

Observation therefore runs in three layers, each catching what the others cannot:

Alloy on the Azure cluster probes a representative set of the estate’s public endpoints from Azure East US — a different region and a different network path — and writes the results to Grafana Cloud.

Three properties are deliberate:

One target needs a non-default check. mcp.idealogic.co.za is OAuth-protected and returns 401 at its root, so it is probed with a module that accepts 401 alongside 2xx. A 401 there has already proved DNS, TLS and liveness; treating it as a failure would have meant a permanently firing alert.

When an external probe fails, the first diagnostic is the in-cluster probe for the same endpoint. If that one is still passing, the fault is in the path to the cluster rather than in the application behind it — which is precisely the distinction the layer exists to make.

A further rule watches for the external results stopping altogether, because a lost vantage point is otherwise indistinguishable from a healthy estate. That is the same silent-failure shape the heartbeat rules address, so it sits in the same group.

Alerts are Prometheus- and Alertmanager-managed; Grafana’s own unified alerting holds no rules. Routing is first-match, with info and none severities routed to a null receiver — on a three-node estate they are noise — and inhibition rules written only where both alerts genuinely share a join label. A rule whose equal labels are absent on one side never fires and looks identical to one that has simply had no cause to.

Note that a rule engine other than Prometheus — the Loki ruler, for instance — reaches Alertmanager directly and never appears in Prometheus’s ALERTS metric. Anything counting alerts from ALERTS alone will under-report. See Observability Access.