Observability Access

1. Overview

OpenTelemetry Configuration describes how EMS services emit telemetry. This page is its counterpart: how to read it back.

Everything below is available without cluster credentials. Reaching for kubectl logs should be the exception, not the habit — it shows one pod’s current console buffer, while the stack holds every pod’s history across restarts and deletions, with the queries and alerts that give it meaning.

2. Getting access

Grafana is at https://grafana.idealogic.co.za and signs in with Entra ID (Microsoft) using your normal work account. No separate credential is issued.

New users are provisioned automatically on first sign-in and land on the Viewer role. Viewer is not as restrictive as it sounds: the instance sets viewers_can_edit, so a Viewer can open Explore and Drilldown, write ad-hoc queries, and build panels in a scratch dashboard. What Viewer cannot do is save changes to shared dashboards or alter alert rules — which is the intended boundary. Ask an administrator for Editor only when you need to persist something the whole team will see.

3. Which datasource answers which question

Datasource Type Answers

Prometheus

Prometheus (default)

Metrics for the on-premises cluster: service latency, throughput, JVM and pod resource use, and everything the EMS services expose through Actuator/Micrometer.

Loki

Loki

Logs. Both the OTLP application-log path and the Alloy pod-stdout path land here.

Tempo

Tempo

Distributed traces — a single request’s path across services.

Alertmanager

Alertmanager

The current state of alerts: firing, pending, suppressed.

Grafana Cloud (AKS)

Prometheus

Metrics for the Azure (AKS) management cluster only. See the caveat below.

The AKS cluster ships metrics only — no logs and no traces. That is a deliberate cost decision, not a fault. Querying Loki for an AKS workload’s logs correctly returns nothing. Only the on-premises cluster’s logs reach Loki.

This also means a dashboard built against the default Prometheus datasource will show no data when pointed at AKS workloads, and vice versa. Check which datasource a panel is using before concluding a workload is down.

4. Logs (Loki)

4.1. Stream labels

Pod logs carry cluster, namespace, pod, container, app, job and source. job is <namespace>/<container>, and source separates pod-logs from kubernetes-events.

EMS services run in the event-dev, event-stage and ems-prod namespaces.

4.2. Severity is structured metadata, not a label

Level is deliberately not a stream label — putting it on one would multiply stream cardinality by its value count for no query benefit. Filter on it after the stream selector:

{namespace="ems-prod", container="admin-service"} | level = "ERROR"

Loki additionally infers detected_level. Where an explicit level has been extracted at ingest it is the more reliable of the two, because it comes from the log format rather than a heuristic.

4.3. Common queries

# everything from one service
{namespace="ems-prod", container="admin-service"}

# errors only
{namespace="ems-prod", container="admin-service"} | level = "ERROR"

# text search across an environment
{namespace="ems-prod"} |= "OutOfMemory"

# error rate per service — chartable and alertable
sum by (container) (rate({namespace="ems-prod"} | level = "ERROR" [5m]))

For "how many" and "how often", prefer a metric query like the last one. Log queries return a capped number of lines, so counting returned lines will understate the answer.

4.4. Java stack traces

For the EMS namespaces, Alloy joins multi-line stack traces into a single entry at ingest, so one entry is one event and the level on the first line applies to the whole trace. Without this, each frame arrives as its own line with no detectable severity — on 2026-08-24, 8,120 of 8,355 event-dev lines were classified unknown for exactly that reason.

This is configured per format. A service whose console pattern does not match the expected ISO-8601 anchor will still produce frame-per-line output.

4.5. Retention

Loki retains for 720h (30 days). Older data is gone; an empty result for an old incident means expiry, not absence.

4.6. Logs the collector never sees

The Alloy DaemonSet tails container stdout and stderr. A process that writes its real log to a file inside its own container is invisible to Loki no matter how correct the query is. If a service is definitely running and definitely logging but returns nothing, suspect this before suspecting the query.

5. Traces (Tempo)

Loki and Tempo are wired together in both directions:

  • Log to trace. Loki carries a derived field that turns a traceid in a log line into a View trace link. This lights up on OTLP application logs only — Spring’s plain console pattern carries no MDC trace fields, so pod stdout collected by Alloy will not have it.

  • Trace to log. Tempo is configured with tracesToLogsV2, so a span links back to the logs around it, filtered by trace ID with a five-minute window either side.

The practical path from a symptom to a trace is therefore: find the failing request in Loki on the OTLP stream, follow the View trace link.

6. Alerts

Alert state is visible in Grafana’s Alerting section and on the Operational Worklist dashboard, which counts critical, warning, info and suppressed alerts, each linking through to the filtered list.

Alerting is run by Prometheus and Alertmanager, not by Grafana’s own unified alerting. This matters when querying programmatically: Grafana’s alert-rule API returns nothing, because no Grafana-managed rules exist. Query the ALERTS metric on the Prometheus datasource instead:

sum by (alertname, severity) (ALERTS{alertstate="firing"})

ALERTS does not show every firing alert. Prometheus synthesises it only for rules Prometheus itself evaluates. This estate also runs Loki ruler rules (see observability/loki-rules/), which are evaluated by Loki and pushed straight to Alertmanager — they never appear in ALERTS at all.

Verified 2026-08-24: LokiRulerCanary had not appeared in ALERTS once in six hours, while Alertmanager reported it active throughout. Because the absence is permanent rather than intermittent, it does not look like flapping and reads as a healthy quiet.

Cross-check totals against Alertmanager and trust Alertmanager where they differ:

sum by (state) (alertmanager_alerts)

alertmanager_alerts carries only a state label, so it reveals the discrepancy but not the alert names. Full detail requires the Alertmanager datasource — reachable through Grafana’s Explore, or programmatically via the datasource proxy at /api/datasources/proxy/uid/alertmanager/api/v2/alerts.

The Operational Worklist dashboard currently counts ALERTS on the Prometheus datasource and therefore inherits this blind spot.

A permanently-firing Watchdog alert at severity none is normal and proves the alert path itself is alive. It is not a problem to investigate.

7. Dashboards

The EMS folder holds the dashboards maintained in idl-xnl-jhb-rc01. A second, much smaller set lives outside this Grafana entirely — see When Grafana itself is what is down.

Dashboard Purpose

Operational Worklist

Current alert counts by severity, each linking to the filtered list.

EMS Service Health

The EMS services themselves — availability, errors, resource use.

ArgoCD Applications

Sync and health for every ArgoCD application across all clusters.

AKS Cluster Health

Node, pod and certificate state on the Azure management cluster.

Endpoints & Certificates

Blackbox probe results and TLS certificate expiry, from two vantage points: the in-cluster probes at the top, and an External vantage row probing the same estate from Azure East US. Both probe every tenant alias individually, not one host per service, so each hostname’s own certificate and DNS record are covered. Compare the two when something is reported down — see below.

OTel Collector

Collector throughput and export health — start here if telemetry stops arriving.

7.1. Reading the two vantage points

The Endpoints & Certificates dashboard shows each endpoint probed from inside the cluster and from outside it. That pairing is the diagnostic:

In-cluster External Means

Passing

Passing

Healthy.

Passing

Failing

The application is fine and the path to it is not — DNS, upstream network, or a certificate as an outside client sees it. Users are affected; the service is not at fault.

Failing

Failing

The application or its ingress. Start with the service.

Failing

Passing

Unusual. Suspect the in-cluster probe itself — a cluster-DNS or egress problem affecting the prober rather than the endpoint.

The external row also carries DNS lookup time, which the in-cluster probes cannot meaningfully report because they may resolve through cluster DNS.

7.2. When Grafana itself is what is down

Everything above lives in the self-hosted Grafana, which runs on the cluster it observes. If that cluster is unreachable, so are those dashboards — and that is exactly the moment you need them.

The external vantage therefore has a second home: a Grafana Cloud instance, https://christhonie.grafana.net, in the folder Idealogic Sentinel. It holds

  • the Idealogic Sentinel — external view dashboard, showing the same 18 probed hostnames, their certificate expiry and DNS lookup times, sourced entirely from outside the on-premises estate; and

  • the alert rules that cannot be trusted to the cluster itself — the two dead-man’s-switch rules, the external-endpoint-unreachable rule, and the rule that fires when the external prober stops reporting at all.

Nothing in that folder depends on the jhb1 cluster being alive. When the self-hosted Grafana does not answer, go there — it is the only view that survives the outage it is meant to describe.

Both are defined under observability/grafana-cloud/ in idl-xnl-jhb-rc01. They are not provisioned automatically: Grafana Cloud has no GitOps hook here, so changes are pushed with the curl commands in that directory’s README, and the files in Git are the source of truth for what should be there.

A further set of Kubernetes dashboards ships with the monitoring stack and is provisioned automatically. Many depend on recording rules; a mixin dashboard showing no data usually means its recording rule is absent rather than that the workload is idle.

8. Verifying a deployment

Whether a rollout landed is an observability question before it is a cluster question. The ArgoCD Applications dashboard shows sync and health for every application without a cluster login. See ArgoCD Deployment Patterns, which also covers the sync=Unknown failure mode that leaves an application unmanaged while still reporting healthy.

9. Scripted access

Grafana’s HTTP API is reachable with a service-account token, and the datasource proxy (/api/datasources/proxy/uid/<uid>/…​) reaches Loki, Prometheus, Tempo and Alertmanager directly — including the Alertmanager endpoints that no dashboard panel can query.

Requests sent with the default Python-urllib user agent are rejected with 403 Forbidden before they reach Grafana. The token and permissions are irrelevant — the same request succeeds immediately with any other user agent.

This is easy to misread as a permissions problem, because 403 is exactly what an under-privileged token returns. If a script gets 403 while curl succeeds with the same token, set a user agent before investigating the token:

req.add_header("User-Agent", "ems-tooling/1.0")

10. AI-assisted access

Claude Code sessions read this stack through the grafana MCP server, shipped in the idealogic-dev plugin and configured with a per-user read-only service account token. The devops/observability skill defines the routing. Sessions should query the stack rather than connecting to a cluster; cluster access remains for live tail, exec, object-level inspection and anything requiring a write.

11. Related Documentation