Operations

1. Overview

This module documents system-health operations for the Event and Membership Administration System in production environments — keeping the system itself running. It covers API-based management access, cache eviction, observability, and outbound email configuration, none of which require application restarts.

This module focuses on keeping the system healthy. For producing business outcomes through the system (running imports, processing the number/tag lifecycle, generating reports) see the Operational Procedures module. For exceptional one-off recovery see Runbooks. For deployment and infrastructure see Deployment.

2. Environments and Access

  • Environments - Hostnames, cluster namespaces and database schemas per environment, the mapping from a public host to its tenant and organisation, and the environment-specific behaviour that catches people out: the stage mail sink, the nightly stage reset, and the cluster’s primary-key requirement on new tables.

  • Credential Inventory - What each credential is for, where its value is stored, and the rules for adding and rotating one. Records locations and purposes only, never values.

3. Provisioning

  • Component Configuration Checklist - Per-component configuration required to enable each part of the offering for a customer: manifest values, Secret keys, declared WordPress options, wp-admin steps, media assets and out-of-band prerequisites. Used to scope what a customer site needs and what the customer must provide, then worked through as a commissioning checklist. A working document.

4. Management API

  • Management API Access - Authenticate and access Spring Boot Actuator endpoints using API keys for log level management, health checks, and configuration inspection.

5. Cache Management

  • Cache Management - Understand the caching architecture (Spring + Hibernate L2 + Hazelcast) and evict caches cluster-wide without restarting pods.

6. Email

  • SMTP Configuration - Outbound email setup for EMS services: production Exchange Online connector, dev GreenMail, shared spring.mail.* config surface, multi-tenant From-address considerations.

7. Observability

  • Observability Access - Reading telemetry back: Grafana sign-in and roles, which datasource answers which question, LogQL for EMS services, the dashboards that exist, where alerts surface, and the log/trace correlation path.

  • OpenTelemetry Configuration - OTel javaagent + SDK configuration for admin-service, collector topology, trace/metric/log correlation, sampling strategy, and the known gap of OTel not yet enabled on registration-portal.

8. Infrastructure

  • MySQL Connection Capacity - Why connection-pool bloat in a lower environment can exhaust the shared production server, the trap where the operator does not apply a changed server configuration, and the persisted-variable plus per-account fix with its verification queries.

  • Bastion Access - SSH tunnels into cluster services: the connection refusal that reads as a network fault but is per-source penalties charged against a shared source address, and the image that treats its configuration file as a writable seed.

9. Dependencies

  • Dependency Update Review - The weekly review of proposed third-party dependency updates: where to find them, the exposure and magnitude tiers that set the level of scrutiny, the conditions all of which must hold before approving, how to implement through ArgoCD, component verification, and rollback.

Operator guides for running imports, the number/tag lifecycle, and reporting moved to the Operational Procedures module — they produce business outcomes rather than keeping the system healthy.