Environments
1. Overview
EMS runs three environments on one Rancher-managed cluster, each in its own namespace, each against its own MySQL schema on the shared production MySQL cluster. "Production" is a posture as well as a place: stage deliberately runs the same Spring profile as production so that it exercises the production posture. See Spring Bootstrap and Configuration Precedence for what follows from that.
|
Never cache a deployed version number. Which image is live in a given namespace changes without
notice. Read |
2. Service hostnames
| Service | Production | Stage | Development |
|---|---|---|---|
admin-service |
|
|
|
Registration portal |
|
|
|
Membership portal |
|
|
|
Admin portal |
— |
— |
|
Mail sink |
— |
|
|
Two admin-service hostnames exist in production because the service has been running on two deployment lines during the migration to the current line. They serve different namespaces against the same schema, so a row in the database does not tell you which instance wrote it. They also form separate cache clusters, which means a direct SQL change requires a cache eviction against both — see Cache Management. The current state of that migration, and which host serves which tenant, is tracked in the design journal rather than here, because it is a transitional arrangement with an end date.
3. Namespaces and schemas
| Environment | Namespace | Schema | Notes |
|---|---|---|---|
Production |
|
|
The current line. Owns the database migrations. |
Production, legacy line |
|
|
Frozen image, same schema. Retiring. |
Stage |
|
|
Refreshed nightly from production at 04:00 SAST. |
Development |
|
|
Loads fixture data under the |
|
Do not query the |
The Liquibase bookkeeping tables are renamed from their defaults in this estate. They are
change_log_event_admin and change_lock_event_admin, not DATABASECHANGELOG and
DATABASECHANGELOGLOCK. A migration investigation that greps for the default names finds
nothing and reads as a clean database.
4. Host to tenant to organisation
The registration portal resolves a tenant from the request’s hostname, and the tenant carries the registration system that determines the organisation. This is the mapping that makes a public URL resolve to a set of data.
| Host | Tenant | Registration system | Organisation |
|---|---|---|---|
|
HNR |
1502 |
8 — Helderberg Nature Reserve |
|
HNR VC |
1503 |
8 — Helderberg Nature Reserve |
|
WCSC |
1504 |
9 — Western Cape School Cycling |
|
Myriad Events |
1504 |
9 — shared with WCSC |
|
Tour de Worcester |
1505 |
10 — Myriad Events |
|
Default V2 Tenant |
1508 |
10 — Myriad Events |
|
— |
1509 |
10 — Myriad Events |
|
Registration system ids differ between production and development. The same tenant carries a different id in each environment, and a development id may not exist in production at all. Always use production ids when reasoning about production. Tenant rows are operational data — only fixture-context seeds exist in the migrations, so the database is the source of truth. |
5. Reading what is deployed
curl -H "X-API-KEY: $(cat ~/dev/ems/.local/secrets/admin-service-prod-admin.key)" \
https://<host>/management/infoThe production key is rejected by stage, which has its own. Both are catalogued in Credential Inventory.
6. Environment-specific behaviour
6.1. Stage cannot send mail to the outside world
Every Java service in the stage namespace points its mail host at an in-cluster GreenMail sink with authentication and TLS off. The external relay and its credentials are absent from the namespace entirely. This matters because stage’s database is refreshed from production nightly, so it holds real customer addresses — a recipient override would still leave a route out, whereas removing the relay removes the possibility.
The sink also makes delivery assertable. Query the recipients and their messages over its HTTP API on port 8080, or read mail in the browser at the stage webmail host, where any passphrase is accepted because authentication is disabled.
curl http://greenmail.event-stage.svc.cluster.local:8080/api/user
curl http://greenmail.event-stage.svc.cluster.local:8080/api/user/<email>/messagesThere is no /api/mail path; it returns not-found. Development has the same pair.
6.2. Stage is reset nightly
The 04:00 SAST clone from production replaces stage’s data wholesale. Anything written to stage by hand — an API key permission, a test fixture, a corrected row — is gone the next morning unless it is written to the corresponding seed table as well. See Credential Inventory for the API key case, which is the one that recurs.
6.3. Creating tables on the production cluster
The production MySQL runs as an InnoDB cluster with Group Replication, which requires every table
to have a primary key. CREATE TABLE … AS SELECT produces a table without one and is rejected.
Create the table with an explicit primary key first, then populate it:
CREATE TABLE zz_working (id BIGINT NOT NULL PRIMARY KEY, ...);
INSERT INTO zz_working SELECT ... ;mysqldump --where to a file, or a TEMPORARY table, are the other ways round it.
6.4. Running admin-service locally
The local development run uses two different sets of database credentials, which is the usual first stumble: Liquibase connects as the root user with an empty password to create the schema, while the application’s own datasource connects as the development user. Both must exist in the throwaway MySQL container. The service listens on port 12504; the registration portal uses 12505 and the admin portal 12506.
7. Related
-
Credential Inventory — where each environment’s key lives.
-
Management API Access — authenticating to the management endpoints listed above.
-
Cache Management — evicting caches after a direct database change, on every instance sharing the schema.
-
Spring Bootstrap and Configuration Precedence — why stage runs the production profile and what can and cannot be overridden per environment.