ArgoCD Deployment Patterns

1. Overview

EMS services are deployed to Kubernetes via ArgoCD. One ArgoCD Application manifest per (service × environment) declares which chart version to pull, which values to apply, and which namespace to deploy into. ArgoCD watches the manifest repository and reconciles the live cluster state against the declared state.

Manifests live outside this repository — in ~/dev/idl-xnl-jhb-rc01/argocd/ (locally; repository name idl-xnl-jhb-rc01). That repository is the source of truth for what runs where. This page documents the patterns so new services can slot in.

See also: Helm Chart Structure, Jib Docker Build.

2. Manifest Layout

idl-xnl-jhb-rc01/argocd/
├── event-admin-service-dev.yml
├── event-admin-service-stage.yml
├── event-admin-service-prod.yml
├── registration-portal-dev.yml
├── registration-portal-stage.yml
├── registration-portal-prod.yml
├── membership-ui-dev.yml
├── ...
├── admin-portal-dev.yml           # future (WS7)
├── admin-portal-stage.yml
├── admin-portal-prod.yml
├── opentelemetry-collector.yml
├── mysql-idealogic-prod.yml
├── external-dns.yml
├── certificates.yml               # cert-manager issuers
└── ...

Convention: one file per ArgoCD Application, named <service>-<env>.yml. Infrastructure (OTel collector, ingress certs, MySQL operator) has its own files at the top level.

3. Application Manifest Structure

Reference: idl-xnl-jhb-rc01/argocd/event-admin-service-prod.yml.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: event-admin-service-prod
  namespace: argocd
spec:
  project: default
  destination:
    name: idl-xnl-jhb1-rc01
    namespace: event-prod
  source:
    repoURL: registry-1.docker.io
    chart: christhonie/event-admin-service
    targetRevision: 2.3.31-RELEASE
    helm:
      releaseName: prod-event-admin-service
      valuesObject:
        config:
          profiles: "prod,kubernetes"
          existingsecret: event-admin-service
          db:
            url: mysql://idealogic-prod.mysql.svc.cluster.local:6446/wpca_prod?useUnicode=true&characterEncoding=utf8
            username: event-membership-prod
          security:
            oauth2:
              enabled: false
              issuer: https://...
          mail:
            host: myriadevents-co-za.mail.protection.outlook.com
            port: 25
            username: [email protected]
            # password: via existingsecret, NOT inline
          liquibase:
            contexts: "prod"
          otel:
            enabled: true
            url: "http://opentelemetry-collector.observability.svc.cluster.local:4318"
        image:
          pullPolicy: IfNotPresent
        imagePullSecrets:
          - name: christhonie-docker
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true
      - ServerSideApply=true

Key fields:

Field Purpose

metadata.name

<service>-<env> — matches filename

spec.project

default for EMS services; separate projects for infra if RBAC needs it

spec.destination.name

ArgoCD cluster secret name (currently all one cluster: idl-xnl-jhb1-rc01)

spec.destination.namespace

event-dev / event-stage / event-prod / observability / mysql etc.

spec.source.repoURL

registry-1.docker.io for OCI-hosted charts (our pattern); Git URL for Git-hosted charts

spec.source.chart

christhonie/event-admin-service — chart name within the OCI registry

spec.source.targetRevision

Pinned chart version — bump this to promote a release

spec.source.helm.releaseName

<env>-<chartname> — yields service names like prod-event-admin-service

spec.source.helm.valuesObject

Environment-specific values; merged over chart defaults

spec.syncPolicy.automated

prune: true removes resources that disappear from the manifest; selfHeal: true reconciles manual drifts

4. Environments

Env Namespace Purpose

dev

event-dev

Continuous deployment from develop branch builds (-SNAPSHOT images). Auto-upgrade; no guardrails. Shared data.

stage

event-stage

Pre-release testing. Manual promotion of specific release versions. Prod-cloned data (see design-journal/2026-03/prod-to-qa-data-clone.adoc).

prod

event-prod

Live traffic. Manual promotion, approval gate, pinned versions only.

Release flow:

  1. Feature branch merged to develop — CI builds X.Y.Z-SNAPSHOT, dev ArgoCD picks up automatically.

  2. Release branch cut — CI builds X.Y.Z-RELEASE.

  3. Stage ArgoCD manifest updated (commit to idl-xnl-jhb-rc01) to point at X.Y.Z-RELEASE — ArgoCD reconciles within seconds.

  4. UAT in stage.

  5. Prod ArgoCD manifest updated to X.Y.Z-RELEASE — ArgoCD reconciles.

No latest tag ever referenced in stage or prod manifests.

5. Secret Management

Every prod manifest uses existingsecret: <name>. The secret is created out-of-band with kubectl create secret or via SealedSecrets. Never put secrets in the ArgoCD manifest — it is committed to git.

Known secret shapes:

5.1. event-admin-service secret (prod)

Keys:

  • jwtencryptionkey — admin-service signing key

  • dbpassword — MySQL password

  • mailpassword — SMTP password

  • apikey — admin-service’s own copy of the portal-to-admin-service API key

  • externalauthsecretkey — base64 secret for external-auth endpoints

  • jasperreportspassword — credentials for the Jasper Reports integration

5.2. event-admin-portal secret (future prod)

Keys:

  • apikey — portal’s X-API-KEY for calling admin-service

  • oidcclientsecret — OIDC client secret

  • any portal-specific credentials

Create the secret when onboarding the environment:

kubectl create secret generic event-admin-portal \
  -n event-prod \
  --from-literal=apikey=<value> \
  --from-literal=oidcclientsecret=<value>

Rotation is a runbook activity — see API-Key Injection § Rotation Procedure.

5.3. Image pull secret

christhonie-docker must exist in every namespace that pulls EMS images:

kubectl create secret docker-registry christhonie-docker \
  -n event-prod \
  --docker-server=https://index.docker.io/v1/ \
  --docker-username=christhonie \
  --docker-password=<token>

6. Promotion Pattern

Concrete steps to promote admin-service from stage to prod:

  1. Check the ArgoCD Applications dashboard in Grafana first. It shows sync and health for every application across all clusters without a cluster login, and it is the fastest way to confirm the app is actually being managed before you change what it points at. See Observability Access.

  2. Verify stage has the new version running cleanly — kubectl get deploy -n event-stage, smoke-test the URL.

  3. In idl-xnl-jhb-rc01, edit argocd/event-admin-service-prod.yml:

    spec:
  source:
    targetRevision: 2.3.32-RELEASE   # was 2.3.31-RELEASE

  4. Commit + push. ArgoCD picks the change up by polling — see Sync Latency Is 20-30 Minutes, Not Seconds. Use argocd app get <name> --refresh to force it.

  5. Watch rollout:

    argocd app get event-admin-service-prod --refresh
kubectl rollout status deploy/prod-event-admin-service -n event-prod

  6. If rollout fails — rollback is reverting the commit (argocd auto-reconciles back to the previous version).

ArgoCD’s automated.selfHeal: true means manual kubectl edit on a live resource is reverted on the next sync. Always make changes via the manifest, not the cluster.

7. Infrastructure Applications

Non-service ArgoCD apps in the same repo:

App Purpose

opentelemetry-collector.yml

OTel Collector deployment — receives OTLP from all services, exports to the observability backend. See OpenTelemetry Configuration.

mysql-idealogic-prod.yml

Oracle MySQL Operator + cluster. Managed by the operator; we provide the InnoDBCluster CR and backup schedule.

mysql-operator.yml

The operator itself

external-dns.yml

External-DNS configuration for ingress record management

certificates.yml

cert-manager ClusterIssuers (Let’s Encrypt prod + staging)

redis-prod.yml

Redis for session sharing (fallback to Hazelcast; keep for future needs)

memcache-prod.yml

Memcached — legacy, under review

greenmail-dev.yml

GreenMail deployment in the dev cluster for SMTP testing

storage.yml

StorageClass + PV setups

ssh-bastion.yml

Bastion host for MySQL / in-cluster debugging — see memory/reference_mysql_claude_user.md

sonarqube.yml, chatwoot-prod.yml, wordpress-wpca-prod.yml, etc.

Adjacent systems; not EMS core but deployed by the same mechanism

8. admin-portal ArgoCD Onboarding

WS7 tasks:

  1. Create admin-portal-dev.yml / admin-portal-stage.yml / admin-portal-prod.yml in idl-xnl-jhb-rc01/argocd/. Copy from registration-portal-<env>.yml as the shape closest to admin-portal.

  2. Adjust:

    • chart: christhonie/event-admin-portal

    • targetRevision: <first-published-version>

    • releaseName: dev-event-admin-portal / stage-event-admin-portal / prod-event-admin-portal

    • Namespace follows the convention (event-dev / event-stage / event-prod)

    • config.profiles: "dev,kubernetes,api-docs" for dev, "prod,kubernetes,api-docs,otlp" for prod

    • config.services.eventadminservice: in-cluster URL

    • config.security.oauth2: populated with the staff IdP

    • Ingress host: admin-dev.event.idealogic.co.za, admin-stage.event.idealogic.co.za, admin.event.idealogic.co.za

  3. Create the event-admin-portal secret in each namespace with apikey + oidcclientsecret.

  4. Verify christhonie-docker image-pull secret exists in each namespace.

  5. Push the manifests; ArgoCD reconciles.

9. Operational Notes

  • Sync hooks: none in current use. ArgoCD’s pre/post sync hooks (e.g. database migration jobs) are not needed because Liquibase runs on app startup.

  • Sync wave: ordering is not currently enforced via argocd.argoproj.io/sync-wave. All resources come up simultaneously.

  • Health checks: default Kubernetes health (Deployment Ready) is used. Custom health for Helm-managed resources relies on the chart’s own probes (/livez, /readyz).

If a chart upgrade introduces a CRD or new resource type, ArgoCD may need a restart to recognise it — rare; the standard Kubernetes types cover every current resource.

10. Troubleshooting

10.1. ArgoCD app stays OutOfSync after chart bump

  1. argocd app get <name> --refresh — force a re-read from the chart repo.

  2. If still stuck, inspect the rendered manifest: argocd app manifests <name> — validate against kubectl apply --dry-run=server -f -.

  3. Check the chart was actually published: helm pull oci://registry-1.docker.io/christhonie/event-admin-service --version <ver> locally.

10.2. ArgoCD app stays sync=Unknown — chart source cannot be fetched

OutOfSync means ArgoCD compared and found a difference. Unknown means it could not compare at all, because rendering the desired state failed. The distinction matters: an Unknown app is effectively unmanaged, and automated.selfHeal is silently inert because the fetch fails before any comparison happens.

This hides unusually well. The workload keeps running, so health stays Healthy, autosync still reads as enabled, and nothing alerts. Two applications in this estate sat in that state undetected until they were spotted on the ArgoCD Applications dashboard.

Diagnosis — read the conditions, not the metrics. argocd_app_info exposes sync_status and health_status but carries no reason, so the cause is invisible to Prometheus and to any dashboard built on it:

kubectl get application <name> -n argocd -o jsonpath='{.status.conditions}' | jq

Look for type: ComparisonError. The message names the failing source. Then reproduce it directly, which is faster than re-syncing and tells you whether the problem is the version or the repository:

helm pull --version <pinned-version> --repo <chart-repo-url> <chart-name>

Two distinct failures, both seen in this estate:

Failure Symptom Resolution

Pinned version withdrawn upstream

chart "<name>" version "<x.y.z>" not found in <repo> — the repository resolves and its index parses, but that version has been removed. Seen with mysql-operator pinned at 2.1.9 while the index advertised only 2.3.0, so there was no older version to fall back to either.

Re-pin to a published version. Treat a multi-minor jump on an operator managing live data as an upgrade with a maintenance window, not a version bump — check CRD changes first.

Repository itself gone

not a valid chart repository or cannot be reached: …​ 404 Not Found — the publisher has disappeared, not just the version. Seen with a third-party Ansible Semaphore chart repository that returns 404 after the project was renamed and republished elsewhere.

Locate the successor repository and verify it is reachable before migrating. A chart from a different publisher at a much later major version will have a different values schema — treat it as a re-deployment, not a URL swap.

Both cases share a lesson worth acting on separately: an upstream repository can withdraw a version or vanish entirely, and the deployment only discovers it at the next render — which is typically during a rebuild or disaster recovery, at the worst possible moment. Mirroring or vendoring charts removes that exposure.

10.3. Sync Latency Is 20-30 Minutes, Not Seconds

idl-xnl-jhb-rc01 has no GitHub webhook — gh api repos/christhonie/idl-xnl-jhb-rc01/hooks returns an empty list. ArgoCD therefore discovers every push by polling alone, on a slow, staggered per-application schedule.

Measured end to end on 2026-09-07, from one push to main:

Effect Elapsed

grafana-dashboards applied a changed dashboard ConfigMap

~20 min

loki-rules applied a changed rule ConfigMap, and the app-of-apps re-read argocd/

~25 min

A new child Application existed and had applied its own resources

~27 min

Prometheus regenerated its scrape config and the first target reported up

+~1 min, then one scrape interval

A brand-new Application costs two chained cycles: the app-of-apps must sync before the child exists, and only then does the child sync its own contents.

The trap is what the status says while you wait. Every application reported sync_status: Synced and health_status: Healthy throughout — against the previous revision. Synced means "the last revision I fetched is applied", never "your commit is applied". This is the same lesson as Synced and Healthy Is Not Evidence of Reachability, one level earlier in the pipeline.

Verify by effect, not by status:

# Did any sync actually run? ArgoCD's own metrics live on the AKS management
# cluster, so query the `grafanacloud-aks` datasource, not the workload Prometheus.
sum by (name) (increase(argocd_app_sync_total[10m])) > 0

Then confirm the thing itself — up{job=…​} for a ServiceMonitor, the Loki ruler API for a Loki rule, a dashboard search for a dashboard. Budget 30 minutes before concluding a GitOps change has failed, or force it with argocd app get <name> --refresh.

10.4. Secret reference missing

kubectl get events -n <namespace> shows CreateContainerConfigError with the secret name. Either the secret doesn’t exist (create it) or the namespace is wrong (verify destination.namespace).

10.5. Automated sync loop

If you see ArgoCD reverting a manual change, that’s selfHeal: true working as designed. Make the change in the manifest, commit, push.

To pause reconciliation temporarily:

argocd app set <name> --sync-policy none
# do thing
argocd app set <name> --sync-policy auto

11. Destination and Namespace Changes

11.1. Synced and Healthy Is Not Evidence of Reachability

An Application reporting Synced + Healthy means one thing only: the manifests applied cleanly to whatever cluster spec.destination names. It says nothing about whether users can reach the result, and nothing about whether that cluster is the one serving the hostname you have in mind.

The estate has two clusters. ArgoCD itself runs on the Azure management cluster; public traffic is served from the on-premises cluster. destination.server: https://kubernetes.default.svc resolves to the cluster ArgoCD is running on — the management cluster — which is almost never where a public-facing workload belongs. An Application pointed there deploys successfully, reports green indefinitely, and is completely unreachable, because DNS sends the hostname to the on-premises ingress where nothing is listening.

There is nothing in the sync or health status to notice, and nothing alerts. It can persist for years.

Check the destination before you trust the status:

kubectl get application <name> -n argocd -o jsonpath='{.spec.destination}{"\n"}'

destination.name should be idl-xnl-jhb1-rc01 for anything serving public traffic. A destination.server of https://kubernetes.default.svc in this estate means the management cluster and should be treated as wrong until proven otherwise.

Then verify from outside. The only evidence that a deployment works is a response from the real public hostname:

curl -sSI https://<public-hostname>/<a-path-the-app-actually-serves>

This matters most when an Application is about to become a dependency of something else — an ingress delegating authentication to it, for example. Wiring a working service to an unreachable-but-green one converts a dormant misconfiguration into an outage of the working service.

An Application created by hand with kubectl apply, rather than through the app-of-apps, carries no argocd.argoproj.io/tracking-id annotation, no owner references and no finalizer. Deleting it therefore orphans its resources rather than pruning them: the workloads keep running, unmanaged and unowned, and an orphaned Ingress goes on claiming its hostname indefinitely.

11.2. Moving an Application to a New Namespace

No Application in the manifest repository carries the resources-finalizer.argocd.argoproj.io finalizer. Deleting or renaming an Application is therefore non-cascading — the workloads survive it, unowned. That single fact dictates the ordering below, and getting the ordering wrong is what leaves a hostname stuck.

Perform the move in two commits, with a verification gate between them:

  1. Move first, under the unchanged metadata.name. Edit destination.namespace, helm.releaseName, and any in-cluster service URLs in place. ArgoCD still tracks the old-namespace resources through the argocd.argoproj.io/tracking-id annotation, sees them drop out of the desired state, and prunes them itself. Wait until the old namespace is clean.

  2. Rename the file and metadata.name only then. The app-of-apps prunes the old Application non-cascadingly, and the new name adopts the still-running resources. Downtime is zero.

Doing both in one commit is the trap. The old Application is pruned before it has a chance to prune its own resources, so the old namespace’s workloads — the Ingress above all — are orphaned. The orphaned Ingress then keeps claiming the hostname with nothing left to manage or remove it.

11.2.1. Pre-copy the certificate when the hostname does not change

A same-hostname move needs one preparatory step, or the new namespace serves the site without a valid certificate.

The letsencrypt-prod ClusterIssuer uses an HTTP-01 solver, and ingress-nginx resolves a duplicate host by honouring the oldest Ingress. So while the old Ingress still exists, the new namespace’s Ingress is ignored, the ACME challenge cannot route, and the certificate will not issue on its own. Copy the live secret across before step 1:

kubectl -n <old-ns> get secret <old-release>-tls -o json \
 | jq '{apiVersion, kind, type, data, metadata:{name:"<new-release>-tls", namespace:"<new-ns>"}}' \
 | kubectl apply -f -

The chart names the secret <releaseName>-tls, so the name changes with the release name. The result is a zero-gap cutover: the new Ingress serves the copied certificate immediately, and cert-manager re-issues cleanly on its own once the old Ingress is gone.

11.2.2. What needs no change, and what is left behind

  • DNS. External-DNS points the hostname at the ingress-nginx load balancer, which is the same for every namespace on the cluster. Nothing to update.

  • Leftovers. The old <old-release>-tls secret survives in the old namespace — cert-manager garbage-collects the Certificate along with the Ingress, but not the secret it produced. Remove it once the move is verified.

12. Reference

File Role

~/dev/idl-xnl-jhb-rc01/argocd/event-admin-service-prod.yml

Full prod Application example

~/dev/idl-xnl-jhb-rc01/argocd/registration-portal-prod.yml

Portal-shaped Application example

~/dev/idl-xnl-jhb-rc01/argocd/opentelemetry-collector.yml

Infrastructure Application example

admin-service/src/main/helm/

The chart that ArgoCD templates

13. Related Documentation

14. Change History

Date Change

2026-04-24

Initial draft. Grounded in ~/dev/idl-xnl-jhb-rc01/argocd/ prod manifests (event-admin-service-prod, registration-portal-prod).

2026-09-07

Added Destination and Namespace Changes — destination-cluster verification and the two-commit namespace-move recipe with TLS pre-copy.

2026-09-07

Added Sync Latency Is 20-30 Minutes, Not Seconds, measured end to end. Corrected the promotion step, which claimed a push was picked up "within seconds".