MySQL cluster upgrade: idealogic-prod

This is the cluster-specific execution checklist. The generic mechanics, traps and rationale are in Upgrading the MySQL Operator and InnoDB Cluster server version. Read that first if you have not done this before.

1. Why this is needed

The operator was upgraded to 2.1.9 in December 2025 (d0d10fe) and to 2.1.10 in August 2026 (3b15dc5, Bug #1028), but the server has never moved. Because spec.version is unset on the CR, the server stayed at whatever the operator defaulted to when the cluster was created in January 2024.

The cluster is therefore running MySQL 8.3.0 — an Innovation release that has been end-of-life since 2024 and receives no security patches. The sidecar is further behind still, at the 2.1.2 operator image.

2. Current state and target

Item Current After upgrade

Server (mysql container)

community-server:8.3.0

community-server:8.4.8

Sidecar / init containers

community-operator:8.3.0-2.1.2

community-operator:8.4.8-2.1.10

Router

community-router:8.3.0

community-router:8.4.8

Operator

2.1.10 (already current)

unchanged

spec.instances

2

3 (scaled up as a precondition)

spec.version

unset

8.4.8

8.4.8 is the matched pair for operator 2.1.10 (chart appVersion: 8.4.8-2.1.10) — the combination Oracle tested together. Operator 2.1.10 supports servers in the range 8.0.27–8.4.99, so the running 8.3.0 is in range for the upgrade path.

3. Pre-flight

All of the following were verified on 2026-08-24. Re-check on the day.

Check Expected

community-server:8.4.8 image

present in container-registry.oracle.com

community-router:8.4.8 image

present

Longhorn capacity

~1.4 TB free per node; the third 40Gi PVC is trivial

Backups

daily 02:00, completing in ~4 min, last six consecutive runs Completed

Dataset

~2.86 GB across ~1670 tables — data-dictionary upgrade is minutes, not hours

Primary

idealogic-prod-0 (so the roll upgrades the secondary first, primary last)

Set your context first:

kubectl config use-context static/idl-xnl-jhb1-01   # or: export KUBECONFIG=<path>
kubectl -n mysql get innodbcluster idealogic-prod

3.1. Confirm the primary is still ordinal 0

The roll order only works in our favour if the primary is -0. If a failover has happened since, the primary may have moved — re-check, and expect the primary to be restarted first if it is now -2.

kubectl -n mysql exec idealogic-prod-0 -c mysql -- mysql -u localroot -N -B -e \
  "SELECT MEMBER_HOST, MEMBER_ROLE, MEMBER_STATE, MEMBER_VERSION
   FROM performance_schema.replication_group_members;"

3.2. Take an on-demand backup

The server upgrade has no rollback. Do not rely on the nightly run.

kubectl -n mysql get mysqlbackup --sort-by=.metadata.creationTimestamp | tail -3

Trigger a fresh backup and wait for Completed before continuing. See the mysql-operator-backup-restore skill.

3.3. Record the persisted variables

These must still be present afterwards:

for p in idealogic-prod-0 idealogic-prod-1; do
  kubectl -n mysql exec $p -c mysql -- mysql -u localroot -N -B -e \
    "SELECT VARIABLE_NAME, VARIABLE_VALUE FROM performance_schema.persisted_variables
     WHERE VARIABLE_NAME IN ('max_connections','max_user_connections');"
done

Expected on both: max_connections=400, max_user_connections=20.

The per-account caps live in mysql.user, not in Git. They survive a rolling restart, but would be lost by a rebuild from the CR. See MySQL InnoDB Cluster.

4. Phase A — scale to three instances

This is a separate commit with a hard gate. Do not combine it with Phase B: the roll must not begin before the new member has finished cloning and reached ONLINE.

Scaling up is not a restart — the StatefulSet only creates idealogic-prod-2; the existing pods are untouched.

  1. Edit mysql/idealogic-prod/idealogic-prod-cluster.yml, set spec.instances: 3.

  2. Commit and push to main. ArgoCD autosync applies it (mysql-idealogic-prod).

  3. Wait for the new member to clone and join.

kubectl -n mysql get pods -w    # idealogic-prod-2 appears, clones, becomes Ready

4.1. Gate: all three ONLINE

Do not proceed until this returns three rows, all ONLINE:

kubectl -n mysql exec idealogic-prod-0 -c mysql -- mysql -u localroot -N -B -e \
  "SELECT MEMBER_HOST, MEMBER_ROLE, MEMBER_STATE
   FROM performance_schema.replication_group_members;"

kubectl -n mysql get innodbcluster idealogic-prod \
  -o jsonpath='{.status.cluster.status} onlineInstances={.status.cluster.onlineInstances}{"\n"}'

Expected: ONLINE onlineInstances=3.

Also apply the persisted variables to the new member — SET PERSIST does not replicate:

SET PERSIST max_connections=400;
SET PERSIST max_user_connections=20;

5. Phase B — upgrade the server version

  1. Edit the same file, add spec.version: "8.4.8" alongside spec.instances.

  2. Commit and push to main.

  3. Watch the roll.

kubectl -n mysql get pods -w

Expected sequence: idealogic-prod-2 restarts first, then -1, then -0 last. Each pod runs the data-dictionary upgrade on first start of the new binary. The restart of -0 triggers a Group Replication failover — a brief burst of write errors in the applications.

Between each pod, wait for the member to rejoin ONLINE before the next one goes down. The StatefulSet does this automatically via readiness, but confirm rather than assume.

6. Verification

# all three members ONLINE on 8.4.8
kubectl -n mysql exec idealogic-prod-0 -c mysql -- mysql -u localroot -N -B -e \
  "SELECT MEMBER_HOST, MEMBER_ROLE, MEMBER_STATE, MEMBER_VERSION
   FROM performance_schema.replication_group_members;"

# images rolled — mysql AND sidecar
kubectl -n mysql get sts idealogic-prod \
  -o jsonpath='{range .spec.template.spec.containers[*]}{.name}={.image}{"\n"}{end}'

# routers rolled
kubectl -n mysql get deploy idealogic-prod-router \
  -o jsonpath='{.spec.template.spec.containers[0].image}{"\n"}'

# persisted variables intact on every member
for p in idealogic-prod-0 idealogic-prod-1 idealogic-prod-2; do
  kubectl -n mysql exec $p -c mysql -- mysql -u localroot -N -B -e \
    "SELECT VARIABLE_NAME, VARIABLE_VALUE FROM performance_schema.persisted_variables;"
done

Then:

  • ArgoCD mysql-idealogic-prod reports Synced / Healthy with no conditions.

  • Application health — check the admin services and the WordPress sites. WordPress is the canary: it opens a connection per request, so it shows errors first and recovers first.

  • Error rates in Grafana for the switchover window.

7. Rollback

There is none for the server upgrade. Once mysqld has upgraded the data dictionary, the 8.3.0 binary will refuse to start against that datadir. Recovery means restoring the pre-upgrade backup into a fresh cluster.

This is why the on-demand backup in pre-flight is not optional.

8. After the upgrade

  • Leave spec.version pinned. It is now recorded in Git, which closes the gap where the server version existed nowhere in source control.

  • Decide on spec.instances. Three members give real fault tolerance and are worth keeping. If you scale back to 2, do it as a deliberate, separate change.

  • Router replica drift. The CR declares router.instances: 2 but the live Deployment runs 3, from a manual scale in the past. Setting the CR to 2 will not reconcile it — the field handler is change-triggered and the value is already 2. Either scale the Deployment directly or toggle the CR value. Worth folding into this window.

9. Execution record: 2026-08-24

Executed as a single session, ~23:00–23:35 SAST, on a Monday night at low registration volume.

Step Outcome

Pre-flight

Cluster ONLINE/2, primary -0, server 8.3.0, sidecar 8.3.0-2.1.2

On-demand backup

idealogic-prod-preupgrade-260824 — Completed in 04:03, taken from the secondary

Phase A (scale 2→3)

idealogic-prod-2 joined in ~3 min. Existing pods untouched (ages 52d/50d, 0 restarts)

Phase A gate

3 ONLINE, applier queue 0, both secondaries GTID-converged, verified stable over repeated samples

Phase B (roll)

~11 min total, ~3–4 min per pod, order -2 → -1 → -0

Result

All three ONLINE on 8.4.8; router 8.4.8 (3/3); ArgoCD Synced/Healthy

Application impact

Zero pod restarts across all seven DB-consuming services and WordPress

9.1. Two things worth carrying forward

The new member came up on compiled defaults. idealogic-prod-2 joined with max_connections=151 and max_user_connections=0, because SET PERSIST does not replicate and 99-extra.cnf carries only sql_mode. It needed SET PERSIST applied explicitly before the roll. Had this gone unnoticed, the cluster would have carried a member configured for 151 connections that was then promoted to primary by the roll — reproducing the 2026-07-29 exhaustion outage. Always apply the persisted variables to a newly added member as part of Phase A.

The clone path was used, not incremental recovery. The operator first tried add_instance with recoveryMethod=incremental, failed with "GTID state is not compatible" because the binlogs have long been purged, and fell back to clone automatically. The accompanying probe failures and the single pod restart during Phase A are part of that normal fallback, not a fault.

9.2. Post-upgrade state

The primary is now idealogic-prod-2, having moved during the roll when -0 restarted. The pre-flight instruction to confirm which ordinal is primary matters for exactly this reason — do not assume it is still ordinal 0 next time.

VARIABLE_SOURCE for both connection variables now reads PERSISTED rather than DYNAMIC, confirming the values were genuinely loaded from mysqld-auto.cnf on startup rather than merely set at runtime.

Still outstanding: the router Deployment remains at 3 replicas against a CR that declares 2.

10. Pending: podSpec takes effect at the NEXT roll

spec.podSpec was added on 2026-08-24 carrying pod anti-affinity and resource requests, but podSpec is not in the operator’s watched-field list — so it is inert until the StatefulSet is next re-rendered, which happens on the next spec.version change.

Whoever runs the next upgrade should verify it actually landed, as part of post-upgrade checks:

# affinity should now be present (it was absent before)
kubectl -n mysql get sts idealogic-prod -o jsonpath='{.spec.template.spec.affinity}{"\n"}'

# QoS should be Burstable, not BestEffort
kubectl -n mysql get pods -l component=mysqld \
  -o custom-columns='POD:.metadata.name,NODE:.spec.nodeName,QOS:.status.qosClass' --no-headers

Expect Burstable and one pod per node. Until then the members remain BestEffort and are spread across nodes only by scheduler chance.

Two related observations worth acting on separately:

  • innodb_buffer_pool_size is still at the MySQL default of 128Mi against a 2.86 GB dataset. Raising it is likely the single largest available performance win, but it is a SET PERSIST / mycnf change with the usual caveats — and if a memory limit is ever added to podSpec, it must be raised in step or mysqld will be OOMKilled.

  • Longhorn actualSize on the two original datadirs reads 41–43Gi against 40Gi volumes. This is not bloat: filesystem usage is 18G/40G. Longhorn counts blocks ever written and never reclaims them, and the 18G is mostly 30 days of binlogs (binlog_expire_logs_seconds=2592000). The fresh member sits at 4.8G and will converge on the same figure.