Runbook: Upgrading the MySQL Operator and InnoDB Cluster server version

This guide covers the generic mechanics of upgrading an Oracle MySQL Operator deployment and the server version of the clusters it manages. For the cluster-specific execution checklist — exact values, commands and gates for idealogic-prod — see MySQL cluster upgrade: idealogic-prod, which lives in the GitOps repository alongside the manifests.

Cluster access is covered by the mysql-idealogic-prod skill; backup and restore by the mysql-operator-backup-restore skill, both in ~/dev/ai-skills-infra/skills/.

1. Two version axes, not one

The single most common mistake is treating this as one upgrade. There are two independent versions:

Axis Controlled by Changing it

Operator version

targetRevision on the ArgoCD Application (Helm chart version)

Rolls the mysql-operator Deployment only. Does not touch database pods.

Server version

spec.version on the InnoDBCluster custom resource

Rolls every mysqld instance. This is the risky one.

Upgrading the operator does not upgrade the servers it manages. A cluster can sit for years on an old server version while the operator moves forward — and it will, silently, because the operator only renders the StatefulSet’s server image at cluster creation and whenever spec.version changes.

If spec.version is unset on the CR, the server version is frozen at whatever the operator defaulted to when the cluster was first created. It is recorded nowhere in Git. Check the running server directly, never the manifest. 
kubectl -n <ns> get sts <cluster> \
  -o jsonpath='{range .spec.template.spec.containers[*]}{.name}={.image}{"\n"}{end}'

2. Before you start: verify the chart AND the image exist

Oracle publishes only one version at a time in its Helm index. Older versions are deleted outright, so any targetRevision pinned directly at https://mysql.github.io/mysql-operator will eventually fail to resolve. When it does, ArgoCD reports sync=Unknown with a ComparisonError while still showing health=Healthy — the fetch fails before comparison, so autosync is silently inert.

The fix is to mirror charts into a registry we control. See ArgoCD Deployment Patterns.

The chart source survives in upstream Git tags even after the index drops it:

curl -sL https://github.com/mysql/mysql-operator/archive/refs/tags/<appVer>-<chartVer>.tar.gz | tar xz
helm package <dir>/helm/mysql-operator
helm push mysql-operator-<chartVer>.tgz oci://registry-1.docker.io/christhonie

A Git tag does not guarantee a published container image. Upstream tags releases that Oracle never ships images for — 8.4.9-2.1.11 is one. Mirroring that chart produces a clean ArgoCD sync followed by ImagePullBackOff.

Unlike the Helm index, the image registry is not pruned. Always confirm the image before pinning:

T=$(curl -s 'https://container-registry.oracle.com/auth?service=Oracle%20Registry&scope=repository:mysql/community-operator:pull' | jq -r .token)
curl -sI -H "Authorization: Bearer $T" \
  https://container-registry.oracle.com/v2/mysql/community-operator/manifests/<appVersion>

The auth realm is /auth with service=Oracle Registry; the conventional /v2/token endpoints return an empty token.

Check community-server and community-router for the target server version too — all three images must exist before you commit anything.

3. Choosing a target version

Stay on the LTS line. Operator 2.1.x tracks MySQL 8.4 LTS; 2.2.x tracks MySQL 9.x and 2.3.x tracks MySQL 26.7, both Innovation-track releases with much shorter support windows.

Pick the matched pair. Operator and server are released together and the chart’s appVersion encodes both — 8.4.8-2.1.10 means server 8.4.8 with operator 2.1.10. Setting spec.version to the operator’s own DEFAULT_VERSION_TAG keeps you on the combination Oracle tested:

grep -E '^(DEFAULT_VERSION_TAG|MIN_SUPPORTED_MYSQL_VERSION|MAX_SUPPORTED_MYSQL_VERSION)' \
  <src>/mysqloperator/controller/config.py

An operator refuses to manage a server outside MIN_SUPPORTED..MAX_SUPPORTED, so check the running server is in range for the operator version you are moving to.

4. Preconditions for a server upgrade

4.1. Scale to three instances first

A rolling restart of a two-member group leaves the cluster on a single member for the duration of each pod restart — no fault tolerance at precisely the moment it is most likely to be needed. If the survivor stumbles mid-roll the result is an outage plus possible manual quorum recovery (forceQuorumUsingPartitionOf).

Scale spec.instances to 3 and wait for the new member to reach ONLINE before touching spec.version. This must be a separate commit with a verification gate — applying both at once lets the roll begin before the new member has finished cloning.

Scaling up is not a restart: the StatefulSet only creates the new ordinal; existing pods are untouched.

4.2. Take an on-demand backup

The data-dictionary upgrade is one-way. There is no downgrade once mysqld starts on the new binary, so a proven-restorable backup is the only rollback. Do not rely on the nightly schedule — take a fresh one and confirm it reaches Completed.

4.3. Record any non-persisted server variables

Variables set with SET GLOBAL do not survive a restart. Confirm anything that matters is actually persisted, not merely running:

SELECT VARIABLE_NAME, VARIABLE_VALUE FROM performance_schema.persisted_variables;
VARIABLE_SOURCE in performance_schema.variables_info describes how the running value was set, so it reads DYNAMIC even after a successful SET PERSIST — it only flips to PERSISTED once a restart reloads the file. persisted_variables is the authoritative check. Note also that SET PERSIST is per-instance and does not replicate: run it on every member.

5. What actually happens during the roll

Setting spec.version fires the operator’s spec.version field handler, which calls update_mysql_image(). That patches the StatefulSet pod template — the mysql container to the new server image, and the sidecar plus all init containers to the current operator image. On a cluster whose sidecar has drifted behind, this also silently brings the sidecar forward.

The resulting rollout has these properties:

Property Behaviour

updateStrategy

RollingUpdate — one pod at a time, ordinal-descending, waiting for Ready between each

podManagementPolicy: Parallel

Affects scaling only. Rolling updates remain strictly sequential.

Order

Highest ordinal first. Confirm the primary is ordinal 0 so secondaries upgrade first and the primary last — MySQL’s recommended order.

Primary restart

Triggers a Group Replication failover. In-flight writes fail; the router redirects new connections.

Routers

Also rolled, via a separate Deployment. With multiple replicas this layer is transparent.

Check the roles before you start — do not assume ordinal 0 is the primary:

SELECT MEMBER_HOST, MEMBER_ROLE, MEMBER_STATE, MEMBER_VERSION
FROM performance_schema.replication_group_members;

Each mysqld runs the data-dictionary upgrade on first start of the new binary. Duration scales with table count rather than raw size, so check information_schema.tables for a realistic estimate.

6. Application impact

Applications reaching the cluster through MySQL Router see a failover, not an outage — but "rolling" is not "zero-impact":

  • New connections are transparent. Connection pools validate on borrow, evict the dead connection and reconnect through the router to the new primary.

  • In-flight statements fail with a connection exception that reaches application code. For Connector/J this is CommunicationsException (SQLState 08S01), which extends SQLRecoverableException. Nothing retries it automatically; open transactions roll back.

Expect a short burst of errors confined to requests executing SQL during the switchover. Schedule accordingly. WordPress sites are the usual canary — they open a connection per request, so they show errors first and recover first.

7. Verification after the roll

# every member back ONLINE and on the new version
kubectl -n <ns> exec <pod> -c mysql -- mysql -u localroot -N -B -e \
  "SELECT MEMBER_HOST, MEMBER_ROLE, MEMBER_STATE, MEMBER_VERSION
   FROM performance_schema.replication_group_members;"

# images actually rolled (mysql AND sidecar)
kubectl -n <ns> get sts <cluster> \
  -o jsonpath='{range .spec.template.spec.containers[*]}{.name}={.image}{"\n"}{end}'

# persisted variables survived
kubectl -n <ns> exec <pod> -c mysql -- mysql -u localroot -N -B -e \
  "SELECT VARIABLE_NAME, VARIABLE_VALUE FROM performance_schema.persisted_variables;"

Then confirm the ArgoCD Application reports Synced/Healthy with no conditions, and check application health and error rates in Grafana for the switchover window.

8. Rollback

There is none for the server upgrade. Once the data dictionary is upgraded, the old binary will refuse to start against that datadir. Recovery means restoring the pre-upgrade backup into a fresh cluster — see the mysql-operator-backup-restore skill.

The operator chart, by contrast, rolls back cleanly: revert targetRevision and let ArgoCD sync.

That is only true while the version you are reverting to is still fetchable. Reverting targetRevision to a chart the vendor has since pruned from its index leaves the Application unable to render at all — see the next section.

9. Recovering a chart version pruned from the upstream index

Oracle keeps exactly one mysql-operator and one mysql-innodbcluster entry in its published Helm index. Superseded versions are deleted outright: the index entry disappears and the .tgz returns 404. Any targetRevision pinned directly at that index therefore rots on the vendor’s release schedule, not on ours, and it takes both the upgrade path and the rollback path with it.

The failure is quiet — the Application keeps reporting health=Healthy while sync=Unknown, because the chart fetch fails before any comparison happens and autosync is silently inert. The reason appears only in .status.conditions as a ComparisonError; no metric exposes it. See ArgoCD Deployment Patterns for the diagnosis commands.

9.1. The asymmetry that makes recovery possible

Source Pruned? Consequence

Published Helm index

Yes — one version at a time

A pinned version becomes unfetchable without warning.

Upstream Git tags

No

Full chart source survives under helm/mysql-operator at every tag. Tag format is <appVersion>-<chartVersion>, for example 8.4.8-2.1.10.

Oracle container registry

No

Image tags are retained across many releases, well behind the current one.

So a withdrawn chart is always reconstructible, and the images it references are still pullable.

9.2. Mirror rather than re-pin

Re-pinning to whatever the index currently advertises solves the symptom and reintroduces the cause — often while dragging the estate off the LTS line onto an Innovation-track release. The durable fix is to mirror the chart into the OCI registry we control and point targetRevision there. No ArgoCD Application in the GitOps repository should source a chart from a third-party Helm index.

Version-preserving mirror (zero restart). While the version is still fetchable, helm pull the exact pinned version and helm push that tarball unmodified. Do not repackage from source: a rebuilt tarball differs byte-wise, which can change the render and restart workloads for no reason.

helm pull --version <pinned-version> --repo <upstream-index-url> <chart-name>
helm push <chart-name>-<pinned-version>.tgz oci://registry-1.docker.io/christhonie

Then prove the mirror is byte-identical before switching the Application over — pull it back and compare the extracted trees against the upstream tarball:

helm pull oci://registry-1.docker.io/christhonie/<chart-name> --version <pinned-version> --untar --untardir /tmp/mirror
tar xzf <chart-name>-<pinned-version>.tgz -C /tmp/upstream
diff -r /tmp/upstream/<chart-name> /tmp/mirror/<chart-name>

An identical artefact renders identically, so ArgoCD sees no diff and nothing restarts. Confirm the Application returns to Synced/Healthy with every pod age unchanged.

Check dependency bundling first. A chart that resolves dependencies at render time behaves differently once mirrored. Compare a dependencies: key in Chart.yaml against what the tarball actually contains:

tar tzf <chart-name>-<pinned-version>.tgz | grep 'charts/.*/Chart.yaml'

If the version is already gone, rebuild it from the matching Git tag using the recipe in the pre-flight section above, then push that package to the mirror. The result is functionally correct but not byte-identical to the original artefact, so expect a render diff and plan for a roll.

9.3. The trap: a chart tag without a published image

A Git tag is not a release. Upstream tags chart versions for which the container image was never published — 8.4.9-2.1.11 is one. Mirroring such a chart produces a clean, green ArgoCD sync followed by ImagePullBackOff, which is a worse outcome than the failure it was meant to fix because the Application now looks correct.

Always confirm the image exists before packaging or pinning, using the registry check in the pre-flight section above. On the 8.4 LTS line, 2.1.10 is the newest chart version with a real published image.

10. Metrics export

Metrics are enabled on the InnoDBCluster CR, not by deploying an exporter separately:

spec:
  metrics:
    enable: true
    monitor: true
    image: prom/mysqld-exporter:v0.14.0

The operator adds a metrics sidecar on port 9104, creates a ServiceMonitor in the cluster’s namespace, and creates and manages the mysqlmetrics database user itself — PROCESS, REPLICATION CLIENT and SELECT, with max_user_connections=3. No manual grants are needed and none should be added.

Scraping needs no labelling. The monitoring stack’s Prometheus has both serviceMonitorSelector and serviceMonitorNamespaceSelector set to {}, which means every ServiceMonitor in every namespace is discovered.

10.1. Pin the exporter to v0.14.0

The exporter image must be pinned to prom/mysqld-exporter:v0.14.0. Any version from v0.15.0 onward reports mysql_up 0 and collects nothing.

The operator writes an invalid socket path into the generated <cluster>-metricsconf ConfigMap:

[client]
user=mysqlmetrics
socket=unix:///var/run/mysqld/mysql.sock

unix:// is a Go DSN scheme, not valid in a my.cnf socket= value, so the exporter tries to dial a path that does not exist. The socket itself is fine.

The operator also sets DATA_SOURCE_NAME, and that value is correct. v0.14.0 reads it and works. From v0.15.0 the exporter dropped support for the monolithic DATA_SOURCE_NAME environment variable, leaving the broken my.cnf as its only input — which is why the version boundary is exactly there.

Upgrading the operator does not fix this. The same string is still emitted by much later operator releases. Do not treat a newer operator as a reason to unpin.

The upstream signal is the operator’s own test default, OPERATOR_TEST_METRICS_IMAGE_NAME, which is v0.14.0. Its list of tested images also contains v0.15.x releases; the default is the signal, the list is not.

10.2. Changing spec.metrics rolls the cluster

spec.metrics is a watched field. Changing it adds or alters a container and stamps restartedAt on the StatefulSet, which rolls every mysqld pod — roughly six minutes for a three-member cluster. Treat it as a rolling restart and apply the same care as a server upgrade: check roles first, expect a failover when the primary restarts, and see Observability Access for watching the switchover.

The handler patches the live StatefulSet rather than regenerating it, so it does not pick up unrelated drift. Changes to spec.podSpec — anti-affinity, resource requests — still require a spec.version change to take effect.

10.3. Dashboards and folders

Grafana dashboards live in observability/dashboards/ in the GitOps repository and are deployed by the grafana-dashboards ArgoCD Application. Each dashboard is a ConfigMap in the observability namespace carrying two pieces of metadata:

Metadata Effect

Label grafana_dashboard: '1'

Makes the Grafana sidecar pick the ConfigMap up at all.

Annotation grafana_folder: <Folder>

Places the dashboard in that folder. The sidecar creates the folder on demand, so a new folder needs no prior setup — naming it in the annotation is enough.

The mysql-innodb-cluster dashboard covers connections against max_connections and buffer-pool hit rate and pages-by-state. Connection capacity, its alert rules and the limits they watch are covered by MySQL Connection Capacity.

Custom Prometheus alert rules follow the same one-Application-per-directory pattern from observability/prometheus-rules/, and are likewise discovered without labels because ruleSelector and ruleNamespaceSelector are both {}.

Validate a rule’s expression against live data before pushing it. A rule that parses and loads is not a rule that matches anything — invert the threshold once so you can see the series actually return, then confirm health=ok on the rules endpoint after deployment. Loaded is not the same as evaluating.

11. Known traps

  1. Chart withdrawn from the upstream Helm index — mirror charts, do not pin upstream.

  2. Git tag exists but no container image — verify the image, not just the chart.

  3. spec.version unset means the server version is invisible in Git and frozen at creation.

  4. spec.mycnf is not a watched field. Editing it is inert on a running cluster: the operator renders it into the config ConfigMap only at initial provisioning. Values recorded there apply to rebuilt or DR-restored clusters, not to the live one.

  5. spec.router.instances is a change-triggered handler. If the live Deployment has drifted, setting the CR to the value it already holds does nothing — toggle it, or scale the Deployment directly.

  6. SET PERSIST is per-instance and does not replicate.

12. References