Dev site and promotion

The problem immutability creates

Immutability buys security, reproducibility and infrastructure-as-code, and it takes away the place where a site is normally built. With DISALLOW_FILE_MODS set there is nowhere to trial a plugin, discover the right WooCommerce setting, or lay out a page before it has to be correct in front of customers.

The answer is not to relax production. It is a second posture of the same chart and the same image, plus a repeatable way to turn what was discovered there back into the declarative artefacts production is built from.

Two postures, one chart

mode selects the posture. It is a runtime posture, not an environment.

mode: immutable mode: mutable

Where

Production

The dev site — locally and in-cluster

WP_ENV

production → DISALLOW_FILE_MODS on

development → DISALLOW_FILE_MODS off

wp-content/plugins, themes

Baked into the image, read-only

PVCs, seeded from the image on every start

wp/ (core)

Read-only

Read-only. Core is pinned by composer.lock in both postures.

Declared options

Re-applied on every start

Applied only when the declared config itself changes

Two of those rows are the design, not details.

Core stays pinned even in dev

Only plugins and themes become writable. Core does not. A core upgrade goes through composer require roots/wordpress:^X and an image rebuild, which is reproducible and reviewable; clicking Update in wp-admin would desynchronise composer.lock from the running site with nothing to show for it. There is discovery value in installing a plugin to see whether it fits. There is none in pressing an update button.

The mutable posture must not re-apply declared config on every start

In production, re-applying the declared options on every start is the point: the site cannot drift. Doing the same in dev would revert the very setting you went in to change, on the next pod restart, before it could be harvested.

So the mutable posture applies declared config only when the declared config itself has changed. The chart hashes the theme, plugin list and options into DECLARED_CONFIG_HASH; the bootstrap compares it to the hash recorded beside the baseline. Equal means "nothing was promoted since last time" — leave the dev site alone. Different means a promotion landed — re-apply, and re-baseline.

The seed overlay

A PVC mounted at wp-content/plugins hides whatever the image put there. So the mutable image (--target runtime-dev) keeps a pristine copy outside the webroot at /opt/wp-seed, and the bootstrap overlays it onto the PVCs on every start:

cp -r /opt/wp-seed/plugins/. web/app/plugins/
cp -r /opt/wp-seed/themes/.  web/app/themes/
Not cp -a, and no --preserve

cp -a implies --preserve=ownership. The bootstrap runs as non-root www-data, and chown, chmod and utime on a file it does not own all return Operation not permitted — under set -e, any one of them kills the init container. --preserve=timestamps alone fails the same way.

This is not hypothetical: it is what cp -a does the first time the local dev site seeds into bind-mounted trees owned by the host user. Nothing in a plugin tree depends on preserved mode or mtime, so preserve nothing.

Locally the same ownership mismatch means the bind-mounted trees must be made group-writable before the container can seed or install into them.

Overlay, never mirror. The image wins per file, so a rebuilt image’s plugin upgrade lands; nothing is deleted, so a plugin installed through wp-admin survives a restart. The useful consequence: the difference between the PVC and the image is exactly what dev has and production does not — which is what the harvest reads.

A mode: mutable release running an image built without --target runtime-dev has no /opt/wp-seed. The PVCs mount over the baked plugins and themes, the overlay finds nothing to copy, and the site comes up looking empty with no error. The bootstrap warns; heed it.

Namespaces, and how this scales to several sites

The namespace boundary is the posture, not the site. Each WordPress tenant is a release inside it, exactly as event-dev holds several services rather than one namespace per service.

wordpress wordpress-dev

Holds

Every tenant’s production site

A dev variant of whichever tenants currently need one

Per site

One Helm release, its own fullnameOverride, its own schema

The same, plus its own basic-auth credential

Set up once

—

Registry pull secret, RBAC, and whatever network policy the namespace carries

So adding a second tenant does not add a namespace. It adds a release in wordpress, and — only while someone is actually building or changing that site — a release in wordpress-dev.

A dev site is a tool, not a fixture

Nothing requires a standing dev variant per tenant. Most tenants will not have one most of the time. Create it for a build-out or a round of changes, harvest what it taught you, and delete the release.

Deleting the release does not delete its data: the PVCs carry helm.sh/resource-policy: keep, deliberately, so that an ArgoCD prune or a mistaken sync cannot destroy plugin work that has not been harvested yet. That protection is also why the storage does not disappear on its own — clean it up explicitly when the site is genuinely finished with:

kubectl -n wordpress-dev delete pvc \
  <release>-uploads <release>-updraft <release>-plugins <release>-themes

The code PVCs are sized small (2Gi) precisely because they hold a plugin tree of a few hundred megabytes, and every dev site in the namespace pays for them.

Why not one namespace per tenant

It is a defensible alternative, and there is precedent for it in this estate. It buys tenant-to-tenant isolation: today every site in wordpress shares a secret scope, so a single RBAC grant covers all of them.

That is worth having if the tenants are genuinely separate parties. It is not what the prod/dev split is for, and the two decisions are independent — production already makes this trade, and the dev namespace simply mirrors whatever production does. If tenant isolation is ever needed, both sides move together to wp-<tenant> / wp-<tenant>-dev, and nothing in this design resists that.

What the prod/dev boundary buys is different, and is worth keeping either way: a dev site runs a deliberately weaker posture — writable code, WP_DEBUG on, wp-admin installs allowed — and after a content refresh it holds real customer data. Sharing a namespace with production would mean one RBAC grant spanning both.

Three lanes

Code, configuration and content move by different mechanisms, in different directions, with different degrees of automation. Conflating them is the usual way a WordPress promotion goes wrong.

Lane Contents Direction Mechanism

Code

core, plugins, themes

dev → prod

Harvest → composer.lock / packages/ → image rebuild

Config

wp_options, active plugin set, active theme

dev → prod

Harvest → the chart’s bootstrap.* values → bootstrap applies

Content

posts, pages, layouts, media

both, asymmetric

UpdraftPlus Migrator (see plugins/updraftplus.adoc#content-lane)

Code

Whatever is installed in the dev site is compared against three sources of truth: composer.lock, packages/, and the chart’s activation list. Anything in dev and in none of them was installed by hand and needs promoting. Each is then classified against wordpress.org — public ones become a Composer requirement, private ones are staged out of the running dev site as a tarball for packages/.

A private artefact staged this way came out of a site somebody was clicking around in. Check its provenance and licence before it is baked into a production image. The harvest flags it; it does not decide for you.

Config

The bootstrap records a baseline — every option, dumped after the declared state was applied — when the dev site is deployed. The harvest diffs the current options against that baseline, which answers precisely "what did I change since this site was deployed?" rather than the far noisier "how does dev differ from production?".

Survivors pass two filters:

  • harvest/option-denylist.txt — noise (transients, cron, caches, update timestamps) and decisions you have already made. Appending to it is how the list becomes quiet over time.

  • A secret filter that no file can switch off. It matches credential-shaped option names, and credential-shaped sub-keys inside serialized values — because the dangerous case is the one that does not look dangerous. woocommerce_paygate_settings is an innocuous name holding a merchant key.

The secret filter searches an option’s nested keys, recursively, not its text — searching the text would fire on any prose containing the word "key". It errs deliberately towards over-blocking: wrongly withholding a harmless setting costs one line in the denylist, while wrongly promoting a credential puts it in git history.

Credentials are never promoted through a git-tracked values file. They reach the site as environment variables from the Secret, the way the EPA_* settings already do.

The denylist is checked before the secret filter. Both outcomes are "not promoted", so the order cannot leak anything — it only decides which bucket a key is reported in. Labelling known noise like cron as a credential would make the credential list untrustworthy, and a list nobody trusts is a list nobody reads.

Content

Content is the one lane with no declarative form worth building. It moves through UpdraftPlus Migrator, and the two directions are not symmetric — see the UpdraftPlus page for the rules and the reason a forward database restore must never happen.

After a production refresh, scrub

Pulling production content into dev copies real customers, real orders and the live payment and API settings. Until the site has been scrubbed it is a second copy of production’s personal data wired to production’s integrations.

The scrub de-indexes the site, disables every payment gateway, repoints the event payment plugin away from production, resets user passwords and clears the inherited cron schedule. It refuses to run against a site whose siteurl does not look like a dev host, because it is destructive by design.

Basic-auth on the ingress and blog_public = 0 are the standing protections; the scrub is the one that runs per refresh. Both are needed.

What the harvest deliberately does not do

  • It does not commit. The output is a triage file and a git diff. Reviewing that diff is the promotion gate — there is no automatic path from a click in wp-admin to production.

  • It does not delete. An option removed in dev is reported, never applied. Removing a key from the declared set does not remove it from production.

  • It does not rewrite nested values unattended. Scalars are edited in place between markers in the values file; structured values are proposed as a patch for a human to apply.

  • It does not promote credentials, gateway settings, or install identity (siteurl, home, admin_email) — those are per-environment by nature.

Related