Liquibase Migration Traps

1. Purpose

Authoring conventions for a changelog — filenames, changeSet ids, the master changelog, the standard operation snippets — are covered by the liquibase-changelog-create skill in the workspace skills/ directory. This page covers what goes wrong after a changelog is written and shipped, and what to do about it.

All three traps below share a property with the wider catalogue in Recurring Defect Classes: the application starts, reports healthy, and serves traffic while the schema is wrong.

2. A half-applied changeset blocks every later migration

2.1. Mechanism

MySQL DDL is not transactional. A changeset’s ALTER TABLE commits the moment it executes; Liquibase writes the corresponding tracking row afterwards, as a separate statement.

If the process dies between those two points — a pod eviction, an OOM kill, a rollout during startup — the schema object exists while Liquibase still believes the changeset is pending. Every subsequent start retries it, the statement fails on a duplicate object, and the changelog iterator halts the entire run at that point.

The halt is not limited to the failing changeset. Every changeset after it in the master changelog is never applied, on every start, indefinitely. The gap widens with each release while the application continues to report healthy.

2.2. Why it is invisible

The migration runs on a background thread under the asynchronous Liquibase wrapper inherited from the generator’s configuration. The failure lands on that thread, logs once at ERROR, and the application still completes startup and passes its readiness probe. No probe fails. No alert fires. Nothing in the deployment pipeline notices.

An environment can therefore sit weeks and many changelogs behind current, with a green deployment history, until some unrelated feature happens to depend on a column that was never created.

2.3. Detecting it

Three checks, in increasing cost:

  1. Query the logs for the repeating migration failure. One hit per process start, forever:

    # LogQL, via the observability stack
{app="event-admin-service", namespace="event-dev"} |= "Duplicate"
# or, more generally
{app="event-admin-service", namespace="event-dev"} |= "MigrationFailedException"

  2. Compare the tracking table against the master changelog. Take the tail of the changelog tracking table and line it up against the <include> order in database/src/main/resources/liquibase/wpca-database.xml. The gap is everything after the last recorded row.

  3. Cross-check a healthy environment’s schema for the object the failing changeset creates. Its presence in the broken environment, absent from the tracking table, confirms the diagnosis.

2.4. Clearing it

Drop the orphaned object. Do not hand-insert a tracking row, and never edit the failed changeset.

Dropping the object lets Liquibase replay the changeset normally, recompute its own checksum, and converge the schema with the healthy environments. Editing the changeset changes its checksum — and it is already applied elsewhere, so the edit breaks every environment where it succeeded.

 
-- Example: the changeset created a foreign key, then the pod died before the tracking row landed
ALTER TABLE example_child DROP FOREIGN KEY fk_example_child_parent;

Before clearing, audit what is queued behind the blockage. Changesets carrying <preConditions onFail="MARK_RAN"> self-heal on replay; ones without will fail on any object that drifted into the schema independently — turning one blockage into the next. See When a precondition is warranted.

2.5. Verifying the fix

The drop alone changes nothing. The replay happens only on the next process start. After the drop, the schema looks fixed (the object is gone) while the tracking table has not moved and the downstream errors keep streaming from the still-running process. This is easy to misread as "still broken".

Check process age and restart count, not just the database. A long-lived pod with zero restarts means the drop landed after the last start and nothing has replayed yet:

kubectl get pods -n event-dev | grep admin-service
kubectl rollout restart deploy/dev-event-admin-service -n event-dev

Then confirm the replay with three independent checks, because any one alone can mislead:

  1. The maximum ORDEREXECUTED in the tracking table has moved.

  2. The new process logs no migration failure at startup.

  3. The error rate actually reaches zero.

The third needs care. A rate computed over a window decays gradually after the old process dies, so an intermediate non-zero reading is a trailing-window artefact rather than residual failure. Wait a full window out before concluding either way.

3. Fixture loads fail under MySQL strict mode

3.1. Mechanism

The generator’s conventions strip the default from every datetime column with <dropDefaultValue>. A loadData or loadUpdateData fixture that omits such a column then raises:

1364 Field 'x' doesn't have a default value

— but only under a STRICT sql_mode.

That conditionality is the whole trap. The application runs Liquibase over a MySQL Connector/J session, whose sql_mode is STRICT, so the fixture breaks the application at startup. A hand-run insert through a non-strict mysql CLI session succeeds silently, which masks the bug for anyone testing that way.

The error fires for omitted columns only. An explicitly supplied NULL is accepted, so a column with no meaningful fixture value can simply be listed with an empty value rather than left out.

3.2. The rule

Before shipping a fixture, list the dropped-default columns for every table the fixture loads, and ensure each is either included in the load or absent from the dev schema:

grep -rhoE 'dropDefaultValue tableName="[^"]+" columnName="[^"]+"' changelog/*.xml

3.3. Verifying a fixture locally

A local MySQL install is not required — a throwaway container is enough, provided it matches the real session closely enough to reproduce the failure:

  • Start MySQL 8 with --sql-mode=NO_ENGINE_SUBSTITUTION. This matches the dev server. MySQL 8’s default STRICT mode is too strict for this harness: it rejects the inherited WordPress zero-date datetime defaults, so the run fails for an unrelated reason before reaching the fixture.

  • The Liquibase searchPath must cover all three resource roots: the service’s own resources, the database module’s resources (copying email-templates under liquibase/), and the WordPress schema module’s resources.

  • Mount a MySQL JDBC driver into the CLI’s lib directory.

  • Run with contexts dev,faker,sample-data.

The Liquibase CLI strict-validates changelog XSDs; the embedded Spring Liquibase the application uses does not. The CLI therefore rejects usePreparedStatements, which the real changelogs may carry legitimately.

Strip that attribute from the harness copies only. Never change the real files to satisfy the harness — the difference is in the validator, not in the changelog.

Separately, usePreparedStatements is valid on <loadData> and <insert> but not on <loadUpdateData>; adding it there fails XSD validation at application startup, which is a real defect rather than a harness artefact.

Also watch the schemaLocation: it must include the /dbchangelog/ path segment, as in …​/ns/dbchangelog/dbchangelog-3.8.xsd. Dropping that segment is an easy typo and produces a validation error that names the schema rather than the mistake.

The fixture set itself is described in Sample Data.

4. When a precondition is warranted

A defensive <preConditions onFail="MARK_RAN"> is not free: it adds a second thing to keep correct, and a precondition that tests the wrong predicate silently marks a needed change as run. Add one where it earns its place, not by default.

4.1. Why a same-file sibling needs no guard

Liquibase runs changesets sequentially within a file and halts on the first failure. If the prior changeset — the createTable, the addColumn — fails, every subsequent changeset that depends on it never runs. There is no partial state from the same file to defend against. Liquibase’s own tracking table separately prevents a successful changeset from re-running.

The only scenario a precondition guards, in that position, is someone having applied this exact thing out of band. That is a real but narrow case, worth weighing per changeset rather than assuming.

4.2. The rule, by change shape

Change Precondition? Reason

addUniqueConstraint or createIndex on a brand-new table created earlier in the same file

Usually no

The sibling either succeeded or halted the run; there is no in-between state.

addColumn on a pre-existing table

Yes

The column may have been added out of band — a hotfix, a manual patch, a hand-applied migration in one environment.

Schema-drift cleanup

Yes

Handling out-of-band patches is the changeset’s entire purpose.

Data migration — a raw <sql> UPDATE

Usually no

Idempotency comes from the WHERE clause instead. See below.

4.3. Idempotency for data migrations

Gate the WHERE clause on a recognisable starting state, so a second run matches nothing:

-- Only rows still carrying the old default are reclassified;
-- re-running this matches zero rows rather than re-applying.
UPDATE event_setting
SET preferred_timing_identifier = 'RACE_NUMBER'
WHERE preferred_timing_identifier = 'PERSON_ID';

This is stronger than a precondition, because it stays correct when the changeset is replayed for a reason the precondition’s author did not anticipate.

4.4. The second reason to weigh a precondition

A changeset carrying onFail="MARK_RAN" self-heals when a blocked migration chain is cleared and the queue behind it replays — see A half-applied changeset blocks every later migration. One without it will fail against any object that drifted in independently, and the chain blocks again at the next changeset.

That is a genuine argument for a guard on anything touching a pre-existing object. It is not an argument for guarding everything: a guard on a same-file sibling still cannot fire, and a guard on a data migration still tests the wrong thing.

5. Related Documentation

  • Recurring Defect Classes — the wider catalogue of failures that report success.

  • Sample Data — the fixture set, its identity ranges, and its coexistence rules with the faker data.

  • Spring Bootstrap — profiles and contexts, including the profile that skips Liquibase for an out-of-band-managed schema.

  • Observability Access — running the log queries above.

  • skills/liquibase-changelog-create/SKILL.md — authoring conventions: filenames, changeSet ids, operation snippets, and master-changelog registration.

6. Change History

Date Change

2026-09-07

Initial version. The blocked migration chain, fixture loads under strict mode, and the precondition rule.