CI and Release Pipeline Traps

Never put package in front of jib:build in docker-build.yml.

That job exists to containerise what the package job already built and restored from the target cache. Running the lifecycle again rebuilds everything a second time — measured at roughly twenty seconds for jib:build alone against about four minutes with package in front of it.

On front-end services it does not merely waste time, it fails: the docker job sets no NODE_AUTH_TOKEN, so npm install cannot resolve the generated API client from GitHub Packages and the build returns 401; another service failed the webapp build outright with exit 127.

The problem package was reached for — a missing javaagent on a cache miss — is handled correctly by fail-on-cache-miss: true on the cache restore step instead.

Jobs sit queued with no runner assigned and are cancelled at exactly the fifteen-minute mark, with empty step lists and no logs. Frequently one early job (for example Package) runs to completion and then every job after it starves — because that first job spent the last of the account’s Actions budget.

The EMS service repositories are private on a personal GitHub account. Unlimited free Actions minutes apply to public repositories only; private repositories draw on a metered monthly budget and require a spending limit above zero with a valid payment method. Once the budget or free-usage cap is reached, private-repo Actions stop and queued jobs cancel.

This is not a code problem and not a workflow problem. No amount of re-running fixes it.

Raise the Actions spending limit in GitHub billing settings and confirm the payment method, then re-run:

The billing endpoint that would report the remaining balance needs the user OAuth scope, which the gh token does not carry by default, so the exact remaining budget usually cannot be read from the CLI. Diagnose from the symptom instead.

When CI cannot run and a release must ship, the push-to-main pipeline can be reproduced by hand. Registry and package credentials live as server entries in ~/.m2/settings.xml; never inline a credential into a command or a document.

Build from the release or hotfix commit — after the release merge, main holds the release version:

The prod profile performs a self-contained front-end build through frontend-maven-plugin, using its own node and npm, so no nvm setup is required.

Build and push the image. Jib’s authentication must be passed explicitly because the settings server id does not match the registry host, so Jib will not match it automatically:

Package and push the chart. The chart version is the project version suffixed -RELEASE, with appVersion set to the project version:

If the gitflow finish step also failed to run, complete the branch by hand — see Development Workflow.

Finally deploy through GitOps by editing the ArgoCD manifest’s targetRevision to the new <version>-RELEASE and pushing. Stage updates automatically from CI’s deploy step; production is always a manual bump.

Path filters and concurrency live under on:, which belongs to the calling workflow. A workflow_call reusable workflow cannot express either.

This is therefore per-repository by construction. There is nothing to centralise in parent-pom, and attempting to do so produces a filter that is silently inert.

Every EMS service sets failOnError=false on maven-javadoc-plugin, so the Run JavaDoc tests step reports BUILD SUCCESS however many doclint errors it emits. admin-service currently emits 72, across 49 files, on every single run.

Until 2026-09-17 those lines were rendered as failure-level GitHub annotations by the problem matcher actions/setup-java installs, so a passing step — and a green run — displayed what looked like build failures. That misreading cost a full misdiagnosis of the outage described above: javadoc was blamed, a toolchain pin and a thirty-file rewrite were proposed, and the real cause was the changelog. The step now relabels those lines so the matcher does not claim them.

Two rules follow:

The underlying contradiction is real and tracked separately: parent-pom deliberately sets doclint=none with failOnError=true, and the services override that whole block with doclint=all,-missing and failOnError=false — running the checks while discarding their verdict.

A failing Snyk check on a private repository is sometimes the free-tier private-test limit and sometimes a real finding. Generalising the first case is a trap — a Critical transitive CVE has been dismissed this way.

The tell is the neighbouring runs. If recent releases on the same repository report "security tests have passed", the tier clearly permits the scan, and a failure means something was found. Open the report before dismissing it.

Two things in such a report look alarming and are not:

Reconcile first, then raise one ticket for the root cause rather than one per reported line.

Nothing back-merges those commits. gitflow:release-finish is the only thing that carries main into develop, and it runs only at a release. develop therefore accumulates a growing deficit of workflow fixes and fails its own builds on bugs that were fixed on main months earlier. It self-heals at the next release — which is the worst possible moment to discover it.

Remedy: back-merge after every workflow push. Use a real merge, never a cherry-pick or a squash — ancestry is what stops the next back-merge from conflicting.

If that is empty, the merge is a fast-forward of those files and develop keeps its own <revision> SNAPSHOT. Direct pushes to develop may be blocked, in which case open a pull request and merge it with "Create a merge commit".

gh pr merge fails on any pull request that touches .github/workflows/:

The gh token carries repo but not workflow. The behaviour is inconsistent — a single-file pull request has merged while a two-file one did not — so do not rely on it succeeding. Since the remote is SSH, merge locally and push:

GitHub marks the pull request merged automatically once the commits are reachable from main.

These changes go live for every repository the moment they land. Re-running a recent consumer build re-resolves @main fresh, which makes it a cheap end-to-end check. Confirm it actually picked up the change by grepping the job log for the Download action repository line, rather than trusting the green tick.

The back-merge is git merge --no-ff <tag> into develop. The tag’s pom.xml carries the release version; develop’s carries a SNAPSHOT. Both sides changed the same `<revision> line since the merge base, so git reports a content conflict.

develop moves because reserving the next SNAPSHOT on a branch is the convention here. Any branch merging to develop while a release is open therefore collides on exactly that line — this is routine, not an accident, and it cannot be designed away by eliminating the diverging file the way an earlier docker-compose divergence was.

The gitflow plugin has no conflict resolution and simply aborts. The shared workflow takes the develop half away from it — -DskipReleaseMergeDevBranch=true on the release path, -DskipMergeDevBranch=true on the hotfix path — and owns the merge, the resolution and the version bump.

Resolution keeps develop’s side. That rule is deterministic: `develop is the leading edge for <revision>, and the release version belongs to the tag and to main.

Two conditions must both hold, or the job fails with the offending paths named and leaves develop exactly as it was:

The second condition is not redundant. Conflict hunks are line-range based and <revision> sits among the other properties, so a release that also bumped an adjacent property has both changes collapsed into one hunk — and keeping `develop’s side then drops the other edit with no diagnostic at all. The exposure is larger on the hotfix path, where carrying a substantive change is the entire point of the branch.

git checkout --ours pom.xml is not an equivalent shortcut and must not be substituted. It takes develop’s whole file, discarding release edits in hunks that merged cleanly. Resolving hunk by hunk with `git merge-file --ours keeps them.

Silently resolving any conflict would paper over genuine drift between main and develop, which is the failure this narrowness exists to preserve.

develop requires the PR checks status check. GITHUB_TOKEN pushes as github-actions[bot], which is not an administrator and so is not covered by enforce_admins: false; GitHub rejects the back-merge push with GH006. A human administrator pushing the same commit succeeds, which is why the failure only ever appears in CI.

The shared workflow accepts an optional RELEASE_PAT for this — an admin PAT (Contents: read+write) configured as a bypass actor — and checks out with:

The calling workflow must pass it explicitly. A reusable workflow receives only the secrets its caller maps to it, so a caller with no secrets: block resolves that expression to GITHUB_TOKEN no matter how the secret is configured in the repository. Creating the repository secret is not enough on its own, and nothing warns: the expression simply falls back.

The fallback itself is deliberate. A repository whose develop carries no required status check needs no PAT, and mapping a secret that does not exist resolves to empty rather than failing — so the same block is safe to add everywhere.

push-main.yml runs from the copy on main, so a change to the calling workflow reaches the pipeline differently from a change to the shared workflow it calls:

The release commit that brings a corrected caller onto main is the same commit that triggers the run, so that release is covered — but a release already in flight is not.

The generated assert…UpdatableFieldsEquals helpers assert every field as though it were persisted. An entity getter that is @Transient and derived rather than persisted is therefore a downstream-breaking change even though no API signature moved: the assertion is a harmless no-op while the getter returns a constant, but once it derives a value it compares two objects that need not share the state it derives from. A partial-update test then fails comparing a derived label against a default.

Adding a derived getter means checking the generated asserts for that entity in the same change.

This used to be worse. The helpers arrived in a separate tests-classifier jar, so a change could break admin-service’s integration suite while the publishing repository’s own build stayed green — that build never ran admin-service’s tests. Since the merge below, the entity, its helper and the test that consumes it are all in one repository and one build, so the breakage is immediate and local.