Dependency Update Review

1. Overview

Automated checks propose third-party dependency updates as pull requests. They do not apply them. This guide covers the weekly review: where to look, how to decide, how to implement, and how to reverse it.

The design behind the checks — why references are pinned as tag@digest, and why the checks are scoped the way they are — is Dependency Update Checks.

For the GitOps repositories, merging is deploying. ArgoCD runs with automated.prune and automated.selfHeal, so a merge to main reaches production within a few minutes with no further gate. Review accordingly, and do not merge without someone available to verify the result.

2. Cadence and Where to Look

Checks run Monday 07:00 SAST. Proposals appear as pull requests labelled dependencies, plus a component label.

# Open dependency proposals across the GitOps repository
gh pr list --repo christhonie/idl-xnl-jhb-rc01 --label dependencies --state open

# Full detail for one proposal
gh pr view <number> --repo christhonie/idl-xnl-jhb-rc01

A proposal changes the tag and digest together:

-  image: lscr.io/linuxserver/openssh-server:10.2_p1-r0-ls229@sha256:67d4c3a1…
+  image: lscr.io/linuxserver/openssh-server:10.2_p1-r0-ls230@sha256:9f21ab4c…

A proposal against one of our own images (christhonie/event-, christhonie/ems-) is a misconfiguration, not a normal update. Those tags belong to the release-promotion pipeline. Close the pull request and fix the scoping in .github/dependabot.yml before merging anything else.

3. Assessing a Proposal

3.1. Step 1: Classify the Exposure Tier

How much scrutiny a change deserves depends mostly on what can reach the component.

Tier Components Scrutiny

Internet-facing

SSH bastion (/ssh-bastion), anything published through ingress

Highest. Read the upstream release notes in full, with specific attention to changed defaults around authentication, rate limiting, and cryptography.

Cluster-internal

postgres, sonarqube, the Grafana observability components (Loki, Tempo, Grafana, Alloy, OpenTelemetry Collector)

Moderate. Confirm no configuration-surface or data-format change; check for a required migration.

Build-time / non-production

busybox init containers, greenmail

Low. A changelog skim is sufficient.

3.2. Step 2: Classify the Change Magnitude

Magnitude Looks like Handling

Rebuild only

Same upstream version, new build suffix — ls229 → ls230

Usually base-image security patches. Normal weekly approval.

Patch / minor

10.2_p1 → 10.3_p1

Read release notes for behaviour and default changes. Approve if none affect our configuration.

Major

10.x → 11.x

Do not merge on the weekly sweep. Raise a work item and schedule it with time to test.

3.3. Step 3: Apply the Approval Conditions

All of the following must hold. If any fails, defer — see Deferring or Rejecting a Proposal.

  1. Release notes reviewed for every version between the current and proposed reference, not only the newest.

  2. No default-behaviour change that interacts with our configuration. For internet-facing components this is the condition that matters most; see the warning below.

  3. Tag and digest are consistent — the proposed digest genuinely belongs to the proposed tag.

  4. Not inside a change freeze — no release in flight, and no event weekend where the affected system is in use.

  5. A verifier is available — someone can run the post-merge checks now, not tomorrow.

  6. Rollback is understood — you know which commit to revert.

Condition 2 is the one that has actually bitten us. The 2026-08-17 bastion lockout was caused by an upstream default, not a broken build: OpenSSH enables PerSourcePenalties by default from 9.8 onward, which interacted with NodePort source-NAT to refuse every legitimate connection. The image itself was working exactly as designed.

For any sshd or gateway update, explicitly check whether upstream changed defaults for authentication, connection rate limiting, penalties, or accepted algorithms — and record what you checked in the pull request.

4. Implementing an Approved Update

  1. Merge the pull request to main.

    gh pr merge <number> --repo christhonie/idl-xnl-jhb-rc01 --squash

  2. Wait for ArgoCD to sync — typically under three minutes. Do not apply manifests by hand; selfHeal will revert anything applied out of band.

  3. Confirm the rollout completed.

    kubectl rollout status ds/ssh-bastion -n bastion --timeout=180s
kubectl get pods -n bastion -o wide

  4. Run the component verification below.

  5. Record the outcome in the pull request — what you verified, and anything unexpected.

4.1. Verification: SSH Bastion

The bastion is a DaemonSet, so an update rolls one pod at a time and an established tunnel drops when its own node’s pod cycles. Warn anyone holding a tunnel first.

# 1. All pods healthy on every node
kubectl get ds ssh-bastion -n bastion
kubectl get pods -n bastion -o wide

# 2. Authentication works through every node IP.
#    Node IPs and the key path are in the bastion-tunnel skill.
for ip in <node-ip-1> <node-ip-2> <node-ip-3>; do
  ssh -i <key> -o BatchMode=yes -p 30022 tunnel@$ip 'echo OK; hostname'
done

Expected: OK from each, each reporting a different pod name — that confirms externalTrafficPolicy: Local is still routing node-locally. A host-key mismatch warning means the shared host-key Secret was not mounted correctly; stop and investigate.

# 3. Effective sshd configuration is unchanged
kubectl exec -n bastion <pod> -- \
  /usr/sbin/sshd.pam -T -f /config/sshd/sshd_config \
    -h /config/ssh_host_keys/ssh_host_ed25519_key \
  | grep -iE 'persource|penalt|maxstartups|logingrace'

Compare against the recorded baseline:

logingracetime 30
maxstartups 10:30:100
persourcemaxstartups none
persourcenetblocksize 32:128
persourcepenaltyexemptlist none
persourcepenalties crash:90 authfail:5 noauth:1 grace-exceeded:10 \
  refuseconnection:10 max:600 min:15 max-sources4:65536 \
  max-sources6:65536 overflow:permissive overflow6:permissive

Any difference is a behaviour change that arrived with the image. Assess it before leaving the update in place, and update this baseline if you accept it.

The bastion runs with LOG_STDOUT=true, so sshd output reaches Loki via Alloy and survives the rolling restart — pre-update evidence is still queryable afterwards, with 30-day retention. Compare behaviour either side of the update in Grafana:

{namespace="bastion", container="openssh-server"}

# Refused connections — the signature of the 2026-08-17 lockout
{namespace="bastion"} |= "srclimit_penalise"

Without LOG_STDOUT=true the s6 log service diverts sshd output to a file inside the container’s ephemeral layer, where Alloy cannot see it and a restart destroys it. If a component’s logs are missing from Loki, check this first.

Note that Connection closed by 192.168.0.x lines are deliberately dropped by an Alloy stage.drop — they are kubelet health probes, roughly one every few seconds, and they would otherwise bury the stream. Their absence from Loki is expected; they are still visible at source with kubectl logs. Real client connections carry public IPs and are never filtered.

5. Rolling Back

Reverting the merge is the only correct route — selfHeal will undo a manual kubectl change.

git -C ~/dev/idl-xnl-jhb-rc01 revert <merge-sha>
git -C ~/dev/idl-xnl-jhb-rc01 push origin main

ArgoCD re-syncs to the previous digest within a few minutes. Re-run the component verification, then comment on the original pull request explaining what failed, so the proposal is not blindly re-approved next week.

6. Deferring or Rejecting a Proposal

Leaving a proposal open is a decision that decays — next week’s proposal supersedes it and the reasoning is lost. Close it explicitly with a comment recording why.

Situation Action

Needs scheduled work (major version)

Raise an ADO work item, link it in the pull request, then @dependabot close.

This version is bad, later ones may be fine

@dependabot ignore this minor version

Deliberately staying on the current major

@dependabot ignore this major version

Superseded / stale branch

@dependabot rebase or @dependabot recreate

Never use @dependabot ignore this dependency — it silences the component permanently and reintroduces the frozen-pin risk the checks exist to prevent.

7. Escalation

  • Update merged and the component is degraded → roll back first, diagnose afterwards.

  • Bastion unreachable through all node IPs after an update → roll back. kubectl access does not depend on the bastion, so recovery is always available.

  • Proposal appears against one of our own application images → do not merge; fix the scoping in .github/dependabot.yml.

8. Related