Bastion Access

1. Overview

Cluster-internal services — MySQL Router above all — are deliberately not exposed. Reaching them from a workstation goes through an SSH bastion running in the bastion namespace, which forwards a local port to any Kubernetes service FQDN. It is the supported alternative to kubectl port-forward, which adds several seconds of latency per query and drops on idle.

Day-to-day tunnel commands, key handling, GUI-client setup and the routine troubleshooting are covered by the bastion-tunnel skill at ~/dev/ai-skills-infra/skills/bastion-tunnel/SKILL.md, and the database-specific form by ~/dev/ai-skills-infra/skills/mysql-idealogic-prod/SKILL.md. This page covers the deployment — what it is made of, and the two failure modes that are not obvious from the outside.

2. Deployment shape

Aspect Detail

Location

Namespace bastion on the on-premises cluster.

Workload

A DaemonSet, so every node has a local endpoint. It is not a single Deployment, and the reason is the traffic policy below — not redundancy in the usual sense.

Exposure

A Service of type NodePort with externalTrafficPolicy: Local.

Image

lscr.io/linuxserver/openssh-server.

Authentication

Ed25519 public key only. Password authentication is disabled. The SSH user is tunnel.

Host key

Supplied from the Secret ssh-bastion-host-key, so every pod on every node presents the same host key and client known_hosts entries stay valid across pod restarts, node changes and rescheduling.

Manifests

ssh-bastion/ in the GitOps repository idl-xnl-jhb-rc01, deployed by an ArgoCD Application with prune and selfHeal enabled.

Because the Application self-heals, a live kubectl edit is reverted on the next sync. Every change goes through Git. See ArgoCD Deployment Patterns.

Do not record node IPs or the NodePort in documentation or scripts — both change. Discover them from the cluster each time, as the skill describes.

3. Connection refusals that are not a node or network fault

3.1. Symptom

New SSH connections are refused on every node IP. The server-side message is:

Not allowed at this time

which reaches the client as:

kex_exchange_identification: read: Connection reset by peer

Every node behaving identically reads as a node-level or network-level problem. It is neither.

3.2. Cause

OpenSSH PerSourcePenalties is enabled by default from OpenSSH 9.8 onward. It tracks authentication failures per source address and temporarily refuses further connections from a source that has accumulated too many, up to a penalty cap of 600 seconds.

With externalTrafficPolicy: Cluster, kube-proxy masquerades every inbound client to a single address per node before the packet reaches the pod. Every client arriving through a given node therefore shares one source address as far as sshd is concerned.

An internet brute-force campaign against the NodePort then charges thousands of authentication failures to those few shared addresses. They pin at the penalty cap, and every legitimate user behind them is refused — permanently, for as long as the scanning continues.

The three node IPs were never redundancy under the original single-pod Deployment: one pod served all three, so all three failed together and reinforced the impression of a cluster-wide fault.

3.3. Diagnosis

Check What it tells you

Connect via the pod IP or the ClusterIP

These paths bypass the NodePort masquerade. If they return an SSH-2.0-… banner while the NodePort refuses, the pod is healthy and the source address seen by sshd is the problem.

Look for the penalty log line

srclimit_penalise: <addr>/32: activating ipv4 penalty … is the proof. The bastion runs with LOG_STDOUT=true so this reaches Loki and survives pod restarts.

Inspect live connections

conntrack -L on the node shows real client-to-masqueraded-address mappings.

 
{namespace="bastion"} |= "srclimit_penalise"

See Observability Access for reaching Loki.

Two traps while diagnosing:

  • iptables NodePort packet counters prove nothing. The port takes constant internet scan traffic — a few packets per second at all times — so a rising counter is background noise, not evidence of your own attempt. Use conntrack -L instead.

  • Bare TCP probes make it worse. Opening a socket without completing authentication accrues a noauth penalty against your own address. Authenticate properly when testing, or you will lock yourself out while investigating a lockout.

  • Without LOG_STDOUT, sshd writes only to /config/logs/openssh/current inside the container. kubectl logs then shows nothing but s6 init output, and a pod restart destroys the evidence.

3.4. Fix

Two changes, together:

  1. externalTrafficPolicy: Local on the Service. The client’s real source address is preserved all the way to sshd, so penalties are charged to the address that actually earned them. A brute-forcing scanner now locks out only itself.

  2. Convert the workload to a DaemonSet. Local only forwards traffic to endpoints on the node that received it, so a node with no local pod would blackhole the NodePort entirely. A DaemonSet guarantees an endpoint on every node.

Neither change works without the other, and the DaemonSet is what makes the several node IPs genuinely independent for the first time.

4. The image treats its config file as a writable seed

The LinuxServer OpenSSH image does not read /etc/ssh/sshd_config at runtime. Its init script treats that path as a writable seed: it edits the file in place, then copies it to /config/sshd/sshd_config, which is the file sshd is actually started against — visible on the process command line as sshd.pam -f /config/sshd/sshd_config.

Configuration is supplied as the ConfigMap ssh-bastion-sshd-config, mounted read-only over the seed path.

4.1. The harmless errors

Because the seed mount is read-only, two sed -i calls in the init script fail at startup:

sed: can't move '/etc/ssh/sshd_configXXXX': Resource busy

They intend to set a PidFile under /config (sshd runs as a non-root uid) and to comment out an Include directive. Both are content no-ops for this configuration, which contains neither directive — sed -i fails on the read-only mount regardless of whether its pattern matched, because it works by renaming a temporary file over the original. Leave them.

4.2. The mount that looks like the fix and is not

Do not "fix" the errors above by mounting the ConfigMap at /config/sshd/sshd_config.

Later in the same init script, unconditional sed -i calls rewrite that destination to apply the listen port, the password-access setting and the SFTP umask — all of which this deployment sets. A read-only mount there turns two harmless startup errors into three or more real ones, and the settings that do matter never get applied. The destination must stay writable.

4.3. The latent trap

The copy from seed to destination is guarded:

if [[ ! -f /config/sshd/sshd_config ]]; then
  # copy the seed into place
fi

/config is currently the container’s ephemeral writable layer, so the file is absent on every fresh pod and the ConfigMap propagates on every start.

Mount a PersistentVolumeClaim at /config — the idiomatic setup for LinuxServer images, and an easy thing to add for log retention — and the copy is skipped forever after the first start. From then on the ConfigMap is silently ignored: no error, no event, and a Git history that reads as though the configuration had been applied.

4.4. Changing sshd configuration

  1. Edit ssh-bastion/sshd-config.yml in idl-xnl-jhb-rc01 and push. ArgoCD syncs the ConfigMap.

  2. Restart the pods — a ConfigMap change does not restart them on its own, and the copy only runs at pod start.

  3. Verify the directive landed in the file sshd actually runs:

    kubectl -n bastion exec <pod> -- grep -i <directive> /config/sshd/sshd_config

Step 3 is not optional. Every failure mode on this page ends with the ConfigMap looking correct in Git while the running process uses something else.

5. Related Documentation

  • ~/dev/ai-skills-infra/skills/bastion-tunnel/SKILL.md — establishing tunnels, key handling, GUI clients

  • ~/dev/ai-skills-infra/skills/mysql-idealogic-prod/SKILL.md — the database-specific tunnel

  • Observability Access — reading bastion logs in Loki

  • ArgoCD Deployment Patterns — how the manifests are deployed