Ingress and Client Addresses

Every public hostname on the cluster is served by one ingress controller. This page records how traffic reaches it, how nginx and the applications behind it learn the visitor’s real address, and the one place the controller’s configuration can be changed.

Overview

Controller

rke2-ingress-nginx, the ingress-nginx build that RKE2 packages

Namespace

kube-system

Workload

DaemonSet, one controller pod on each of the three nodes

Virtual IP

197.189.214.93, floated between the nodes by kube-vip in ARP mode

Service

rke2-ingress-nginx-controller, type LoadBalancer, with the virtual IP as an externalIPs entry

Traffic policy

externalTrafficPolicy: Local

Managed by

RKE2’s Helm controller, from a HelmChartConfig manifest on the server nodes. Not ArgoCD.

How traffic arrives

There are two ways into the controller, and both end at the nginx pod on the node that received the connection.

Entry point Path Used by

Entry point	Path	Used by
Virtual IP	kube-proxy matches the Service’s `externalIPs` entry and hands the connection to the nginx pod on the same node	Every public hostname. Direct hostnames resolve to it; Cloudflare connects to it as the origin.
A node’s own public IP	The controller pods bind host ports 80 and 443 on every node	Nothing by design. It works, and is useful for testing one node.

Virtual IP

kube-proxy matches the Service’s externalIPs entry and hands the connection to the nginx pod on the same node

Every public hostname. Direct hostnames resolve to it; Cloudflare connects to it as the origin.

A node’s own public IP

The controller pods bind host ports 80 and 443 on every node

Nothing by design. It works, and is useful for testing one node.

kube-vip moves the virtual IP with the control-plane leader lease. It does not watch nginx.

DNS is hosted at Cloudflare. Some hostnames are proxied through Cloudflare and some are DNS-only, so the controller has to produce the right visitor address for both.

Preserving the visitor’s address

Two settings work together. The first makes nginx see the true source of each connection; the second lets it look one step further when that source is Cloudflare.

Traffic policy Local

With the default policy (Cluster), kube-proxy masquerades every connection that arrives on the virtual IP and forwards it to a controller pod on any node. nginx then sees a node or pod address, and so does every application behind it. Per-address controls such as rate limits collapse onto a handful of internal addresses.

externalTrafficPolicy: Local removes the masquerade. The connection is delivered only to the controller pod on the node that holds the virtual IP, with its source address intact.

The usual drawback of Local is that a node without a local endpoint drops the traffic. That does not arise here in normal running, because the DaemonSet puts a controller pod on every node.

Cloudflare’s header

For a proxied hostname the connection comes from Cloudflare, so the source address is Cloudflare’s. Cloudflare passes the visitor’s address in the CF-Connecting-IP header.

The controller is configured to take the address from that header only when the connection itself comes from one of Cloudflare’s published ranges:

enable-real-ip: "true"
forwarded-for-header: "CF-Connecting-IP"
proxy-real-ip-cidr: "<Cloudflare's IPv4 ranges, comma separated>"

The result for each kind of request:

Request Address nginx uses

Request	Address nginx uses
Through Cloudflare	The visitor’s, from `CF-Connecting-IP`
Direct to a DNS-only hostname	The connection’s own source address
Direct, with a forged `CF-Connecting-IP` header	The connection’s own source address. The header is ignored because the source is not a Cloudflare range.

Through Cloudflare

The visitor’s, from CF-Connecting-IP

Direct to a DNS-only hostname

The connection’s own source address

Direct, with a forged CF-Connecting-IP header

The connection’s own source address. The header is ignored because the source is not a Cloudflare range.

The forged-header case is only safe because of the Local policy. Under Cluster, every connection appeared to come from inside the cluster, so nginx could not tell Cloudflare from a direct caller.

use-forwarded-headers is deliberately not enabled. It would also make the controller accept X-Forwarded-Host and X-Forwarded-Proto from callers, which is not wanted on hostnames that are reachable directly.

Only the IPv4 ranges are listed. The nodes have IPv4 public addresses only, so Cloudflare always connects to the origin over IPv4.

What applications receive

nginx sets X-Forwarded-For to the single address it resolved, so an application behind the ingress sees the visitor’s address whichever way the request came in. Calls made from inside the cluster to a public hostname still arrive with an internal address.

Trade-offs

One node carries all ingress traffic. Only the controller pod on the node holding the virtual IP serves requests. The other two stand by.
No fallback while that pod is down. If the controller pod on the node holding the virtual IP is not ready, new connections are dropped until it is ready again or the virtual IP moves. The DaemonSet replaces pods one at a time with no surge, so a controller upgrade or pod restart on that node is a short outage for every hostname.
The Cloudflare ranges need upkeep. If Cloudflare adds a range, requests arriving from it show Cloudflare’s address until the list is updated. Nothing breaks.
A health-check port is open on each node. A LoadBalancer Service with the Local policy is given a health-check node port. It reports only whether the node has a local endpoint.

Where the configuration lives

The controller’s overrides are a HelmChartConfig manifest that must exist, identical, on all three server nodes:

/var/lib/rancher/rke2/server/manifests/rke2-ingress-nginx-config.yaml

The record of that file is rancher/rke2-ingress-nginx-config.yaml in the cluster repository. RKE2 watches the manifests directory, and within seconds of a change it runs a Helm upgrade of the packaged chart with these values. A change to the config block makes nginx reload; it does not restart the pods.

To change the controller, edit the file in the repository, copy it to all three nodes, verify, then commit.

Every other route is overwritten sooner or later.

A manual helm upgrade, or the Rancher Apps page. RKE2 owns the release and reinstalls it from its own values. Rancher shows the warning "Rancher manages deployment and upgrade of the rke2-ingress-nginx app" for this reason.
rke2-ingress-nginx.yaml in the same directory. It is the packaged chart, and RKE2 rewrites it on every server start.
rancher/managed-chart-config.yaml on the nodes. Rancher generates it from the cluster’s add-on configuration.
kubectl edit or kubectl patch of the Service or ConfigMap. It takes effect at once and is reverted by the next Helm upgrade. That makes it useful for a trial and wrong as a fix.

Deleting the manifest from a node does not remove its settings from the cluster. RKE2 leaves the resources a deleted manifest created in place, so a missing file is not evidence that the settings are gone. Compare against the live object instead:

kubectl -n kube-system get helmchartconfig rke2-ingress-nginx -o jsonpath='{.spec.valuesContent}'

Verifying

Send a marked request through each entry point, then read the address nginx logged for it. The first field of each access-log line is the address nginx resolved.

HOST=<a hostname served by the cluster>

# Through the virtual IP, bypassing Cloudflare, with a forged header
curl -sk -o /dev/null -H 'CF-Connecting-IP: 1.2.3.4' \
  --resolve $HOST:443:197.189.214.93 "https://$HOST/srcip-probe-vip"

# Through Cloudflare (use a proxied hostname)
curl -s -o /dev/null "https://<proxied hostname>/srcip-probe-cf"

# What nginx logged
for p in $(kubectl -n kube-system get pods -o name | grep rke2-ingress-nginx-controller); do
  kubectl -n kube-system logs $p --since=2m | grep srcip-probe | awk '{print $1, $7}'
done

Every line should show the public address of the machine that ran curl. An internal address means the traffic policy has reverted; a Cloudflare address on the second probe means the real-IP settings are missing or the ranges are out of date.

To confirm what is configured:

# Traffic policy and virtual IP on the Service
kubectl -n kube-system get svc rke2-ingress-nginx-controller \
  -o jsonpath='{.spec.externalTrafficPolicy} {.spec.externalIPs}{"\n"}'

# Real-IP settings as nginx received them
kubectl -n kube-system get cm rke2-ingress-nginx-controller -o jsonpath='{.data}'

Rolling back

For an immediate effect, patch the Service back. This lasts until the next Helm upgrade, so follow it with the durable change.

kubectl -n kube-system patch svc rke2-ingress-nginx-controller \
  --type merge -p '{"spec":{"externalTrafficPolicy":"Cluster"}}'

For the durable change, remove the setting from the manifest in the repository and copy it to all three nodes. Removing externalTrafficPolicy restores the masquerade, so remove the three real-IP keys with it: without the Local policy the header trust has nothing to stand on.

Keeping the Cloudflare ranges current

Cloudflare publishes its ranges at https://www.cloudflare.com/ips-v4 and changes them rarely. Compare the published list with the proxy-real-ip-cidr value from time to time, and whenever requests through Cloudflare start showing Cloudflare addresses.

The idealogic-infra:ingress-cloudflare-ips skill carries the comparison and the update procedure.