Bug: init-iptables.sh picks wrong backend on nft-only nodes (readability probe ≠ liveness)

[huang195]

Summary

init-iptables.sh's detect_iptables_cmd() selects the wrong iptables backend on nft-only nodes (ROSA, recent RHEL/OpenShift, EKS, recent Ubuntu). It decides which backend to use by checking whether the legacy nat table can be listed, treats that read as proof the legacy backend is in use, and prefers legacy whenever the read succeeds. On nft-only hosts the legacy nat table is still readable (just empty), so the script picks iptables-legacy and programs our chains into a table the live datapath never consults. The result is a silent no-op: enforce-redirect (and redirect) rules insert successfully but never take effect.

File: authbridge/authproxy/init-iptables.sh (detect_iptables_cmd).

Current logic

detect_iptables_cmd() {
  if [ -n "${IPTABLES_CMD:-}" ]; then
    echo "${IPTABLES_CMD}"; return
  fi
  if command -v iptables-legacy >/dev/null 2>&1 && \
     iptables-legacy -t nat -L -n >/dev/null 2>&1; then   # readability probe
    echo "iptables-legacy"                                # legacy-first bias
  else
    echo "iptables"                                       # nft backend
  fi
}

Root cause

Linux exposes two independent packet-filtering subsystems behind the same CLI syntax: legacy x_tables (iptables-legacy) and nf_tables (iptables, the nft backend on Alpine). Rules written to one are not evaluated against the other. Our rules only work if they go into the backend the host (kube-proxy / CNI / Istio) actually uses.

Two problems with the current detection:

1. Readability ≠ liveness. iptables-legacy -t nat -L -n only answers "can I enumerate the legacy nat table?" — a read. It does not answer "is the legacy backend where the live rules live?" On an nft-only node the legacy nat table typically lists fine (empty), so the probe passes and legacy is chosen even though every real NAT rule is in nf_tables.
2. Legacy-first bias. The "prefer legacy for compatibility" default reflects an era when K8s/Istio used legacy. Most current clusters are nft-only, so the default is backwards.

Failure mode is silent

Because the rules insert successfully into a real (but dead) table, every command returns success: the init container exits 0, the pod is healthy, and iptables-legacy -t nat -L shows the chains present — yet egress that should be captured/redirected leaks straight out. There is no error to alert on; you only notice that traffic isn't being intercepted.

This is distinct from the capability/SCC failure seen on ROSA (a hard EPERM initializing the nat table that CrashLoops the init container). That one is loud and is about CAP_NET_ADMIN/SCC admission, not backend selection. The detection bug doesn't crash — it leaks. Both can be present on the same platform.

Proposed fix

1. Probe liveness, not readability. Pick the backend that already holds the host's rules — e.g. compare iptables-legacy-save -t nat vs iptables-nft-save -t nat and prefer the backend that contains the existing kube-proxy / Istio chains.
2. Write-probe the chosen backend (insert a throwaway rule, then delete it) so failures cleanly distinguish "wrong backend" from "no CAP_NET_ADMIN", and so a silent no-op can't happen.
3. Default to nft when ambiguous, not legacy.
4. Keep the existing IPTABLES_CMD / IP6TABLES_CMD override as the escape hatch, and have the operator set it explicitly on known platforms.

Impact

• enforce-redirect (proxy-sidecar / lite) silently fails open on nft-only nodes — egress enforcement is bypassed without any signal.
• redirect (envoy-sidecar) interception silently does nothing on the same nodes.

cf. upstream caution that spawn/list success ≠ a functioning backend.

[huang195]

Update: ROSA HCP testing shows the failure is deeper than detection, and the proposed write-probe fix is insufficient

Empirical findings from testing the actual proxy-init image on ROSA HCP (managed OpenShift, Red Hat-controlled nodes). These refine both the root cause and the fix proposed above.

The detection bug is real, but a write-probe won't catch it

The original fix idea here was: probe writability in the chosen backend (insert + delete a throwaway rule) to detect the wrong backend. That is insufficient.

Three increasingly aggressive probes were tried under iptables-legacy — table list, chain creation, and REDIRECT rule creation — and all three pass, because the nft_compat kernel module successfully emulates individual iptables-legacy operations. The failure only surfaces when the full nat pipeline initializes inside setup_enforce_redirect(). So any lightweight probe ("can I create a rule?") reports success and the script proceeds to fail mid-setup.

Implication: detection cannot rely on legacy being usable just because individual ops succeed. The fix must not prefer legacy at all on nft hosts, and must verify the nft backend end-to-end rather than via a single-op probe.

The two backends fail for different reasons — only one is structural

Tested both backends with the real image on ROSA HCP:

Backend	Result	Root cause
iptables-legacy	fails	iptable_nat kernel module not loaded (node lsmod shows only nf_tables + nft_compat), and SELinux blocks container_t from triggering its autoload. On ROSA HCP there is no MachineConfig access to load kernel modules — nodes are Red Hat-managed. Structural / unfixable from the workload side.
iptables (nft)	fails	Could not fetch rule set generation id: Permission denied

The nft failure is not a missing-module problem: nf_tables is loaded, and the nft nat table needs no extra module. With root + NET_ADMIN + NET_RAW already granted (the pod is admitted under the kagenti-authbridge SCC), a Permission denied fetching the nf_tables generation id is the classic SELinux container_t → nf_tables netlink denial — documented in Red Hat KB 5990311. The fix is to set seLinuxOptions on the container's securityContext (e.g. a less-confined type).

SCC blocks the SELinux fix as currently written

charts/kagenti/templates/kagenti-authbridge-scc.yaml sets seLinuxContext: type: MustRunAs. Under MustRunAs, a pod that specifies a custom seLinuxOptions is rejected at admission — so the KB 5990311 fix cannot even be applied until the SCC is changed to seLinuxContext: type: RunAsAny. Any nft+SELinux test that omits this SCC change has not actually exercised the fix.

Net takeaways for this issue

1. Detection fix: stop preferring iptables-legacy; default to nft; do not treat single-op probe success as proof the backend works (the nat pipeline init is the real test).
2. Legacy is dead on managed OpenShift (ROSA HCP) — missing iptable_nat, no way to load it. Not a detection or SCC problem.
3. nft is viable if SELinux is addressed: seLinuxContext: MustRunAs → RunAsAny in the SCC plus seLinuxOptions on the proxy-init container. This needs a definitive yes/no test on ROSA HCP. If it works, enforce-redirect (full ext_proc enforcement) is preserved on managed OpenShift.
4. Posture note: the SELinux relaxation likely means a near-unconfined type escaping MCS confinement — a conscious security decision on managed platforms, not a free win.

Credit: ROSA HCP backend/SELinux/lsmod testing by Akram.

[huang195]

PR for the detection fix: #518 — switches detect_iptables_cmd() to a /proc/modules (iptable_nat) functional check + adds require_jump loud-fail. Scoped to detection only; it does not make ROSA work (selects nft + fails loud on the SELinux denial). The ROSA SELinux/SCC track and the operator IPTABLES_CMD companion remain separate.

[huang195]

PR merged. closing