From 8aa04f256cdfa0710bfae78373544a0a2a17d971 Mon Sep 17 00:00:00 2001 From: Paul O'Reilly Date: Sat, 2 May 2026 14:41:15 +1200 Subject: [PATCH] kubernetes.md: Cilium entities apply beyond monitoring; ipBlock no-op for nodes Reframe "Cilium Entity Identities for Monitoring Scraping" as cross-cutting -- the same entity table applies to any pod that needs to reach cluster infrastructure (apiserver, kubelets, node-exporter, host services), not just Prometheus. Add the gotcha that bit M22 Phase 7: standard NetworkPolicy ipBlock CIDR rules do NOT match cluster node IPs. Nodes carry the Cilium remote-node/kube-apiserver identity and ipBlock only matches off-cluster IPs. The misleading symptom is a 30s hang followed by a generic upstream error like "permission denied" (seen on OpenBao TokenReview, would also affect ESO+vault k8s auth, and any controller calling subjectaccessreviews). Same gotcha applies to namespaceSelector: kube-system -- the apiserver runs hostNetwork=true and is not selectable that way. Source incident: agent-runtimes M22 Phase 7 F-OPENBAO-K8S-AUTH-1 (homelab/openbao-deploy@f7bd64d). Co-Authored-By: Claude Opus 4.7 (1M context) --- kubernetes.md | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/kubernetes.md b/kubernetes.md index f3e3630..7444c5c 100644 --- a/kubernetes.md +++ b/kubernetes.md @@ -59,9 +59,9 @@ Manual bootstrap secrets (encryption keys, OIDC client secrets) must be document - **Document privileged namespace requirements.** When a workload needs elevated privileges, document the specific requirement (e.g., "Docker-in-Docker for CI builds") alongside the namespace label. - **Monitoring namespace requires privileged PodSecurity for node-exporter.** kube-prometheus-stack's node-exporter DaemonSet mounts host paths and uses `hostPID: true`. The monitoring namespace must be labelled `pod-security.kubernetes.io/enforce: privileged` or node-exporter pods will be silently blocked. Set this via GitOps namespace metadata — don't apply it manually or it will be reverted by the GitOps controller. -## Cilium Entity Identities for Monitoring Scraping +## Cilium Entity Identities (any pod-to-cluster-infrastructure egress) -When writing Cilium network policies to allow Prometheus to scrape targets, the correct entity identity depends on the node role: +Applies to any pod that needs to reach cluster infrastructure — kube-apiserver, kubelets, node-exporter, host services. Not just Prometheus scraping. Common cases that trip on this: OpenBao/Vault calling TokenReview, External Secrets Operator authenticating to Vault via k8s auth, controllers calling subjectaccessreviews, anything that hits `https://kubernetes.default.svc`. | Target | Cilium entity | |---|---| @@ -69,7 +69,11 @@ When writing Cilium network policies to allow Prometheus to scrape targets, the | Worker node kubelet / node-exporter | `remote-node` | | Same-node kubelet (DaemonSet on same node) | `host` | -Using the wrong entity results in silent policy drops. Test with `cilium monitor --type drop` to identify mismatches. +**Standard `NetworkPolicy` `ipBlock` CIDR rules do NOT match cluster node IPs.** Cluster nodes carry the Cilium `remote-node` (or `kube-apiserver`) identity, and `ipBlock` only matches IPs *without* a Cilium identity (i.e., off-cluster). Listing `10.X.X.0/24` for the control-plane subnet and expecting it to allow apiserver egress will silently fail — the rule is treated as a no-op and traffic is dropped. The CNI-native expression is `CiliumNetworkPolicy` with `toEntities: [...]`. + +The `namespaceSelector: kube-system` rule is also ineffective for kube-apiserver because the API server runs `hostNetwork: true` and is not selectable by namespace selector — its identity is host/remote-node/kube-apiserver, not the kube-system pod identity. + +Using the wrong entity (or the wrong policy kind) results in silent policy drops. Symptom for apiserver-bound traffic: HTTPS calls hang until the client times out (typically 30s for Go HTTP defaults), then the upstream returns a generic error like `permission denied`. Test with `cilium monitor --type drop` to confirm the drop is at L3, or — if you can `exec` into the pod — try `wget --timeout=5 https://kubernetes.default.svc/healthz` and see if it `Terminated`s. ## Kustomize Overlay `images:` Blocks Silently Override Base Tags