# Kubernetes Patterns ## Volume Mounts - **Avoid `subPath` volume mounts** for Secrets and ConfigMaps. The kubelet does not auto-update `subPath` mounts when the source changes — the pod must be restarted. Use directory mounts instead and adjust the application's config path. - **Secret volume propagation is async.** After updating a Secret, the kubelet takes seconds to sync mounted volumes. A `rollout restart` issued immediately after may start pods with stale data. Add a short delay (5s) before restarting. ## Deployment Strategies - **RWO PVC + RollingUpdate = Deadlock.** New pod can't attach the volume while the old pod holds it. Use `strategy: Recreate` for single-replica deployments with RWO PVCs. - **SSA + strategy change conflict.** Switching from RollingUpdate to Recreate via ServerSideApply fails because SSA won't remove the old `rollingUpdate` field. Must patch the live resource first. ## Naming - `metadata.name` must be DNS-1035 compliant — no dots allowed. Replace dots with dashes (e.g., `oreillyit-nz` not `oreillyit.nz`). Label values CAN contain dots. ## Bootstrap Ordering Some components have chicken-and-egg dependencies: 1. CNI (e.g., Cilium) must be installed before anything else — nodes are NotReady without it 2. GitOps controller (e.g., ArgoCD) installed second 3. Root app applied last — the GitOps controller then "adopts" CLI-installed releases Manual bootstrap secrets (encryption keys, OIDC client secrets) must be documented as explicit steps. ## Network Policies - DNS egress for `toFQDNs` rules must use `toEndpoints` targeting kube-dns pods with `rules.dns` — this triggers the DNS proxy. Using `toCIDRSet` for DNS bypasses the proxy and FQDN rules never populate. - Cross-namespace policies need explicit namespace matching (e.g., `matchExpressions` on namespace label). - Always test from the actual consumer namespace, not same-namespace test pods. ## Probe Strategy - **Liveness vs readiness probes serve different purposes.** TCP checks confirm the process is listening (liveness). Exec/command checks confirm the application is ready to serve (readiness). Don't conflate them. - **Probes must match application host validation.** Applications that validate Host headers (e.g., Next.js `ALLOWED_HOSTS`) will reject probes sent to the pod IP. Set `httpGet.httpHeaders` with the expected Host value. - **Don't load credentials into liveness probes.** If readiness requires an authenticated check (e.g., `sqlcmd`), use a simple TCP check for liveness and reserve the authenticated check for readiness only. - **`timeoutSeconds: 1` is too tight for services with DB connections or async startup.** The default probe timeout is 1 second, which causes spurious failures when a service is initialising a connection pool or running async startup tasks. Use 3–5 seconds as a minimum for any service that touches a database or has an async lifespan handler. ## Init Container Patterns - **Writable config via init container + emptyDir.** When apps require writable directories but ConfigMaps are read-only, use an init container to copy config into an emptyDir volume that the main container mounts read-write. - **Privilege separation.** Init containers can run as root to create directories or set ownership, while the main container runs as a non-root UID. Prefer this over running the entire workload as root. - **Non-root images have hidden filesystem requirements.** Many modern images (e.g., MSSQL 2022, UID 10001) need writable directories beyond the obvious ones. Always check image documentation or `docker inspect` before writing manifests. ## StatefulSet Edge Cases - **CrashLoopBackOff pods won't auto-replace on spec update.** The StatefulSet controller won't delete and recreate a crashing pod when you update the spec — manual `kubectl delete pod` is required to force recreation. - **Immutable field diffs can deadlock auto-sync.** StatefulSet fields like `volumeClaimTemplates` are immutable after creation. GitOps controllers (ArgoCD) will show permanent OutOfSync if the desired state differs from the live immutable fields. Force sync or recreate the StatefulSet. - **SSA causes perpetual OutOfSync from defaulted fields.** Kubernetes defaults fields on StatefulSets (`persistentVolumeClaimRetentionPolicy`, `revisionHistoryLimit`, `updateStrategy.rollingUpdate.partition`) that aren't in the Helm template. With `ServerSideApply=true`, GitOps controllers see these as diffs and report OutOfSync even though the app is Healthy. The app functions correctly — this is cosmetic. Consider ArgoCD `ignoreDifferences` for these fields. ## GitOps: Imperative vs Declarative - **Never use imperative operations on GitOps-managed resources.** `kubectl rollout restart` adds annotations that conflict with the GitOps controller's desired state, causing permanent OutOfSync. Use declarative paths instead — update a configmap hash annotation in Git, or change a pod template label. - **ArgoCD reconciliation has latency.** New Application manifests don't appear immediately due to polling intervals. Use manual refresh annotations when automation needs immediate reconciliation. - **Self-managed GitOps controllers revert their own live config.** When the GitOps controller manages itself via its own Helm chart with `selfHeal: true` (e.g., ArgoCD reconciling `argocd-cm`/`argocd-rbac-cm`), direct `kubectl patch`/`apply` on its ConfigMaps is reverted within seconds. Change the controller's Helm `values.yaml` in the deploy repo (accounts, RBAC policy CSV, server settings) — never the live ConfigMaps. Applies to any account/RBAC/config change on a self-managed controller. - **A newly-created API account/token 403s until the config sync completes.** If you generate an API token for a GitOps-managed service account before the account exists in the live config (still mid-Helm-sync), the token returns 403. Wait for self-sync to complete, confirm the account is present, then generate the token. - **Grant the narrowest role for the job.** A service that only reads controller/app status (e.g., a preview-readiness check) needs a read-only role, not a sync/deploy role. Scope the GitOps API account to exactly what it does. - **Runtime-only "poke" annotations cause persistent OutOfSync.** Annotations added imperatively to trigger controller behaviour — e.g., a `force-sync`/`reconcile` annotation on an ExternalSecret to make External Secrets Operator refresh — are not present in Git, so the GitOps controller reports the resource `OutOfSync` indefinitely. Remove the annotation once it has done its job: `kubectl annotate -n -`. Applies to any "poke the controller" annotation not stored in the source manifest. ## PodSecurity Alignment - **Namespace PodSecurity labels must match container security contexts.** DinD, CSI drivers, and other privileged workloads need `pod-security.kubernetes.io/enforce: privileged` on their namespace. A `baseline` or `restricted` namespace silently blocks privileged pods. - **Document privileged namespace requirements.** When a workload needs elevated privileges, document the specific requirement (e.g., "Docker-in-Docker for CI builds") alongside the namespace label. - **Monitoring namespace requires privileged PodSecurity for node-exporter.** kube-prometheus-stack's node-exporter DaemonSet mounts host paths and uses `hostPID: true`. The monitoring namespace must be labelled `pod-security.kubernetes.io/enforce: privileged` or node-exporter pods will be silently blocked. Set this via GitOps namespace metadata — don't apply it manually or it will be reverted by the GitOps controller. ## Cilium Entity Identities (any pod-to-cluster-infrastructure egress) Applies to any pod that needs to reach cluster infrastructure — kube-apiserver, kubelets, node-exporter, host services. Not just Prometheus scraping. Common cases that trip on this: OpenBao/Vault calling TokenReview, External Secrets Operator authenticating to Vault via k8s auth, controllers calling subjectaccessreviews, anything that hits `https://kubernetes.default.svc`. | Target | Cilium entity | |---|---| | kube-apiserver (port 6443) | `kube-apiserver` | | Worker node kubelet / node-exporter | `remote-node` | | Same-node kubelet (DaemonSet on same node) | `host` | **Standard `NetworkPolicy` `ipBlock` CIDR rules do NOT match cluster node IPs.** Cluster nodes carry the Cilium `remote-node` (or `kube-apiserver`) identity, and `ipBlock` only matches IPs *without* a Cilium identity (i.e., off-cluster). Listing `10.X.X.0/24` for the control-plane subnet and expecting it to allow apiserver egress will silently fail — the rule is treated as a no-op and traffic is dropped. The CNI-native expression is `CiliumNetworkPolicy` with `toEntities: [...]`. The `namespaceSelector: kube-system` rule is also ineffective for kube-apiserver because the API server runs `hostNetwork: true` and is not selectable by namespace selector — its identity is host/remote-node/kube-apiserver, not the kube-system pod identity. Using the wrong entity (or the wrong policy kind) results in silent policy drops. Symptom for apiserver-bound traffic: HTTPS calls hang until the client times out (typically 30s for Go HTTP defaults), then the upstream returns a generic error like `permission denied`. Test with `cilium monitor --type drop` to confirm the drop is at L3, or — if you can `exec` into the pod — try `wget --timeout=5 https://kubernetes.default.svc/healthz` and see if it `Terminated`s. **Worked example (recurring incident pattern):** A secrets backend pod calls `auth/kubernetes/login` against the apiserver and hangs for exactly 30 seconds before returning a generic `permission denied`. The first hypothesis is RBAC or token misconfiguration — neither is the cause. Standard `NetworkPolicy ipBlock` rules listing the control-plane subnet don't match because cluster nodes carry the `kube-apiserver` / `remote-node` Cilium identity, and `ipBlock` only matches IPs without an identity. Fix is `CiliumNetworkPolicy` with `toEntities: [kube-apiserver]` on TCP 6443. Same pattern affects every workload that hits `https://kubernetes.default.svc` — External Secrets Operator, custom controllers, anything doing TokenReview / SubjectAccessReview. ## Every New K8s API Surface Needs Explicit RBAC in the Deploy Repo When a service starts touching a new K8s API resource (CRDs, custom controllers, ExternalSecrets, Jobs, Leases), add a namespace-scoped `Role` + `RoleBinding` to the deploy repo in the **same commit** as the code that touches the API. **Why this matters:** - Manual `kubectl apply` during development masks the gap because the local kubectl admin context bypasses RBAC. Production then 403s silently and stalls reconcile loops. - The GitOps reconciliation tries to apply the controller config but fails to read the underlying API; the symptom is a "permission denied" log line lost among thousands of other logs. - The RBAC commit lands AFTER the feature commit, leaving an interval where the deployed code is broken in any environment that doesn't have the dev's admin context. **Pattern:** every new API call requires either: 1. A namespace-scoped `Role` with the specific verbs (`get`, `list`, `watch`, `create`, `update`, `patch`, `delete` — only the ones actually used) on the specific resource 2. A `RoleBinding` to the service's `ServiceAccount` Bundle both into the same deploy-repo commit as the code change. A reviewer should be able to grep for any new K8s API call in the code diff and find the matching `Role` verbs in the manifest diff. **Common miss:** CRD custom resources need explicit `apiGroups` entries (e.g., `apiGroups: ["external-secrets.io"]`), not just `resources`. Forgetting the API group looks like a working Role definition but matches nothing. ## Kustomize Overlay `images:` Blocks Silently Override Base Tags Kustomize `images:` blocks in an overlay apply to the entire rendered manifest, including any images defined in `base/`. If the base defines `image: my-app:v1.0.0` and the overlay has an `images:` block targeting `my-app`, the overlay's `newTag` silently wins — even if you intended the base tag to remain. When deploying a new image version via Kustomize, always update the `images:` block in the overlay, not just the base manifest. If the overlay doesn't have an `images:` block, add one rather than editing the base tag directly. ## ArgoCD Source Type Detection - **ArgoCD auto-detects Kustomize.** When a source directory contains `kustomization.yaml`, ArgoCD runs Kustomize automatically. Adding an explicit `directory:` source type overrides this detection and causes ArgoCD to try applying `kustomization.yaml` as a raw K8s resource, which fails with schema errors. Remove explicit directory source types from Kustomize sources. - **Credential template URL-prefix must match exactly.** ArgoCD repo-creds secrets use URL prefix matching. When migrating Git server URLs (hostname, protocol, or port changes), update the credential template to match the new prefix. Stale credentials cause "authentication required" errors on all apps using that prefix. ## Miscellaneous - `enableServiceLinks: false` may be needed when K8s-injected service env vars conflict with app config (e.g., Authelia interprets `AUTHELIA_*` service vars as configuration). A related symptom: pytest test collection fails with errors like `PORT=tcp://10.96.0.1:443` — K8s injects `_PORT` as a full TCP URI, which many frameworks try to parse as an integer and crash. Setting `enableServiceLinks: false` on the pod removes all injected service env vars and resolves this class of error. - Proxmox VM names must match K8s node hostnames for cloud controller manager integration. - Metrics-server on Talos needs `--kubelet-insecure-tls` (self-signed kubelet certs). ## ArgoCD SSA + StatefulSet volumeClaimTemplates = Perpetual OutOfSync Kubernetes injects `apiVersion` and `kind` fields into StatefulSet `volumeClaimTemplates` on apply. These don't exist in source manifests, causing ArgoCD with ServerSideApply to report perpetual OutOfSync. Fix: add `ignoreDifferences` on the ArgoCD Application targeting `.spec.volumeClaimTemplates[]?.apiVersion` and `.spec.volumeClaimTemplates[]?.kind` (using `jqPathExpressions`), plus `RespectIgnoreDifferences=true` in syncOptions. ## ArgoCD SSA May Not Detect ConfigMap Data Changes ArgoCD with ServerSideApply sometimes fails to detect changes to ConfigMap `data` values, reporting "Synced + Healthy" while the live ConfigMap has stale content. Root cause: SSA field ownership conflicts between Helm's managed fields and a prior `kubectl apply` annotation. After syncing ConfigMaps managed by Helm+SSA, verify content with `kubectl get cm -o jsonpath='{.data.}'`. ## CSI VolumeAttachment Stuck After Hot-Plug Failure CSI hot-plug of storage devices can fail silently — the VolumeAttachment object says `attached: true` but the device never appeared on the node. Pods get stuck in `ContainerCreating` with "device not found." Fix: delete the stale VolumeAttachment (`kubectl delete volumeattachment `). The CSI driver recreates it and retries the attach. **QMP-timeout leading indicator for hypervisor CSI hotplug failures.** When a hypervisor-backed CSI driver (Proxmox, vSphere) returns `ControllerPublishVolume` "published" but the block device never appears in the guest (`/dev/disk/by-id/wwn-0x...` absent) and the pod stays in `ContainerCreating`, check the hypervisor logs for a QMP command timeout during the preceding unpublish (e.g. `qmp command 'query-pci' failed - got timeout`). This points to an unstable backing VM/host, not a Kubernetes bug. Short-term workaround: add `nodeAffinity` with `NotIn: ` on the deployment to steer the workload off the flaky VM while the hypervisor-side QMP stability is investigated. **Targeted SCSI scan instead of full-bus rescan when hotplugged disks don't appear.** After a CSI hotplug, forcing a device rescan with a full-bus wildcard (`echo "- - -" > /sys/class/scsi_host/host*/scan`) hangs on some controllers. Use a targeted single-LUN scan instead: `echo "0 0 " > /sys/class/scsi_host/host/scan` to probe a specific LUN without blocking the whole bus. ## Delete and Recreate ArgoCD Apps on Source Type Changes When changing an ArgoCD Application's source type (e.g., multi-source Helm to single-source Kustomize), the repo-server may serve cached manifests from the old configuration, and old Helm hook resources become ghost entries that block deletion via finalizers. Delete the Application entirely and let the root app recreate it rather than patching source types in-place. ## etcd on Slow Storage Requires Timeout Tuning etcd requires sub-10ms fsync for stable operation. On slow storage (HDD, network-attached, overloaded SSD), default timeouts (heartbeat 100ms, election 1000ms) cause leader election flapping, cascading scheduler/controller-manager restarts, and widespread probe failures. Symptoms: "leader failed to send out heartbeat on time", "apply request took too long." Mitigation: increase heartbeat-interval (e.g., 500ms) and election-timeout (e.g., 5000ms). Fix: move etcd to SSD storage. Periodic defrag also helps. ## CrashLoopBackOff Delays New Image Pickup After CI builds a fix for a crashing pod, the CrashLoopBackOff exponential backoff (up to 5 minutes) means the kubelet won't pull the new image until the next retry window. Run `kubectl rollout restart deployment/` immediately after CI completes to create a fresh pod instead of waiting. ## K8s Secret Volumes Are Read-Only with Root Ownership K8s Secret volume mounts are read-only — you cannot write or modify files in them. Files are owned by root regardless of fsGroup settings. Non-root processes need `defaultMode: 0444` (world-readable) to access the files. Additionally, mounting a Secret at a parent path shadows any other Secret mounts at child paths — mount each Secret at its own non-overlapping path. ## K8s Env Var Size Limits for Payloads Kubernetes has a hard limit on environment variable sizes (~228KB base64). Large payloads embedded as env vars cause containers to crash with exit 255 and zero logs. For inter-task artifact passing, use git branches or mounted volumes instead of env var payloads. ## Non-Blocking Registration in FastAPI Lifespan Handlers Blocking operations (external API calls, service registration) in application lifespan handlers prevent the HTTP server from starting. K8s liveness probes fail and the pod enters CrashLoopBackOff before the operation completes. Use background tasks (e.g., `asyncio.create_task`) for registration so health endpoints respond immediately while registration happens asynchronously. This applies to any K8s-deployed app framework with startup hooks (FastAPI, Flask, etc.). ## Verify Live Cluster State vs Deploy Repo Before Planning Changes GitOps controllers (ArgoCD, Flux) preserve fields added by manual `kubectl patch`/`apply` when those fields aren't in the deploy repo — unknown fields are not removed unless the controller sees a conflicting managed field. Symptom: live ConfigMap/IngressRoute has values not in Git, causing "it works differently than the manifests say" debugging. Before changing GitOps-managed resources, diff `kubectl get -o yaml` against the deploy repo and add explicit values to Git so subsequent syncs reset any manual drift. ## Force-Delete Pods Stuck Terminating After Node Disruption After power cut, kernel panic, or abrupt node loss, pods can be stuck in Terminating indefinitely (observed 22h+). The owning controller (Deployment, StatefulSet, ArgoCD application-controller) is blocked from creating a replacement, so upstream symptoms look like "GitOps stuck on old commit" or "service unreachable". Fix: `kubectl delete pod --grace-period=0 --force`. The controller recreates immediately and reconciliation resumes. ## CNI L2 LoadBalancer Announcements: Use externalTrafficPolicy Cluster With L2-announced LoadBalancer IPs (Cilium, MetalLB), `externalTrafficPolicy: Local` silently drops packets whenever the node winning the ARP lease doesn't run a backend pod — only that node holds a BPF/iptables LB entry. Use `externalTrafficPolicy: Cluster` and recover source IP at L7 (X-Forwarded-For, Proxy Protocol) instead. ## Restart CNI Agents After Agent-Affecting Config Changes CNI Helm values that land in the agent ConfigMap (L2 announcements, Hubble, envoy features) do not take effect until agent pods restart — the agent logs a "config drift" warning but keeps running the old config. After changing agent-affecting values, `kubectl rollout restart daemonset/` then `kubectl rollout restart deployment/`. ## Upgrade Storage-Consuming Nodes Sequentially, Not Concurrently Rolling upgrades that reboot multiple nodes concurrently can race external CSI controllers (Proxmox, vSphere, any hypervisor plugin doing hotplug). Parallel `ControllerPublishVolume`/`Unpublish` calls leave VolumeAttachments attached to the wrong VM or in a state where `attached: true` but the device is absent. Wait for each node Ready and CSI pods stable before upgrading the next; verify with `kubectl get volumeattachment`. ## Split Multi-Host IngressRoutes With Separate TLS Secrets A Traefik IngressRoute using `Host(a.example) || Host(b.example)` can only reference one `tls.secretName`; the second domain silently falls back to Traefik's self-signed default cert. Split into one IngressRoute per Host match with its own `tls.secretName`. Generalises to any ingress controller pairing a single TLS secret per ingress object. ## Delete-and-Recreate, Don't Patch, When Adopting Manually-Applied Resources into GitOps When a resource was first created with `kubectl apply` and then placed under GitOps ServerSideApply management, stale field-manager metadata causes perpetual OutOfSync that patching cannot resolve. Fix: `kubectl delete` the resource and let the GitOps controller recreate it with clean field ownership. Applies to any SSA-managed CRD adoption. ## etcd extraArgs Changes Require a Node Reboot on Immutable-OS Distros On immutable-OS distros (Talos, Bottlerocket, Flatcar), etcd runs as a system service. Patching machine config with `cluster.etcd.extraArgs` reports "Applied without reboot" but etcd keeps old args until the process restarts — which only happens on a full node reboot. After etcd flag changes, roll the control plane (non-leader first, leader last) and verify with `etcd status`/logs. ## etcd Defrag Reclaims Space from Deleted Keys etcd does not auto-reclaim space from deleted keys; the DB grows over time and hurts fsync latency on slow disks. Periodic `etcdctl defrag` (or Talos `etcd defrag`) reclaims 30-50% on typical clusters. Run on non-leader members first, leader last. Especially important on HDD or contended SSD. ## Hard-Reset Immutable-OS Nodes Stuck in Kernel-Level Boot Immutable-OS API reboots (e.g., `talosctl reboot`) require the OS API running in userspace. When a node is stuck pre-userspace — XFS quotacheck after unclean shutdown, fsck, long kernel init — the API is unreachable. Use hypervisor-level hard reset (`qm reset`, `virsh reset`, cloud provider stop/start) to force a clean boot; kernel-level recovery usually completes in seconds. ## Reconciliation Controller Pattern: Pure Diff, I/O Reconciler When writing a GitOps reconciliation controller, split into a pure function (desired vs actual → DiffResult, no I/O) and a separate reconciler class that handles all side effects (API calls, safety limits, logging). The pure diff is testable with zero mocks; the reconciler is mocked at its I/O boundary. Clean architecture for any controller reconciling declarative state against an external API. ## Controller Safety: Manage Only Declared Resources by Default A reconciliation controller should touch only resources explicitly declared in its source-of-truth config; unmanaged resources should be logged but never deleted. Avoid "bulk replace" APIs that atomically overwrite everything — prefer per-record create/update/delete so incomplete declarations can't wipe records (NS, SOA, operator-managed). Design opt-in flags (`managed: all`, `conflict: alert|automatic`) from day one. ## Webhook-Triggered Reconciliation with Token Auth Pair periodic reconciliation with an authenticated POST `/reconcile` endpoint so push events can trigger immediate sync. Use a 32+ char Bearer token with constant-time comparison, fail-closed (return 501) if the token is not configured. Avoids worst-case polling latency when a human just committed. ## kubernetes-py CustomObjectsApi: SSA Requires a Dedicated ApiClient `CustomObjectsApi.patch_namespaced_custom_object(force=True)` fails with HTTP 422 on kubernetes-py v35 (`PatchOptions.meta.k8s.io is invalid: force: Forbidden: may not be specified for non-apply patch`). The default Content-Type is `application/merge-patch+json`; `force` is only valid on real server-side applies (`application/apply-patch+yaml`). The `_content_type` kwarg that older docs reference is not exposed in v35. Workaround: build a dedicated `ApiClient` and `set_default_header("Content-Type", "application/apply-patch+yaml")` on it; pass that client to a separate `CustomObjectsApi` used only for SSA patches. Reads/deletes use the default client (no body, default Content-Type harmless). The bug is silent in tests because mocks accept any kwargs — only a real apiserver round-trip surfaces it. ## `kubectl debug node/` Fails Under Enforced PodSecurity In namespaces/clusters enforcing PodSecurity `baseline` or `restricted`, `kubectl debug node/...` is rejected because its debug pod uses `hostPID` and `hostPath` (a baseline violation). Workaround: manually create a debug pod in a `privileged`-labelled namespace (e.g. the CSI driver's namespace) with `nodeSelector: kubernetes.io/hostname: ` and `securityContext.privileged: true`, rather than relying on `kubectl debug node`. ## `kubectl apply` of a New Image Tag May Not Roll Pods Re-applying a Deployment with a bumped image tag does not reliably trigger a new rollout — the in-cluster image reference can stay cached, and `kubectl apply --force` does not fix it. When bumping an image version imperatively, `kubectl delete deployment ` before re-applying to guarantee a fresh pull. (In GitOps flows, prefer a digest pin or a template-hash annotation change; this delete-then-apply pattern is for imperative/dev workflows only.) ## A Correct CiliumNetworkPolicy Egress Rule Won't Help If Ingress Is Gated by a Plain NetworkPolicy When a client pod times out reaching a service despite a correct `CiliumNetworkPolicy` egress rule on the client side, check the target's **ingress** policy — it may be a standard Kubernetes `NetworkPolicy` (not Cilium) that whitelists only specific source namespaces. Cilium and plain NetworkPolicy coexist and are additive; both directions must permit the flow. The ingress policy is often owned by the deploy repo, separate from the application and cluster-bootstrap repos, so grep there first. When debugging cross-namespace connectivity, enumerate both the client's egress rules and every ingress policy selecting the target pod.