best-practices/kubernetes.md

Kubernetes Patterns

Volume Mounts

• Avoid subPath volume mounts for Secrets and ConfigMaps. The kubelet does not auto-update subPath mounts when the source changes — the pod must be restarted. Use directory mounts instead and adjust the application's config path.
• Secret volume propagation is async. After updating a Secret, the kubelet takes seconds to sync mounted volumes. A rollout restart issued immediately after may start pods with stale data. Add a short delay (5s) before restarting.

Deployment Strategies

• RWO PVC + RollingUpdate = Deadlock. New pod can't attach the volume while the old pod holds it. Use strategy: Recreate for single-replica deployments with RWO PVCs.
• SSA + strategy change conflict. Switching from RollingUpdate to Recreate via ServerSideApply fails because SSA won't remove the old rollingUpdate field. Must patch the live resource first.

Naming

• metadata.name must be DNS-1035 compliant — no dots allowed. Replace dots with dashes (e.g., oreillyit-nz not oreillyit.nz). Label values CAN contain dots.

Bootstrap Ordering

Some components have chicken-and-egg dependencies:

1. CNI (e.g., Cilium) must be installed before anything else — nodes are NotReady without it
2. GitOps controller (e.g., ArgoCD) installed second
3. Root app applied last — the GitOps controller then "adopts" CLI-installed releases

Manual bootstrap secrets (encryption keys, OIDC client secrets) must be documented as explicit steps.

Network Policies

• DNS egress for toFQDNs rules must use toEndpoints targeting kube-dns pods with rules.dns — this triggers the DNS proxy. Using toCIDRSet for DNS bypasses the proxy and FQDN rules never populate.
• Cross-namespace policies need explicit namespace matching (e.g., matchExpressions on namespace label).
• Always test from the actual consumer namespace, not same-namespace test pods.

Probe Strategy

• Liveness vs readiness probes serve different purposes. TCP checks confirm the process is listening (liveness). Exec/command checks confirm the application is ready to serve (readiness). Don't conflate them.
• Probes must match application host validation. Applications that validate Host headers (e.g., Next.js ALLOWED_HOSTS) will reject probes sent to the pod IP. Set httpGet.httpHeaders with the expected Host value.
• Don't load credentials into liveness probes. If readiness requires an authenticated check (e.g., sqlcmd), use a simple TCP check for liveness and reserve the authenticated check for readiness only.
• timeoutSeconds: 1 is too tight for services with DB connections or async startup. The default probe timeout is 1 second, which causes spurious failures when a service is initialising a connection pool or running async startup tasks. Use 3–5 seconds as a minimum for any service that touches a database or has an async lifespan handler.

Init Container Patterns

• Writable config via init container + emptyDir. When apps require writable directories but ConfigMaps are read-only, use an init container to copy config into an emptyDir volume that the main container mounts read-write.
• Privilege separation. Init containers can run as root to create directories or set ownership, while the main container runs as a non-root UID. Prefer this over running the entire workload as root.
• Non-root images have hidden filesystem requirements. Many modern images (e.g., MSSQL 2022, UID 10001) need writable directories beyond the obvious ones. Always check image documentation or docker inspect before writing manifests.

StatefulSet Edge Cases

• CrashLoopBackOff pods won't auto-replace on spec update. The StatefulSet controller won't delete and recreate a crashing pod when you update the spec — manual kubectl delete pod is required to force recreation.
• Immutable field diffs can deadlock auto-sync. StatefulSet fields like volumeClaimTemplates are immutable after creation. GitOps controllers (ArgoCD) will show permanent OutOfSync if the desired state differs from the live immutable fields. Force sync or recreate the StatefulSet.
• SSA causes perpetual OutOfSync from defaulted fields. Kubernetes defaults fields on StatefulSets (persistentVolumeClaimRetentionPolicy, revisionHistoryLimit, updateStrategy.rollingUpdate.partition) that aren't in the Helm template. With ServerSideApply=true, GitOps controllers see these as diffs and report OutOfSync even though the app is Healthy. The app functions correctly — this is cosmetic. Consider ArgoCD ignoreDifferences for these fields.

GitOps: Imperative vs Declarative

• Never use imperative operations on GitOps-managed resources. kubectl rollout restart adds annotations that conflict with the GitOps controller's desired state, causing permanent OutOfSync. Use declarative paths instead — update a configmap hash annotation in Git, or change a pod template label.
• ArgoCD reconciliation has latency. New Application manifests don't appear immediately due to polling intervals. Use manual refresh annotations when automation needs immediate reconciliation.
• Self-managed GitOps controllers revert their own live config. When the GitOps controller manages itself via its own Helm chart with selfHeal: true (e.g., ArgoCD reconciling argocd-cm/argocd-rbac-cm), direct kubectl patch/apply on its ConfigMaps is reverted within seconds. Change the controller's Helm values.yaml in the deploy repo (accounts, RBAC policy CSV, server settings) — never the live ConfigMaps. Applies to any account/RBAC/config change on a self-managed controller.
• A newly-created API account/token 403s until the config sync completes. If you generate an API token for a GitOps-managed service account before the account exists in the live config (still mid-Helm-sync), the token returns 403. Wait for self-sync to complete, confirm the account is present, then generate the token.
• Grant the narrowest role for the job. A service that only reads controller/app status (e.g., a preview-readiness check) needs a read-only role, not a sync/deploy role. Scope the GitOps API account to exactly what it does.
• Runtime-only "poke" annotations cause persistent OutOfSync. Annotations added imperatively to trigger controller behaviour — e.g., a force-sync/reconcile annotation on an ExternalSecret to make External Secrets Operator refresh — are not present in Git, so the GitOps controller reports the resource OutOfSync indefinitely. Remove the annotation once it has done its job: kubectl annotate <kind> <name> -n <ns> <annotation-key>-. Applies to any "poke the controller" annotation not stored in the source manifest.

PodSecurity Alignment

• Namespace PodSecurity labels must match container security contexts. DinD, CSI drivers, and other privileged workloads need pod-security.kubernetes.io/enforce: privileged on their namespace. A baseline or restricted namespace silently blocks privileged pods.
• Document privileged namespace requirements. When a workload needs elevated privileges, document the specific requirement (e.g., "Docker-in-Docker for CI builds") alongside the namespace label.
• Monitoring namespace requires privileged PodSecurity for node-exporter. kube-prometheus-stack's node-exporter DaemonSet mounts host paths and uses hostPID: true. The monitoring namespace must be labelled pod-security.kubernetes.io/enforce: privileged or node-exporter pods will be silently blocked. Set this via GitOps namespace metadata — don't apply it manually or it will be reverted by the GitOps controller.

Cilium Entity Identities (any pod-to-cluster-infrastructure egress)

Applies to any pod that needs to reach cluster infrastructure — kube-apiserver, kubelets, node-exporter, host services. Not just Prometheus scraping. Common cases that trip on this: OpenBao/Vault calling TokenReview, External Secrets Operator authenticating to Vault via k8s auth, controllers calling subjectaccessreviews, anything that hits https://kubernetes.default.svc.

Target	Cilium entity
kube-apiserver (port 6443)	kube-apiserver
Worker node kubelet / node-exporter	remote-node
Same-node kubelet (DaemonSet on same node)	host

Standard NetworkPolicy ipBlock CIDR rules do NOT match cluster node IPs. Cluster nodes carry the Cilium remote-node (or kube-apiserver) identity, and ipBlock only matches IPs without a Cilium identity (i.e., off-cluster). Listing 10.X.X.0/24 for the control-plane subnet and expecting it to allow apiserver egress will silently fail — the rule is treated as a no-op and traffic is dropped. The CNI-native expression is CiliumNetworkPolicy with toEntities: [...].

The namespaceSelector: kube-system rule is also ineffective for kube-apiserver because the API server runs hostNetwork: true and is not selectable by namespace selector — its identity is host/remote-node/kube-apiserver, not the kube-system pod identity.

Using the wrong entity (or the wrong policy kind) results in silent policy drops. Symptom for apiserver-bound traffic: HTTPS calls hang until the client times out (typically 30s for Go HTTP defaults), then the upstream returns a generic error like permission denied. Test with cilium monitor --type drop to confirm the drop is at L3, or — if you can exec into the pod — try wget --timeout=5 https://kubernetes.default.svc/healthz and see if it Terminateds.

Worked example (recurring incident pattern): A secrets backend pod calls auth/kubernetes/login against the apiserver and hangs for exactly 30 seconds before returning a generic permission denied. The first hypothesis is RBAC or token misconfiguration — neither is the cause. Standard NetworkPolicy ipBlock rules listing the control-plane subnet don't match because cluster nodes carry the kube-apiserver / remote-node Cilium identity, and ipBlock only matches IPs without an identity. Fix is CiliumNetworkPolicy with toEntities: [kube-apiserver] on TCP 6443. Same pattern affects every workload that hits https://kubernetes.default.svc — External Secrets Operator, custom controllers, anything doing TokenReview / SubjectAccessReview.

Every New K8s API Surface Needs Explicit RBAC in the Deploy Repo

When a service starts touching a new K8s API resource (CRDs, custom controllers, ExternalSecrets, Jobs, Leases), add a namespace-scoped Role + RoleBinding to the deploy repo in the same commit as the code that touches the API.

Why this matters:

• Manual kubectl apply during development masks the gap because the local kubectl admin context bypasses RBAC. Production then 403s silently and stalls reconcile loops.
• The GitOps reconciliation tries to apply the controller config but fails to read the underlying API; the symptom is a "permission denied" log line lost among thousands of other logs.
• The RBAC commit lands AFTER the feature commit, leaving an interval where the deployed code is broken in any environment that doesn't have the dev's admin context.

Pattern: every new API call requires either:

1. A namespace-scoped Role with the specific verbs (get, list, watch, create, update, patch, delete — only the ones actually used) on the specific resource
2. A RoleBinding to the service's ServiceAccount

Bundle both into the same deploy-repo commit as the code change. A reviewer should be able to grep for any new K8s API call in the code diff and find the matching Role verbs in the manifest diff.

Common miss: CRD custom resources need explicit apiGroups entries (e.g., apiGroups: ["external-secrets.io"]), not just resources. Forgetting the API group looks like a working Role definition but matches nothing.

Kustomize Overlay images: Blocks Silently Override Base Tags

Kustomize images: blocks in an overlay apply to the entire rendered manifest, including any images defined in base/. If the base defines image: my-app:v1.0.0 and the overlay has an images: block targeting my-app, the overlay's newTag silently wins — even if you intended the base tag to remain. When deploying a new image version via Kustomize, always update the images: block in the overlay, not just the base manifest. If the overlay doesn't have an images: block, add one rather than editing the base tag directly.

ArgoCD Source Type Detection

• ArgoCD auto-detects Kustomize. When a source directory contains kustomization.yaml, ArgoCD runs Kustomize automatically. Adding an explicit directory: source type overrides this detection and causes ArgoCD to try applying kustomization.yaml as a raw K8s resource, which fails with schema errors. Remove explicit directory source types from Kustomize sources.
• Credential template URL-prefix must match exactly. ArgoCD repo-creds secrets use URL prefix matching. When migrating Git server URLs (hostname, protocol, or port changes), update the credential template to match the new prefix. Stale credentials cause "authentication required" errors on all apps using that prefix.

Miscellaneous

• enableServiceLinks: false may be needed when K8s-injected service env vars conflict with app config (e.g., Authelia interprets AUTHELIA_* service vars as configuration). A related symptom: pytest test collection fails with errors like PORT=tcp://10.96.0.1:443 — K8s injects <SERVICE>_PORT as a full TCP URI, which many frameworks try to parse as an integer and crash. Setting enableServiceLinks: false on the pod removes all injected service env vars and resolves this class of error.
• Proxmox VM names must match K8s node hostnames for cloud controller manager integration.
• Metrics-server on Talos needs --kubelet-insecure-tls (self-signed kubelet certs).

ArgoCD SSA + StatefulSet volumeClaimTemplates = Perpetual OutOfSync

Kubernetes injects apiVersion and kind fields into StatefulSet volumeClaimTemplates on apply. These don't exist in source manifests, causing ArgoCD with ServerSideApply to report perpetual OutOfSync. Fix: add ignoreDifferences on the ArgoCD Application targeting .spec.volumeClaimTemplates[]?.apiVersion and .spec.volumeClaimTemplates[]?.kind (using jqPathExpressions), plus RespectIgnoreDifferences=true in syncOptions.

ArgoCD SSA May Not Detect ConfigMap Data Changes

ArgoCD with ServerSideApply sometimes fails to detect changes to ConfigMap data values, reporting "Synced + Healthy" while the live ConfigMap has stale content. Root cause: SSA field ownership conflicts between Helm's managed fields and a prior kubectl apply annotation. After syncing ConfigMaps managed by Helm+SSA, verify content with kubectl get cm <name> -o jsonpath='{.data.<key>}'.

CSI VolumeAttachment Stuck After Hot-Plug Failure

CSI hot-plug of storage devices can fail silently — the VolumeAttachment object says attached: true but the device never appeared on the node. Pods get stuck in ContainerCreating with "device not found." Fix: delete the stale VolumeAttachment (kubectl delete volumeattachment <name>). The CSI driver recreates it and retries the attach.

QMP-timeout leading indicator for hypervisor CSI hotplug failures. When a hypervisor-backed CSI driver (Proxmox, vSphere) returns ControllerPublishVolume "published" but the block device never appears in the guest (/dev/disk/by-id/wwn-0x... absent) and the pod stays in ContainerCreating, check the hypervisor logs for a QMP command timeout during the preceding unpublish (e.g. qmp command 'query-pci' failed - got timeout). This points to an unstable backing VM/host, not a Kubernetes bug. Short-term workaround: add nodeAffinity with NotIn: <bad-node> on the deployment to steer the workload off the flaky VM while the hypervisor-side QMP stability is investigated.

Targeted SCSI scan instead of full-bus rescan when hotplugged disks don't appear. After a CSI hotplug, forcing a device rescan with a full-bus wildcard (echo "- - -" > /sys/class/scsi_host/host*/scan) hangs on some controllers. Use a targeted single-LUN scan instead: echo "0 0 <lun>" > /sys/class/scsi_host/host<X>/scan to probe a specific LUN without blocking the whole bus.

Delete and Recreate ArgoCD Apps on Source Type Changes

When changing an ArgoCD Application's source type (e.g., multi-source Helm to single-source Kustomize), the repo-server may serve cached manifests from the old configuration, and old Helm hook resources become ghost entries that block deletion via finalizers. Delete the Application entirely and let the root app recreate it rather than patching source types in-place.

etcd on Slow Storage Requires Timeout Tuning

etcd requires sub-10ms fsync for stable operation. On slow storage (HDD, network-attached, overloaded SSD), default timeouts (heartbeat 100ms, election 1000ms) cause leader election flapping, cascading scheduler/controller-manager restarts, and widespread probe failures. Symptoms: "leader failed to send out heartbeat on time", "apply request took too long." Mitigation: increase heartbeat-interval (e.g., 500ms) and election-timeout (e.g., 5000ms). Fix: move etcd to SSD storage. Periodic defrag also helps.

CrashLoopBackOff Delays New Image Pickup

After CI builds a fix for a crashing pod, the CrashLoopBackOff exponential backoff (up to 5 minutes) means the kubelet won't pull the new image until the next retry window. Run kubectl rollout restart deployment/<name> immediately after CI completes to create a fresh pod instead of waiting.

K8s Secret Volumes Are Read-Only with Root Ownership

K8s Secret volume mounts are read-only — you cannot write or modify files in them. Files are owned by root regardless of fsGroup settings. Non-root processes need defaultMode: 0444 (world-readable) to access the files. Additionally, mounting a Secret at a parent path shadows any other Secret mounts at child paths — mount each Secret at its own non-overlapping path.

K8s Env Var Size Limits for Payloads

Kubernetes has a hard limit on environment variable sizes (~228KB base64). Large payloads embedded as env vars cause containers to crash with exit 255 and zero logs. For inter-task artifact passing, use git branches or mounted volumes instead of env var payloads.

Non-Blocking Registration in FastAPI Lifespan Handlers

Blocking operations (external API calls, service registration) in application lifespan handlers prevent the HTTP server from starting. K8s liveness probes fail and the pod enters CrashLoopBackOff before the operation completes. Use background tasks (e.g., asyncio.create_task) for registration so health endpoints respond immediately while registration happens asynchronously. This applies to any K8s-deployed app framework with startup hooks (FastAPI, Flask, etc.).

Verify Live Cluster State vs Deploy Repo Before Planning Changes

GitOps controllers (ArgoCD, Flux) preserve fields added by manual kubectl patch/apply when those fields aren't in the deploy repo — unknown fields are not removed unless the controller sees a conflicting managed field. Symptom: live ConfigMap/IngressRoute has values not in Git, causing "it works differently than the manifests say" debugging. Before changing GitOps-managed resources, diff kubectl get -o yaml against the deploy repo and add explicit values to Git so subsequent syncs reset any manual drift.

Force-Delete Pods Stuck Terminating After Node Disruption

After power cut, kernel panic, or abrupt node loss, pods can be stuck in Terminating indefinitely (observed 22h+). The owning controller (Deployment, StatefulSet, ArgoCD application-controller) is blocked from creating a replacement, so upstream symptoms look like "GitOps stuck on old commit" or "service unreachable". Fix: kubectl delete pod <name> --grace-period=0 --force. The controller recreates immediately and reconciliation resumes.

CNI L2 LoadBalancer Announcements: Use externalTrafficPolicy Cluster

With L2-announced LoadBalancer IPs (Cilium, MetalLB), externalTrafficPolicy: Local silently drops packets whenever the node winning the ARP lease doesn't run a backend pod — only that node holds a BPF/iptables LB entry. Use externalTrafficPolicy: Cluster and recover source IP at L7 (X-Forwarded-For, Proxy Protocol) instead.

Restart CNI Agents After Agent-Affecting Config Changes

CNI Helm values that land in the agent ConfigMap (L2 announcements, Hubble, envoy features) do not take effect until agent pods restart — the agent logs a "config drift" warning but keeps running the old config. After changing agent-affecting values, kubectl rollout restart daemonset/<cni-agent> then kubectl rollout restart deployment/<cni-operator>.

Upgrade Storage-Consuming Nodes Sequentially, Not Concurrently

Rolling upgrades that reboot multiple nodes concurrently can race external CSI controllers (Proxmox, vSphere, any hypervisor plugin doing hotplug). Parallel ControllerPublishVolume/Unpublish calls leave VolumeAttachments attached to the wrong VM or in a state where attached: true but the device is absent. Wait for each node Ready and CSI pods stable before upgrading the next; verify with kubectl get volumeattachment.

Split Multi-Host IngressRoutes With Separate TLS Secrets

A Traefik IngressRoute using Host(a.example) || Host(b.example) can only reference one tls.secretName; the second domain silently falls back to Traefik's self-signed default cert. Split into one IngressRoute per Host match with its own tls.secretName. Generalises to any ingress controller pairing a single TLS secret per ingress object.

Delete-and-Recreate, Don't Patch, When Adopting Manually-Applied Resources into GitOps

When a resource was first created with kubectl apply and then placed under GitOps ServerSideApply management, stale field-manager metadata causes perpetual OutOfSync that patching cannot resolve. Fix: kubectl delete the resource and let the GitOps controller recreate it with clean field ownership. Applies to any SSA-managed CRD adoption.

etcd extraArgs Changes Require a Node Reboot on Immutable-OS Distros

On immutable-OS distros (Talos, Bottlerocket, Flatcar), etcd runs as a system service. Patching machine config with cluster.etcd.extraArgs reports "Applied without reboot" but etcd keeps old args until the process restarts — which only happens on a full node reboot. After etcd flag changes, roll the control plane (non-leader first, leader last) and verify with etcd status/logs.

etcd Defrag Reclaims Space from Deleted Keys

etcd does not auto-reclaim space from deleted keys; the DB grows over time and hurts fsync latency on slow disks. Periodic etcdctl defrag (or Talos etcd defrag) reclaims 30-50% on typical clusters. Run on non-leader members first, leader last. Especially important on HDD or contended SSD.

Hard-Reset Immutable-OS Nodes Stuck in Kernel-Level Boot

Immutable-OS API reboots (e.g., talosctl reboot) require the OS API running in userspace. When a node is stuck pre-userspace — XFS quotacheck after unclean shutdown, fsck, long kernel init — the API is unreachable. Use hypervisor-level hard reset (qm reset, virsh reset, cloud provider stop/start) to force a clean boot; kernel-level recovery usually completes in seconds.

Reconciliation Controller Pattern: Pure Diff, I/O Reconciler

When writing a GitOps reconciliation controller, split into a pure function (desired vs actual → DiffResult, no I/O) and a separate reconciler class that handles all side effects (API calls, safety limits, logging). The pure diff is testable with zero mocks; the reconciler is mocked at its I/O boundary. Clean architecture for any controller reconciling declarative state against an external API.

Controller Safety: Manage Only Declared Resources by Default

A reconciliation controller should touch only resources explicitly declared in its source-of-truth config; unmanaged resources should be logged but never deleted. Avoid "bulk replace" APIs that atomically overwrite everything — prefer per-record create/update/delete so incomplete declarations can't wipe records (NS, SOA, operator-managed). Design opt-in flags (managed: all, conflict: alert|automatic) from day one.

Webhook-Triggered Reconciliation with Token Auth

Pair periodic reconciliation with an authenticated POST /reconcile endpoint so push events can trigger immediate sync. Use a 32+ char Bearer token with constant-time comparison, fail-closed (return 501) if the token is not configured. Avoids worst-case polling latency when a human just committed.

kubernetes-py CustomObjectsApi: SSA Requires a Dedicated ApiClient

CustomObjectsApi.patch_namespaced_custom_object(force=True) fails with HTTP 422 on kubernetes-py v35 (PatchOptions.meta.k8s.io is invalid: force: Forbidden: may not be specified for non-apply patch). The default Content-Type is application/merge-patch+json; force is only valid on real server-side applies (application/apply-patch+yaml). The _content_type kwarg that older docs reference is not exposed in v35. Workaround: build a dedicated ApiClient and set_default_header("Content-Type", "application/apply-patch+yaml") on it; pass that client to a separate CustomObjectsApi used only for SSA patches. Reads/deletes use the default client (no body, default Content-Type harmless). The bug is silent in tests because mocks accept any kwargs — only a real apiserver round-trip surfaces it.

kubectl debug node/<node> Fails Under Enforced PodSecurity

In namespaces/clusters enforcing PodSecurity baseline or restricted, kubectl debug node/... is rejected because its debug pod uses hostPID and hostPath (a baseline violation). Workaround: manually create a debug pod in a privileged-labelled namespace (e.g. the CSI driver's namespace) with nodeSelector: kubernetes.io/hostname: <node> and securityContext.privileged: true, rather than relying on kubectl debug node.

kubectl apply of a New Image Tag May Not Roll Pods

Re-applying a Deployment with a bumped image tag does not reliably trigger a new rollout — the in-cluster image reference can stay cached, and kubectl apply --force does not fix it. When bumping an image version imperatively, kubectl delete deployment <name> before re-applying to guarantee a fresh pull. (In GitOps flows, prefer a digest pin or a template-hash annotation change; this delete-then-apply pattern is for imperative/dev workflows only.)

A Correct CiliumNetworkPolicy Egress Rule Won't Help If Ingress Is Gated by a Plain NetworkPolicy

When a client pod times out reaching a service despite a correct CiliumNetworkPolicy egress rule on the client side, check the target's ingress policy — it may be a standard Kubernetes NetworkPolicy (not Cilium) that whitelists only specific source namespaces. Cilium and plain NetworkPolicy coexist and are additive; both directions must permit the flow. The ingress policy is often owned by the deploy repo, separate from the application and cluster-bootstrap repos, so grep there first. When debugging cross-namespace connectivity, enumerate both the client's egress rules and every ingress policy selecting the target pod.