best-practices/kubernetes.md

# Kubernetes Patterns

## Volume Mounts

- **Avoid `subPath` volume mounts** for Secrets and ConfigMaps. The kubelet does not auto-update `subPath` mounts when the source changes — the pod must be restarted. Use directory mounts instead and adjust the application's config path.
- **Secret volume propagation is async.** After updating a Secret, the kubelet takes seconds to sync mounted volumes. A `rollout restart` issued immediately after may start pods with stale data. Add a short delay (5s) before restarting.

## Deployment Strategies

- **RWO PVC + RollingUpdate = Deadlock.** New pod can't attach the volume while the old pod holds it. Use `strategy: Recreate` for single-replica deployments with RWO PVCs.
- **SSA + strategy change conflict.** Switching from RollingUpdate to Recreate via ServerSideApply fails because SSA won't remove the old `rollingUpdate` field. Must patch the live resource first.

## Naming

- `metadata.name` must be DNS-1035 compliant — no dots allowed. Replace dots with dashes (e.g., `oreillyit-nz` not `oreillyit.nz`). Label values CAN contain dots.

## Bootstrap Ordering

Some components have chicken-and-egg dependencies:
1. CNI (e.g., Cilium) must be installed before anything else — nodes are NotReady without it
2. GitOps controller (e.g., ArgoCD) installed second
3. Root app applied last — the GitOps controller then "adopts" CLI-installed releases

Manual bootstrap secrets (encryption keys, OIDC client secrets) must be documented as explicit steps.

## Network Policies

- DNS egress for `toFQDNs` rules must use `toEndpoints` targeting kube-dns pods with `rules.dns` — this triggers the DNS proxy. Using `toCIDRSet` for DNS bypasses the proxy and FQDN rules never populate.
- Cross-namespace policies need explicit namespace matching (e.g., `matchExpressions` on namespace label).
- Always test from the actual consumer namespace, not same-namespace test pods.

## Probe Strategy

- **Liveness vs readiness probes serve different purposes.** TCP checks confirm the process is listening (liveness). Exec/command checks confirm the application is ready to serve (readiness). Don't conflate them.
- **Probes must match application host validation.** Applications that validate Host headers (e.g., Next.js `ALLOWED_HOSTS`) will reject probes sent to the pod IP. Set `httpGet.httpHeaders` with the expected Host value.
- **Don't load credentials into liveness probes.** If readiness requires an authenticated check (e.g., `sqlcmd`), use a simple TCP check for liveness and reserve the authenticated check for readiness only.
- **`timeoutSeconds: 1` is too tight for services with DB connections or async startup.** The default probe timeout is 1 second, which causes spurious failures when a service is initialising a connection pool or running async startup tasks. Use 3–5 seconds as a minimum for any service that touches a database or has an async lifespan handler.

## Init Container Patterns

- **Writable config via init container + emptyDir.** When apps require writable directories but ConfigMaps are read-only, use an init container to copy config into an emptyDir volume that the main container mounts read-write.
- **Privilege separation.** Init containers can run as root to create directories or set ownership, while the main container runs as a non-root UID. Prefer this over running the entire workload as root.
- **Non-root images have hidden filesystem requirements.** Many modern images (e.g., MSSQL 2022, UID 10001) need writable directories beyond the obvious ones. Always check image documentation or `docker inspect` before writing manifests.

## StatefulSet Edge Cases

- **CrashLoopBackOff pods won't auto-replace on spec update.** The StatefulSet controller won't delete and recreate a crashing pod when you update the spec — manual `kubectl delete pod` is required to force recreation.
- **Immutable field diffs can deadlock auto-sync.** StatefulSet fields like `volumeClaimTemplates` are immutable after creation. GitOps controllers (ArgoCD) will show permanent OutOfSync if the desired state differs from the live immutable fields. Force sync or recreate the StatefulSet.
- **SSA causes perpetual OutOfSync from defaulted fields.** Kubernetes defaults fields on StatefulSets (`persistentVolumeClaimRetentionPolicy`, `revisionHistoryLimit`, `updateStrategy.rollingUpdate.partition`) that aren't in the Helm template. With `ServerSideApply=true`, GitOps controllers see these as diffs and report OutOfSync even though the app is Healthy. The app functions correctly — this is cosmetic. Consider ArgoCD `ignoreDifferences` for these fields.

## GitOps: Imperative vs Declarative

- **Never use imperative operations on GitOps-managed resources.** `kubectl rollout restart` adds annotations that conflict with the GitOps controller's desired state, causing permanent OutOfSync. Use declarative paths instead — update a configmap hash annotation in Git, or change a pod template label.
- **ArgoCD reconciliation has latency.** New Application manifests don't appear immediately due to polling intervals. Use manual refresh annotations when automation needs immediate reconciliation.

## PodSecurity Alignment

- **Namespace PodSecurity labels must match container security contexts.** DinD, CSI drivers, and other privileged workloads need `pod-security.kubernetes.io/enforce: privileged` on their namespace. A `baseline` or `restricted` namespace silently blocks privileged pods.
- **Document privileged namespace requirements.** When a workload needs elevated privileges, document the specific requirement (e.g., "Docker-in-Docker for CI builds") alongside the namespace label.
- **Monitoring namespace requires privileged PodSecurity for node-exporter.** kube-prometheus-stack's node-exporter DaemonSet mounts host paths and uses `hostPID: true`. The monitoring namespace must be labelled `pod-security.kubernetes.io/enforce: privileged` or node-exporter pods will be silently blocked. Set this via GitOps namespace metadata — don't apply it manually or it will be reverted by the GitOps controller.

## Cilium Entity Identities for Monitoring Scraping

When writing Cilium network policies to allow Prometheus to scrape targets, the correct entity identity depends on the node role:

| Target | Cilium entity |
|---|---|
| kube-apiserver (port 6443) | `kube-apiserver` |
| Worker node kubelet / node-exporter | `remote-node` |
| Same-node kubelet (DaemonSet on same node) | `host` |

Using the wrong entity results in silent policy drops. Test with `cilium monitor --type drop` to identify mismatches.

## Kustomize Overlay `images:` Blocks Silently Override Base Tags

Kustomize `images:` blocks in an overlay apply to the entire rendered manifest, including any images defined in `base/`. If the base defines `image: my-app:v1.0.0` and the overlay has an `images:` block targeting `my-app`, the overlay's `newTag` silently wins — even if you intended the base tag to remain. When deploying a new image version via Kustomize, always update the `images:` block in the overlay, not just the base manifest. If the overlay doesn't have an `images:` block, add one rather than editing the base tag directly.

## ArgoCD Source Type Detection

- **ArgoCD auto-detects Kustomize.** When a source directory contains `kustomization.yaml`, ArgoCD runs Kustomize automatically. Adding an explicit `directory:` source type overrides this detection and causes ArgoCD to try applying `kustomization.yaml` as a raw K8s resource, which fails with schema errors. Remove explicit directory source types from Kustomize sources.
- **Credential template URL-prefix must match exactly.** ArgoCD repo-creds secrets use URL prefix matching. When migrating Git server URLs (hostname, protocol, or port changes), update the credential template to match the new prefix. Stale credentials cause "authentication required" errors on all apps using that prefix.

## Miscellaneous

- `enableServiceLinks: false` may be needed when K8s-injected service env vars conflict with app config (e.g., Authelia interprets `AUTHELIA_*` service vars as configuration). A related symptom: pytest test collection fails with errors like `PORT=tcp://10.96.0.1:443` — K8s injects `<SERVICE>_PORT` as a full TCP URI, which many frameworks try to parse as an integer and crash. Setting `enableServiceLinks: false` on the pod removes all injected service env vars and resolves this class of error.
- Proxmox VM names must match K8s node hostnames for cloud controller manager integration.
- Metrics-server on Talos needs `--kubelet-insecure-tls` (self-signed kubelet certs).

## ArgoCD SSA + StatefulSet volumeClaimTemplates = Perpetual OutOfSync

Kubernetes injects `apiVersion` and `kind` fields into StatefulSet `volumeClaimTemplates` on apply. These don't exist in source manifests, causing ArgoCD with ServerSideApply to report perpetual OutOfSync. Fix: add `ignoreDifferences` on the ArgoCD Application targeting `.spec.volumeClaimTemplates[]?.apiVersion` and `.spec.volumeClaimTemplates[]?.kind` (using `jqPathExpressions`), plus `RespectIgnoreDifferences=true` in syncOptions.

## ArgoCD SSA May Not Detect ConfigMap Data Changes

ArgoCD with ServerSideApply sometimes fails to detect changes to ConfigMap `data` values, reporting "Synced + Healthy" while the live ConfigMap has stale content. Root cause: SSA field ownership conflicts between Helm's managed fields and a prior `kubectl apply` annotation. After syncing ConfigMaps managed by Helm+SSA, verify content with `kubectl get cm <name> -o jsonpath='{.data.<key>}'`.

## CSI VolumeAttachment Stuck After Hot-Plug Failure

CSI hot-plug of storage devices can fail silently — the VolumeAttachment object says `attached: true` but the device never appeared on the node. Pods get stuck in `ContainerCreating` with "device not found." Fix: delete the stale VolumeAttachment (`kubectl delete volumeattachment <name>`). The CSI driver recreates it and retries the attach.

## Delete and Recreate ArgoCD Apps on Source Type Changes

When changing an ArgoCD Application's source type (e.g., multi-source Helm to single-source Kustomize), the repo-server may serve cached manifests from the old configuration, and old Helm hook resources become ghost entries that block deletion via finalizers. Delete the Application entirely and let the root app recreate it rather than patching source types in-place.

## etcd on Slow Storage Requires Timeout Tuning

etcd requires sub-10ms fsync for stable operation. On slow storage (HDD, network-attached, overloaded SSD), default timeouts (heartbeat 100ms, election 1000ms) cause leader election flapping, cascading scheduler/controller-manager restarts, and widespread probe failures. Symptoms: "leader failed to send out heartbeat on time", "apply request took too long." Mitigation: increase heartbeat-interval (e.g., 500ms) and election-timeout (e.g., 5000ms). Fix: move etcd to SSD storage. Periodic defrag also helps.

## CrashLoopBackOff Delays New Image Pickup

After CI builds a fix for a crashing pod, the CrashLoopBackOff exponential backoff (up to 5 minutes) means the kubelet won't pull the new image until the next retry window. Run `kubectl rollout restart deployment/<name>` immediately after CI completes to create a fresh pod instead of waiting.

## K8s Secret Volumes Are Read-Only with Root Ownership

K8s Secret volume mounts are read-only — you cannot write or modify files in them. Files are owned by root regardless of fsGroup settings. Non-root processes need `defaultMode: 0444` (world-readable) to access the files. Additionally, mounting a Secret at a parent path shadows any other Secret mounts at child paths — mount each Secret at its own non-overlapping path.

## K8s Env Var Size Limits for Payloads

Kubernetes has a hard limit on environment variable sizes (~228KB base64). Large payloads embedded as env vars cause containers to crash with exit 255 and zero logs. For inter-task artifact passing, use git branches or mounted volumes instead of env var payloads.

## Non-Blocking Registration in FastAPI Lifespan Handlers

Blocking operations (external API calls, service registration) in application lifespan handlers prevent the HTTP server from starting. K8s liveness probes fail and the pod enters CrashLoopBackOff before the operation completes. Use background tasks (e.g., `asyncio.create_task`) for registration so health endpoints respond immediately while registration happens asynchronously. This applies to any K8s-deployed app framework with startup hooks (FastAPI, Flask, etc.).

## Verify Live Cluster State vs Deploy Repo Before Planning Changes

GitOps controllers (ArgoCD, Flux) preserve fields added by manual `kubectl patch`/`apply` when those fields aren't in the deploy repo — unknown fields are not removed unless the controller sees a conflicting managed field. Symptom: live ConfigMap/IngressRoute has values not in Git, causing "it works differently than the manifests say" debugging. Before changing GitOps-managed resources, diff `kubectl get -o yaml` against the deploy repo and add explicit values to Git so subsequent syncs reset any manual drift.

## Force-Delete Pods Stuck Terminating After Node Disruption

After power cut, kernel panic, or abrupt node loss, pods can be stuck in Terminating indefinitely (observed 22h+). The owning controller (Deployment, StatefulSet, ArgoCD application-controller) is blocked from creating a replacement, so upstream symptoms look like "GitOps stuck on old commit" or "service unreachable". Fix: `kubectl delete pod <name> --grace-period=0 --force`. The controller recreates immediately and reconciliation resumes.

## CNI L2 LoadBalancer Announcements: Use externalTrafficPolicy Cluster

With L2-announced LoadBalancer IPs (Cilium, MetalLB), `externalTrafficPolicy: Local` silently drops packets whenever the node winning the ARP lease doesn't run a backend pod — only that node holds a BPF/iptables LB entry. Use `externalTrafficPolicy: Cluster` and recover source IP at L7 (X-Forwarded-For, Proxy Protocol) instead.

## Restart CNI Agents After Agent-Affecting Config Changes

CNI Helm values that land in the agent ConfigMap (L2 announcements, Hubble, envoy features) do not take effect until agent pods restart — the agent logs a "config drift" warning but keeps running the old config. After changing agent-affecting values, `kubectl rollout restart daemonset/<cni-agent>` then `kubectl rollout restart deployment/<cni-operator>`.

## Upgrade Storage-Consuming Nodes Sequentially, Not Concurrently

Rolling upgrades that reboot multiple nodes concurrently can race external CSI controllers (Proxmox, vSphere, any hypervisor plugin doing hotplug). Parallel `ControllerPublishVolume`/`Unpublish` calls leave VolumeAttachments attached to the wrong VM or in a state where `attached: true` but the device is absent. Wait for each node Ready and CSI pods stable before upgrading the next; verify with `kubectl get volumeattachment`.

## Split Multi-Host IngressRoutes With Separate TLS Secrets

A Traefik IngressRoute using `Host(a.example) || Host(b.example)` can only reference one `tls.secretName`; the second domain silently falls back to Traefik's self-signed default cert. Split into one IngressRoute per Host match with its own `tls.secretName`. Generalises to any ingress controller pairing a single TLS secret per ingress object.

## Delete-and-Recreate, Don't Patch, When Adopting Manually-Applied Resources into GitOps

When a resource was first created with `kubectl apply` and then placed under GitOps ServerSideApply management, stale field-manager metadata causes perpetual OutOfSync that patching cannot resolve. Fix: `kubectl delete` the resource and let the GitOps controller recreate it with clean field ownership. Applies to any SSA-managed CRD adoption.

## etcd extraArgs Changes Require a Node Reboot on Immutable-OS Distros

On immutable-OS distros (Talos, Bottlerocket, Flatcar), etcd runs as a system service. Patching machine config with `cluster.etcd.extraArgs` reports "Applied without reboot" but etcd keeps old args until the process restarts — which only happens on a full node reboot. After etcd flag changes, roll the control plane (non-leader first, leader last) and verify with `etcd status`/logs.

## etcd Defrag Reclaims Space from Deleted Keys

etcd does not auto-reclaim space from deleted keys; the DB grows over time and hurts fsync latency on slow disks. Periodic `etcdctl defrag` (or Talos `etcd defrag`) reclaims 30-50% on typical clusters. Run on non-leader members first, leader last. Especially important on HDD or contended SSD.

## Hard-Reset Immutable-OS Nodes Stuck in Kernel-Level Boot

Immutable-OS API reboots (e.g., `talosctl reboot`) require the OS API running in userspace. When a node is stuck pre-userspace — XFS quotacheck after unclean shutdown, fsck, long kernel init — the API is unreachable. Use hypervisor-level hard reset (`qm reset`, `virsh reset`, cloud provider stop/start) to force a clean boot; kernel-level recovery usually completes in seconds.

## Reconciliation Controller Pattern: Pure Diff, I/O Reconciler

When writing a GitOps reconciliation controller, split into a pure function (desired vs actual → DiffResult, no I/O) and a separate reconciler class that handles all side effects (API calls, safety limits, logging). The pure diff is testable with zero mocks; the reconciler is mocked at its I/O boundary. Clean architecture for any controller reconciling declarative state against an external API.

## Controller Safety: Manage Only Declared Resources by Default

A reconciliation controller should touch only resources explicitly declared in its source-of-truth config; unmanaged resources should be logged but never deleted. Avoid "bulk replace" APIs that atomically overwrite everything — prefer per-record create/update/delete so incomplete declarations can't wipe records (NS, SOA, operator-managed). Design opt-in flags (`managed: all`, `conflict: alert|automatic`) from day one.

## Webhook-Triggered Reconciliation with Token Auth

Pair periodic reconciliation with an authenticated POST `/reconcile` endpoint so push events can trigger immediate sync. Use a 32+ char Bearer token with constant-time comparison, fail-closed (return 501) if the token is not configured. Avoids worst-case polling latency when a human just committed.