claude-foundations/best-practices/kubernetes.md

Kubernetes Patterns

Volume Mounts

• Avoid subPath volume mounts for Secrets and ConfigMaps. The kubelet does not auto-update subPath mounts when the source changes — the pod must be restarted. Use directory mounts instead and adjust the application's config path.
• Secret volume propagation is async. After updating a Secret, the kubelet takes seconds to sync mounted volumes. A rollout restart issued immediately after may start pods with stale data. Add a short delay (5s) before restarting.

Deployment Strategies

• RWO PVC + RollingUpdate = Deadlock. New pod can't attach the volume while the old pod holds it. Use strategy: Recreate for single-replica deployments with RWO PVCs.
• SSA + strategy change conflict. Switching from RollingUpdate to Recreate via ServerSideApply fails because SSA won't remove the old rollingUpdate field. Must patch the live resource first.

Naming

• metadata.name must be DNS-1035 compliant — no dots allowed. Replace dots with dashes (e.g., oreillyit-nz not oreillyit.nz). Label values CAN contain dots.

Bootstrap Ordering

Some components have chicken-and-egg dependencies:

1. CNI (e.g., Cilium) must be installed before anything else — nodes are NotReady without it
2. GitOps controller (e.g., ArgoCD) installed second
3. Root app applied last — the GitOps controller then "adopts" CLI-installed releases

Manual bootstrap secrets (encryption keys, OIDC client secrets) must be documented as explicit steps.

Network Policies

• DNS egress for toFQDNs rules must use toEndpoints targeting kube-dns pods with rules.dns — this triggers the DNS proxy. Using toCIDRSet for DNS bypasses the proxy and FQDN rules never populate.
• Cross-namespace policies need explicit namespace matching (e.g., matchExpressions on namespace label).
• Always test from the actual consumer namespace, not same-namespace test pods.

Probe Strategy

• Liveness vs readiness probes serve different purposes. TCP checks confirm the process is listening (liveness). Exec/command checks confirm the application is ready to serve (readiness). Don't conflate them.
• Probes must match application host validation. Applications that validate Host headers (e.g., Next.js ALLOWED_HOSTS) will reject probes sent to the pod IP. Set httpGet.httpHeaders with the expected Host value.
• Don't load credentials into liveness probes. If readiness requires an authenticated check (e.g., sqlcmd), use a simple TCP check for liveness and reserve the authenticated check for readiness only.

Init Container Patterns

• Writable config via init container + emptyDir. When apps require writable directories but ConfigMaps are read-only, use an init container to copy config into an emptyDir volume that the main container mounts read-write.
• Privilege separation. Init containers can run as root to create directories or set ownership, while the main container runs as a non-root UID. Prefer this over running the entire workload as root.
• Non-root images have hidden filesystem requirements. Many modern images (e.g., MSSQL 2022, UID 10001) need writable directories beyond the obvious ones. Always check image documentation or docker inspect before writing manifests.

StatefulSet Edge Cases

• CrashLoopBackOff pods won't auto-replace on spec update. The StatefulSet controller won't delete and recreate a crashing pod when you update the spec — manual kubectl delete pod is required to force recreation.
• Immutable field diffs can deadlock auto-sync. StatefulSet fields like volumeClaimTemplates are immutable after creation. GitOps controllers (ArgoCD) will show permanent OutOfSync if the desired state differs from the live immutable fields. Force sync or recreate the StatefulSet.

GitOps: Imperative vs Declarative

• Never use imperative operations on GitOps-managed resources. kubectl rollout restart adds annotations that conflict with the GitOps controller's desired state, causing permanent OutOfSync. Use declarative paths instead — update a configmap hash annotation in Git, or change a pod template label.
• ArgoCD reconciliation has latency. New Application manifests don't appear immediately due to polling intervals. Use manual refresh annotations when automation needs immediate reconciliation.

PodSecurity Alignment

• Namespace PodSecurity labels must match container security contexts. DinD, CSI drivers, and other privileged workloads need pod-security.kubernetes.io/enforce: privileged on their namespace. A baseline or restricted namespace silently blocks privileged pods.
• Document privileged namespace requirements. When a workload needs elevated privileges, document the specific requirement (e.g., "Docker-in-Docker for CI builds") alongside the namespace label.

ArgoCD Source Type Detection

• ArgoCD auto-detects Kustomize. When a source directory contains kustomization.yaml, ArgoCD runs Kustomize automatically. Adding an explicit directory: source type overrides this detection and causes ArgoCD to try applying kustomization.yaml as a raw K8s resource, which fails with schema errors. Remove explicit directory source types from Kustomize sources.
• Credential template URL-prefix must match exactly. ArgoCD repo-creds secrets use URL prefix matching. When migrating Git server URLs (hostname, protocol, or port changes), update the credential template to match the new prefix. Stale credentials cause "authentication required" errors on all apps using that prefix.

Miscellaneous

• enableServiceLinks: false may be needed when K8s-injected service env vars conflict with app config (e.g., Authelia interprets AUTHELIA_* service vars as configuration).
• Proxmox VM names must match K8s node hostnames for cloud controller manager integration.
• Metrics-server on Talos needs --kubelet-insecure-tls (self-signed kubelet certs).