# Kubernetes Patterns

## Volume Mounts

- **Avoid `subPath` volume mounts** for Secrets and ConfigMaps. The kubelet does not auto-update `subPath` mounts when the source changes — the pod must be restarted. Use directory mounts instead and adjust the application's config path.
- **Secret volume propagation is async.** After updating a Secret, the kubelet takes seconds to sync mounted volumes. A `rollout restart` issued immediately after may start pods with stale data. Add a short delay (5s) before restarting.

## Deployment Strategies

- **RWO PVC + RollingUpdate = Deadlock.** New pod can't attach the volume while the old pod holds it. Use `strategy: Recreate` for single-replica deployments with RWO PVCs.
- **SSA + strategy change conflict.** Switching from RollingUpdate to Recreate via ServerSideApply fails because SSA won't remove the old `rollingUpdate` field. Must patch the live resource first.

## Naming

- `metadata.name` must be DNS-1035 compliant — no dots allowed. Replace dots with dashes (e.g., `oreillyit-nz` not `oreillyit.nz`). Label values CAN contain dots.

## Bootstrap Ordering

Some components have chicken-and-egg dependencies:
1. CNI (e.g., Cilium) must be installed before anything else — nodes are NotReady without it
2. GitOps controller (e.g., ArgoCD) installed second
3. Root app applied last — the GitOps controller then "adopts" CLI-installed releases

Manual bootstrap secrets (encryption keys, OIDC client secrets) must be documented as explicit steps.

## Network Policies

- DNS egress for `toFQDNs` rules must use `toEndpoints` targeting kube-dns pods with `rules.dns` — this triggers the DNS proxy. Using `toCIDRSet` for DNS bypasses the proxy and FQDN rules never populate.
- Cross-namespace policies need explicit namespace matching (e.g., `matchExpressions` on namespace label).
- Always test from the actual consumer namespace, not same-namespace test pods.

## Probe Strategy

- **Liveness vs readiness probes serve different purposes.** TCP checks confirm the process is listening (liveness). Exec/command checks confirm the application is ready to serve (readiness). Don't conflate them.
- **Probes must match application host validation.** Applications that validate Host headers (e.g., Next.js `ALLOWED_HOSTS`) will reject probes sent to the pod IP. Set `httpGet.httpHeaders` with the expected Host value.
- **Don't load credentials into liveness probes.** If readiness requires an authenticated check (e.g., `sqlcmd`), use a simple TCP check for liveness and reserve the authenticated check for readiness only.

## Init Container Patterns

- **Writable config via init container + emptyDir.** When apps require writable directories but ConfigMaps are read-only, use an init container to copy config into an emptyDir volume that the main container mounts read-write.
- **Privilege separation.** Init containers can run as root to create directories or set ownership, while the main container runs as a non-root UID. Prefer this over running the entire workload as root.
- **Non-root images have hidden filesystem requirements.** Many modern images (e.g., MSSQL 2022, UID 10001) need writable directories beyond the obvious ones. Always check image documentation or `docker inspect` before writing manifests.

## StatefulSet Edge Cases

- **CrashLoopBackOff pods won't auto-replace on spec update.** The StatefulSet controller won't delete and recreate a crashing pod when you update the spec — manual `kubectl delete pod` is required to force recreation.
- **Immutable field diffs can deadlock auto-sync.** StatefulSet fields like `volumeClaimTemplates` are immutable after creation. GitOps controllers (ArgoCD) will show permanent OutOfSync if the desired state differs from the live immutable fields. Force sync or recreate the StatefulSet.

## GitOps: Imperative vs Declarative

- **Never use imperative operations on GitOps-managed resources.** `kubectl rollout restart` adds annotations that conflict with the GitOps controller's desired state, causing permanent OutOfSync. Use declarative paths instead — update a configmap hash annotation in Git, or change a pod template label.
- **ArgoCD reconciliation has latency.** New Application manifests don't appear immediately due to polling intervals. Use manual refresh annotations when automation needs immediate reconciliation.

## PodSecurity Alignment

- **Namespace PodSecurity labels must match container security contexts.** DinD, CSI drivers, and other privileged workloads need `pod-security.kubernetes.io/enforce: privileged` on their namespace. A `baseline` or `restricted` namespace silently blocks privileged pods.
- **Document privileged namespace requirements.** When a workload needs elevated privileges, document the specific requirement (e.g., "Docker-in-Docker for CI builds") alongside the namespace label.

## Miscellaneous

- `enableServiceLinks: false` may be needed when K8s-injected service env vars conflict with app config (e.g., Authelia interprets `AUTHELIA_*` service vars as configuration).
- Proxmox VM names must match K8s node hostnames for cloud controller manager integration.
- Metrics-server on Talos needs `--kubelet-insecure-tls` (self-signed kubelet certs).