e94417b896
Includes: spec-driven and test-driven development best practices, reproduce-before-fixing debugging workflow, require-plan-file hook, find-project-root script, session logs, memory files for decisions/ gotchas/process-lessons, and updates to existing best practice topics. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
64 lines
5.1 KiB
Markdown
64 lines
5.1 KiB
Markdown
# Kubernetes Patterns
|
|
|
|
## Volume Mounts
|
|
|
|
- **Avoid `subPath` volume mounts** for Secrets and ConfigMaps. The kubelet does not auto-update `subPath` mounts when the source changes — the pod must be restarted. Use directory mounts instead and adjust the application's config path.
|
|
- **Secret volume propagation is async.** After updating a Secret, the kubelet takes seconds to sync mounted volumes. A `rollout restart` issued immediately after may start pods with stale data. Add a short delay (5s) before restarting.
|
|
|
|
## Deployment Strategies
|
|
|
|
- **RWO PVC + RollingUpdate = Deadlock.** New pod can't attach the volume while the old pod holds it. Use `strategy: Recreate` for single-replica deployments with RWO PVCs.
|
|
- **SSA + strategy change conflict.** Switching from RollingUpdate to Recreate via ServerSideApply fails because SSA won't remove the old `rollingUpdate` field. Must patch the live resource first.
|
|
|
|
## Naming
|
|
|
|
- `metadata.name` must be DNS-1035 compliant — no dots allowed. Replace dots with dashes (e.g., `oreillyit-nz` not `oreillyit.nz`). Label values CAN contain dots.
|
|
|
|
## Bootstrap Ordering
|
|
|
|
Some components have chicken-and-egg dependencies:
|
|
1. CNI (e.g., Cilium) must be installed before anything else — nodes are NotReady without it
|
|
2. GitOps controller (e.g., ArgoCD) installed second
|
|
3. Root app applied last — the GitOps controller then "adopts" CLI-installed releases
|
|
|
|
Manual bootstrap secrets (encryption keys, OIDC client secrets) must be documented as explicit steps.
|
|
|
|
## Network Policies
|
|
|
|
- DNS egress for `toFQDNs` rules must use `toEndpoints` targeting kube-dns pods with `rules.dns` — this triggers the DNS proxy. Using `toCIDRSet` for DNS bypasses the proxy and FQDN rules never populate.
|
|
- Cross-namespace policies need explicit namespace matching (e.g., `matchExpressions` on namespace label).
|
|
- Always test from the actual consumer namespace, not same-namespace test pods.
|
|
|
|
## Probe Strategy
|
|
|
|
- **Liveness vs readiness probes serve different purposes.** TCP checks confirm the process is listening (liveness). Exec/command checks confirm the application is ready to serve (readiness). Don't conflate them.
|
|
- **Probes must match application host validation.** Applications that validate Host headers (e.g., Next.js `ALLOWED_HOSTS`) will reject probes sent to the pod IP. Set `httpGet.httpHeaders` with the expected Host value.
|
|
- **Don't load credentials into liveness probes.** If readiness requires an authenticated check (e.g., `sqlcmd`), use a simple TCP check for liveness and reserve the authenticated check for readiness only.
|
|
|
|
## Init Container Patterns
|
|
|
|
- **Writable config via init container + emptyDir.** When apps require writable directories but ConfigMaps are read-only, use an init container to copy config into an emptyDir volume that the main container mounts read-write.
|
|
- **Privilege separation.** Init containers can run as root to create directories or set ownership, while the main container runs as a non-root UID. Prefer this over running the entire workload as root.
|
|
- **Non-root images have hidden filesystem requirements.** Many modern images (e.g., MSSQL 2022, UID 10001) need writable directories beyond the obvious ones. Always check image documentation or `docker inspect` before writing manifests.
|
|
|
|
## StatefulSet Edge Cases
|
|
|
|
- **CrashLoopBackOff pods won't auto-replace on spec update.** The StatefulSet controller won't delete and recreate a crashing pod when you update the spec — manual `kubectl delete pod` is required to force recreation.
|
|
- **Immutable field diffs can deadlock auto-sync.** StatefulSet fields like `volumeClaimTemplates` are immutable after creation. GitOps controllers (ArgoCD) will show permanent OutOfSync if the desired state differs from the live immutable fields. Force sync or recreate the StatefulSet.
|
|
|
|
## GitOps: Imperative vs Declarative
|
|
|
|
- **Never use imperative operations on GitOps-managed resources.** `kubectl rollout restart` adds annotations that conflict with the GitOps controller's desired state, causing permanent OutOfSync. Use declarative paths instead — update a configmap hash annotation in Git, or change a pod template label.
|
|
- **ArgoCD reconciliation has latency.** New Application manifests don't appear immediately due to polling intervals. Use manual refresh annotations when automation needs immediate reconciliation.
|
|
|
|
## PodSecurity Alignment
|
|
|
|
- **Namespace PodSecurity labels must match container security contexts.** DinD, CSI drivers, and other privileged workloads need `pod-security.kubernetes.io/enforce: privileged` on their namespace. A `baseline` or `restricted` namespace silently blocks privileged pods.
|
|
- **Document privileged namespace requirements.** When a workload needs elevated privileges, document the specific requirement (e.g., "Docker-in-Docker for CI builds") alongside the namespace label.
|
|
|
|
## Miscellaneous
|
|
|
|
- `enableServiceLinks: false` may be needed when K8s-injected service env vars conflict with app config (e.g., Authelia interprets `AUTHELIA_*` service vars as configuration).
|
|
- Proxmox VM names must match K8s node hostnames for cloud controller manager integration.
|
|
- Metrics-server on Talos needs `--kubelet-insecure-tls` (self-signed kubelet certs).
|