distill: 49 best practices from 5 projects (2026-03-27..2026-04-05)

Add 37 new entries and update 7 existing entries across 13 topic files.
Major contributions from agent-runtimes (K8s secrets, CI, Docker gotchas),
cluster-bootstrap (ArgoCD SSA, etcd tuning, DB migrations, Compose networking),
and cluster-apps/octopus-deploy (Helm vs raw manifests, ArgoCD source types).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

kubernetes.md

@@ -67,3 +67,39 @@ Manual bootstrap secrets (encryption keys, OIDC client secrets) must be document
 - `enableServiceLinks: false` may be needed when K8s-injected service env vars conflict with app config (e.g., Authelia interprets `AUTHELIA_*` service vars as configuration).
 - Proxmox VM names must match K8s node hostnames for cloud controller manager integration.
 - Metrics-server on Talos needs `--kubelet-insecure-tls` (self-signed kubelet certs).
+
+## ArgoCD SSA + StatefulSet volumeClaimTemplates = Perpetual OutOfSync
+
+Kubernetes injects `apiVersion` and `kind` fields into StatefulSet `volumeClaimTemplates` on apply. These don't exist in source manifests, causing ArgoCD with ServerSideApply to report perpetual OutOfSync. Fix: add `ignoreDifferences` on the ArgoCD Application targeting `.spec.volumeClaimTemplates[]?.apiVersion` and `.spec.volumeClaimTemplates[]?.kind` (using `jqPathExpressions`), plus `RespectIgnoreDifferences=true` in syncOptions.
+
+## ArgoCD SSA May Not Detect ConfigMap Data Changes
+
+ArgoCD with ServerSideApply sometimes fails to detect changes to ConfigMap `data` values, reporting "Synced + Healthy" while the live ConfigMap has stale content. Root cause: SSA field ownership conflicts between Helm's managed fields and a prior `kubectl apply` annotation. After syncing ConfigMaps managed by Helm+SSA, verify content with `kubectl get cm <name> -o jsonpath='{.data.<key>}'`.
+
+## CSI VolumeAttachment Stuck After Hot-Plug Failure
+
+CSI hot-plug of storage devices can fail silently — the VolumeAttachment object says `attached: true` but the device never appeared on the node. Pods get stuck in `ContainerCreating` with "device not found." Fix: delete the stale VolumeAttachment (`kubectl delete volumeattachment <name>`). The CSI driver recreates it and retries the attach.
+
+## Delete and Recreate ArgoCD Apps on Source Type Changes
+
+When changing an ArgoCD Application's source type (e.g., multi-source Helm to single-source Kustomize), the repo-server may serve cached manifests from the old configuration, and old Helm hook resources become ghost entries that block deletion via finalizers. Delete the Application entirely and let the root app recreate it rather than patching source types in-place.
+
+## etcd on Slow Storage Requires Timeout Tuning
+
+etcd requires sub-10ms fsync for stable operation. On slow storage (HDD, network-attached, overloaded SSD), default timeouts (heartbeat 100ms, election 1000ms) cause leader election flapping, cascading scheduler/controller-manager restarts, and widespread probe failures. Symptoms: "leader failed to send out heartbeat on time", "apply request took too long." Mitigation: increase heartbeat-interval (e.g., 500ms) and election-timeout (e.g., 5000ms). Fix: move etcd to SSD storage. Periodic defrag also helps.
+
+## CrashLoopBackOff Delays New Image Pickup
+
+After CI builds a fix for a crashing pod, the CrashLoopBackOff exponential backoff (up to 5 minutes) means the kubelet won't pull the new image until the next retry window. Run `kubectl rollout restart deployment/<name>` immediately after CI completes to create a fresh pod instead of waiting.
+
+## K8s Secret Volumes Are Read-Only with Root Ownership
+
+K8s Secret volume mounts are read-only — you cannot write or modify files in them. Files are owned by root regardless of fsGroup settings. Non-root processes need `defaultMode: 0444` (world-readable) to access the files. Additionally, mounting a Secret at a parent path shadows any other Secret mounts at child paths — mount each Secret at its own non-overlapping path.
+
+## K8s Env Var Size Limits for Payloads
+
+Kubernetes has a hard limit on environment variable sizes (~228KB base64). Large payloads embedded as env vars cause containers to crash with exit 255 and zero logs. For inter-task artifact passing, use git branches or mounted volumes instead of env var payloads.
+
+## Non-Blocking Registration in FastAPI Lifespan Handlers
+
+Blocking operations (external API calls, service registration) in application lifespan handlers prevent the HTTP server from starting. K8s liveness probes fail and the pod enters CrashLoopBackOff before the operation completes. Use background tasks (e.g., `asyncio.create_task`) for registration so health endpoints respond immediately while registration happens asynchronously. This applies to any K8s-deployed app framework with startup hooks (FastAPI, Flask, etc.).