distill: 49 best practices from 5 projects (2026-03-27..2026-04-05)

Add 37 new entries and update 7 existing entries across 13 topic files.
Major contributions from agent-runtimes (K8s secrets, CI, Docker gotchas),
cluster-bootstrap (ArgoCD SSA, etcd tuning, DB migrations, Compose networking),
and cluster-apps/octopus-deploy (Helm vs raw manifests, ArgoCD source types).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

validation.md

@@ -63,6 +63,8 @@ Doing all steps at once risks losing access if any step fails. Plan the ordering
 
 Before pushing Dockerfile or service configuration changes, run `docker compose up` locally with a real database and real service images. Unit tests cannot catch deployment-category bugs: Dockerfile CMD syntax, import path errors, env var prefix mismatches, URL encoding issues, factory patterns, startup ordering. A single local `docker compose up` catches these in seconds vs. the 3+ minute CI cycle per fix.
 
+**This applies to ANY iteration on K8s-deployed features, not just initial setup.** Budget 2-3 fixup deploy cycles (~7-10 min each) for any feature first deployed to K8s. Infrastructure gaps between dev and prod always surface issues that unit tests cannot catch: missing COPY directives in Dockerfiles, missing RBAC permissions, wrong file permissions, read-only filesystem constraints. Accepting this cost upfront and smoke-testing locally before each push minimises the number of wasted cycles.
+
 ## Stream Secrets from Source Files, Never from Context
 
 When piping secrets into commands (base64 encoding, kubectl create secret, etc.), always stream from the source file in the same pipeline: `cat /path/to/secret | base64`. Never reconstruct a secret value from conversation context or memory — single-character typos in tokens cause authentication failures that are extremely difficult to diagnose. Save generated secrets to `local_secrets/` immediately upon creation, then reference that file for all subsequent uses.
@@ -78,3 +80,35 @@ When starting a milestone with multiple architectural choices, batch all decisio
 ## Explore the Target Environment Before Planning
 
 For infrastructure-heavy work, research the target environment's actual state before making design decisions. This means checking: what ingress controller is in use, how DNS resolves, what TLS strategy exists, what storage backends are available, what auth middleware is configured. Discovering these facts during planning (not implementation) prevents architectural surprises. In agent-orchestrated workflows, dedicated exploration agents that survey the target environment pay for themselves by eliminating implementation detours.
+
+## Two-Commit Pattern for In-Cluster Database Migrations
+
+When migrating a K8s service to a new database backend: commit 1 = additive (deploy new database alongside existing setup), commit 2 = config switch (point the app at the new database). This avoids fighting GitOps controllers with `selfHeal: true`, which immediately revert manual scale-down operations. The pattern also provides a rollback path — if the config switch fails, revert commit 2.
+
+## Rehearse Database Migrations on a Disposable Instance
+
+Before running any production database migration, rehearse the full path on a disposable instance (Docker container, test namespace). Migration tooling has undocumented quirks: missing commands, flag-dependent output formats, permission side effects. A 5-minute rehearsal catches these, vs. 30+ minutes debugging live.
+
+## Read the App's Entrypoint Script Before Configuring Env Vars
+
+For containerized apps with custom env var conventions, read the entrypoint script once before writing any configuration. Many apps use prefix-based conventions that aren't fully documented. Discovering these through trial and error costs a push-restart-debug cycle per mistake. Five minutes reading the entrypoint saves 30-60 minutes of iterative fixing.
+
+## New Platform Service Deployment Checklist
+
+When adding any new service to a platform/cluster, use a standard checklist: (1) application manifests, (2) secrets management, (3) GitOps application definition, (4) auth/SSO integration, (5) dashboard/UI registration, (6) ingress/routing rules, (7) reverse proxy config, (8) DNS records, (9) deploy automation, (10) full-chain test. A written checklist prevents the "forgot to add the DNS record" class of errors.
+
+## Evaluate Content Home Before Building
+
+Before creating a new system, document type, or knowledge artifact, discuss where it belongs conceptually. Different content types have different lifecycles: accumulated learnings (MEMORY.md) vs authoritative maintained maps (SPEC, CLAUDE.md) vs behavioural contracts (spec files) vs current focus (CONTEXT.md). Picking the wrong home means future maintenance friction.
+
+## Add SSH-Authenticating User as Collaborator When Creating Repos via API
+
+When creating Git repos via API token (which authenticates as one user) but pushing via SSH (which authenticates as a different user based on SSH key config), always add the SSH user as a collaborator with write access before the first push. A 403 on push after a successful API create is the symptom.
+
+## Use kubectl exec to Verify Deployed Container Contents
+
+When confirming whether a fix is deployed, `kubectl exec deploy/<name> -- ls <path>` or `kubectl exec deploy/<name> -- cat <path>` is faster and more reliable than correlating CI build timestamps with commit times or checking registry tags.
+
+## Think Through K8s Constraints Before Coding Docker-First Solutions
+
+Before implementing a feature that works in Docker, enumerate the K8s differences: read-only Secret volumes, root-owned files, no host-path mounts, separate pod filesystem, env var size limits. Design for both backends upfront. Planning for both environments from the start eliminates costly iteration cycles.