distill: 49 best practices from 5 projects (2026-03-27..2026-04-05)

Add 37 new entries and update 7 existing entries across 13 topic files.
Major contributions from agent-runtimes (K8s secrets, CI, Docker gotchas),
cluster-bootstrap (ArgoCD SSA, etcd tuning, DB migrations, Compose networking),
and cluster-apps/octopus-deploy (Helm vs raw manifests, ArgoCD source types).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

spec-driven-development.md

@@ -219,6 +219,12 @@ This works significantly better than defining all tasks upfront because spec age
 
 Agent orchestration leaves integration gaps at component boundaries. Each agent completes its assigned scope correctly, but nobody owns the integration points between them (e.g., stub comments, ORM mapping methods not updated for new fields). After every orchestration run, include an explicit integration verification step that checks cross-component contracts — call sites, shared data models, and handoff points.
 
+Integration failures fall into a taxonomy of root causes. Categorize failures before fixing and address them category-by-category:
+1. **New required fields on shared dataclasses without defaults** — An agent adds a field to a shared data model without a default value, breaking every other agent's code that constructs that model.
+2. **Mock targets that don't match actual code structure** — Agents patch `"module.ClassName.method"` but the actual code uses a different import path or method name, so tests pass against mocks but fail against real code.
+3. **Tests written before implementation is finalized** — Tests assume behaviour that changed during implementation. The spec said one thing, the implementation diverged, and the test was never updated.
+4. **Tests with `clear=True` on `os.environ` missing required env vars** — Tests that clear the environment forget to set variables the code requires at import time or during setup, causing failures unrelated to the tested behaviour.
+
 ### Decompose Along File Boundaries
 
 When splitting work into parallel agent tasks, ensure each task writes to distinct files. When two agents must modify the same file, make the shared changes small and predictable — identify the conflict point upfront so the merge is trivial. File-boundary decomposition produces zero-conflict assemblies.
@@ -248,3 +254,27 @@ When changes are small per file (5-15 lines) but tightly coupled across many fil
 ### Untested requirements
 **Symptom:** Requirements exist in the spec but no test references them. They drift without anyone noticing.
 **Fix:** Spec coverage check in CI. Every requirement ID must appear in at least one test function name.
+
+## Multi-Model Review for Security-Critical Specs
+
+Running the same security review with two different LLM models and comparing outputs catches significantly more issues than either alone. In measured experiments, only 62% of findings overlapped — the union covered 38% more issues. For security-critical specs, the cost of a second model review is justified by the coverage improvement. When models agree, confidence is high; when they disagree, escalate to human review.
+
+## Security Review Before Agent Implementation
+
+Run a security review of the plan before decomposing into implementation tasks. A 15-minute review before coding catches real vulnerabilities that would otherwise ship to production. Example: a `source` injection vulnerability in a wrapper script was caught during plan review that would have been a production security hole if caught only after implementation.
+
+## Forward-Looking Annotations Must Be Labelled
+
+RBAC annotations and other forward-looking requirements in specs (e.g., "requires admin role" when RBAC isn't implemented yet) must be prefixed with "(Future MN)" to indicate they describe future enforcement, not current behaviour. Without the label, agents may implement access checks prematurely or build infrastructure not needed for the current milestone.
+
+## New Fields on Shared Data Models Must Have Defaults
+
+When adding fields to shared data models (dataclasses, Pydantic models, Protobuf messages), new fields must always have default values. Code in other branches, agents, or callers constructs instances without the new field. A required field without a default breaks every existing caller. Use `field(default_factory=list)` for collections and `None` or sentinel values for optionals.
+
+## Agent Prompts Must Include Mock Targets and Import Conventions
+
+Agents working in isolated worktrees or containers cannot discover mock targets or import conventions from sibling test files. Every implementation prompt must explicitly state: the exact function paths to mock (e.g., `patch.object(instance, "_method_name")` not `patch("module.function")`), the project's import convention, and test fixture patterns.
+
+## Plan-First Approach Eliminates Fix Cycles for Cross-Cutting Changes
+
+For changes touching 5+ files across multiple subsystems, invest 30-45 minutes in exploration and planning before writing code. Measured results: sessions with plan-first had 0 fix commits; sessions with code-first had 7:1 fix:forward ratios. Use parallel exploration agents to cover different dimensions of the problem space.