distill: 48 cross-project best-practices from 2026-07 reflection sweep

Promotions from reflecting 21 projects' session logs (incl. agent-runtimes
122-log drain). Adds coverage across networking (eBPF VIP/VPN SNAT/VLAN
bridge/forward-auth preflight/ingress TLS), kubernetes (CSI hotplug/PodSecurity
debug/self-managed GitOps/runtime annotations), CI (dispatch tokens/runner
death/base image), git (CI-rebase/shallow reset/PR governance), python (async
session pool/httpx redirects/logging), TDD (AsyncMock/xfail lifecycle),
api-integration (SDK parse/token-scope 404/schema probing), plus docker,
scripting, debugging, security-architecture, secrets, react, octopus.

State: .distill-state.json refreshed with current HEADs + 5 newly-tracked projects.

debugging.md

@@ -31,6 +31,10 @@ When `/etc/hosts` or internal DNS points a public hostname at an internal IP, lo
 
 Applies to reverse-proxy routing bugs, HTTP/2 SAN mismatches, and TLS configuration that differs between internal and external ingress.
 
+**Trace multi-hop DNS chains end-to-end, not just the endpoints.** For split-horizon / VPN DNS that flows through several resolvers (e.g. VPN MagicDNS → local forwarder → authoritative server), a record existing in the authoritative server does NOT mean a client resolves it. Each hop can drop the query: a missing conditional-forward rule, a resolver pushed to clients that doesn't know a downstream zone, or a forwarder pointed at a target with no authoritative zone. Walk the full resolution path hop-by-hop (client → each forwarder → authoritative), querying each resolver directly (`dig @<resolver> <name>`), rather than only confirming the record exists at the source. Every individual server can look healthy while the chain is broken at one forwarding link.
+
+**Corollary — conditional forwarding needs an authoritative target.** `server=/zone/<ip>` (dnsmasq) or any conditional-forward rule returns empty answers if the forward target has no real zone for that domain — e.g. a DHCP-only resolver that answers bare hostnames but has no SOA. The forward is syntactically valid but there is nothing authoritative to forward to; work around it by ingesting the records another way (poll the source API, write a hosts file).
+
 ## When Something Doesn't Sync/Apply
 
 - Check resource exclusions in the GitOps controller immediately
@@ -50,6 +54,7 @@ Before starting any OIDC integration, research:
 - **CrashLoopBackOff: check logs first.** Error messages in pod logs usually point directly to the fix. Don't tweak configuration or security contexts blindly — `kubectl logs <pod>` first.
 - **Discriminate transient from persistent errors.** CSI lock contention, etcd timeouts during first install, and brief connectivity blips are self-healing. Don't spend time debugging errors that resolve on retry. If you see retry/backoff patterns in logs, wait before intervening.
 - **Trust controller retry logic.** CSI controllers, operators, and reconciliation loops have built-in retry. Transient failures during rapid provisioning are expected, not bugs.
+- **Framework-sanitised error bodies are unreliable to assert on.** Security-conscious frameworks strip informative detail from response bodies (FastAPI's `RequestValidationError` handler returns `{"detail": "Request validation failed"}` regardless of the actual cause; GraphQL `formatError` can sanitise messages). Don't write tests or downstream parsers that depend on the stripped body — only status code is reliable. The informative version is usually written to server-side logs; check there, not the wire response.
 
 ## Reproduce Before Fixing
 
@@ -128,3 +133,92 @@ After deploying a new version of an application that uses an ORM or schema migra
 **Quick check:** run the application's schema validation command, or compare `alembic current` vs `alembic head`, or run `SELECT column_name FROM information_schema.columns WHERE table_name='<table>'` and diff against the model definition.
 
 **When to check:** after every deployment that touches models or migrations — not just on explicit migration commits. An ORM auto-create (e.g., SQLAlchemy `create_all`) can silently succeed while leaving optional columns missing, causing subtle bugs rather than hard crashes.
+
+## Lost-Webhook Zombie Pattern
+
+Distributed task/job systems that depend on a webhook or callback to advance state can leave records stuck in `running` indefinitely when the callback is lost. The state record looks active; nothing is actually happening.
+
+**Pattern signature:**
+- `state=running` for >20 min with no progress
+- Claim/lease set, but `current_load=0` on the worker
+- Zero log progress since the claim event
+- The work artifact (output branch, file, queue entry) exists or is partially populated
+
+**Diagnosis rules:**
+- Don't trust the orchestrator's state record — verify side effects directly. Check whether the work-artifact branch was pushed, the output file written, the queue entry produced.
+- Don't expect the system's DELETE endpoint to recover — most refuse to terminate non-terminal records. You will need to manually transition the state or wait for a timeout that may never come.
+- Cross-reference dispatcher/worker logs (if still alive) for the claim event and any subsequent webhook attempt. A missing "callback succeeded" log entry confirms the lost-webhook hypothesis.
+
+**Prevention:** make the orchestrator poll for terminal artifacts in addition to listening for webhooks. The webhook is an optimisation; the artifact poll is the source of truth. Generalises to CI pipelines, async job queues, agent runtime systems — anything where worker completion depends on an out-of-band callback.
+
+## Detection Before Auto-Remediation
+
+For any recurring failure mode (storage hotplug failures, network policy drops, stale leases, queue zombies, rate-limit hits), ship a detection signal — Prometheus alert, log-pattern check, scheduled audit — BEFORE building any auto-remediation.
+
+**Rationale:** auto-remediation has its own failure modes (drain timeouts, PodDisruptionBudget conflicts, recursive failures, race conditions with the underlying bug). Shipping it without observability hides those failures; shipping observability first lets you measure how often the bug occurs and how often a manual fix succeeds before you trust automation with the same fix.
+
+**Order of operations:**
+1. **Detect** — alert with sufficient context to recover manually (resource name, pod, host, last-known state)
+2. **Document the manual fix** — capture the working recovery sequence as a runbook or script the alert links to
+3. **Build automation behind a feature flag** — auto-remediation in shadow mode (log what it would do; don't act)
+4. **Compare shadow decisions against operator actions** — if they agree at high rate, flip the flag
+
+Detection alone turns multi-hour incidents into ~15-minute ones at near-zero risk. Automation built without the prior alert layer is unauditable.
+
+## Distinguish Mitigations from Cures in Writing
+
+When a recurring bug is reduced but not eliminated by a fix, explicitly label the fix as a "mitigation" in the gotcha file and track the recurrence vector. The temptation to mark "fixed" leads to surprise when the bug returns and to wasted effort re-debugging from scratch.
+
+**Conventions:**
+- Use the phrase "mitigations, NOT a full fix" in the gotcha file's title or first paragraph when the underlying root cause persists.
+- Link to the long-term replacement plan (e.g., a FUTURE.md item or upstream issue).
+- Record the **recurrence vector**: under what conditions does the mitigated bug come back? "Recurs when load > X", "recurs on host rebuild", "recurs after Y days".
+- When the bug recurs, append the new incident to the same gotcha entry — don't open a new one. The history of recurrence is the evidence that the fix is a mitigation, not a cure.
+
+Generalises across all projects that track incidents in `memory/gotchas-*.md`. Prevents future sessions (and future Claude instances) from misreading a mitigation as a cure.
+
+## Hypervisor- or Platform-Level Diagnostics Before Application-Level Blame
+
+When a symptom appears at the application layer (pod stuck `ContainerCreating`, device missing, mount failing, port unreachable) but the platform underneath has its own lifecycle model, check the platform's task/event API first.
+
+**Concrete examples:**
+- Pod stuck attaching a volume → `qm pending <vmid>` and `pvesh get /nodes/<host>/tasks --vmid <id>` reveal Proxmox hotplug failures invisible from `qm config`
+- AWS PV not attaching → EC2 attachment-state events reveal failures invisible from `kubectl`
+- Systemd unit failed → `journalctl -u` reveals failures invisible from `ps` or service-level health checks
+- VM unresponsive → hypervisor console output / serial-line buffer reveals kernel panics invisible from inside the guest
+
+**Rule:** five seconds on the platform API beats fifteen minutes debugging the wrong layer. Build a habit of "check one layer down" before forming a hypothesis at the application layer. Applies to any layered stack: K8s-on-hypervisor, container-on-host, application-on-systemd, agent-on-orchestrator.
+
+## Test From Outside the Broken Thing
+
+When something is unreachable, test from a known-good external vantage point — different host, cellular network, cloud VM, `curl --resolve` with the public IP — before assuming the local machine is at fault.
+
+**Symptoms this catches:**
+- "No route to host" from your laptop while remote users can reach the service fine (your VPN dropped)
+- "DNS gives public IPs but I expected internal" (your `/etc/hosts` or split-horizon DNS is bypassed)
+- "Latency spike" that's actually a single ISP-side route flap (test from a different ISP)
+
+**The inverse trap:** when split-horizon DNS or local `/etc/hosts` overrides bypass the production path, your local "it works" tells you nothing about what real users see. Always verify production behaviour through the actual public path (`curl --resolve domain:443:<public-ip> https://domain/...`, or test from a phone on cellular / a cloud VM in a different region).
+
+Cross-cutting diagnostic discipline applicable to any distributed system, web-facing service, or VPN/proxy stack.
+
+## Verify User-Reported Identifiers Before Recovery
+
+When a user reports a problem by name ("the X pod is stuck", "the Y volume is broken", "the Z VM won't start"), confirm the actual identifiers with `kubectl get`, `qm config`, `docker inspect`, `pvesh ls`, or the equivalent on whichever platform owns the resource — BEFORE running any recovery command.
+
+**Why:** humans under pressure confuse names (VM IDs, node hostnames, PVC names, container names). A 30-second verification prevents executing the wrong recovery on the wrong resource. Recovery commands are often destructive (`qm reset`, `kubectl delete pod --force`, volume detach); running one on a healthy resource turns a small incident into a bigger one.
+
+**Verification pattern:**
+1. Restate the user's claim: "You said pod `foo` is stuck on node `bar`."
+2. Verify each identifier exists and is in the claimed state: `kubectl get pod foo -o wide` (does it exist? is it on `bar`? is it actually stuck?)
+3. Confirm with the user before destructive action: "Pod `foo` on `bar` is in `ContainerCreating`. Running `kubectl delete pod foo --force --grace-period=0`. OK?"
+
+General operational-safety rule, applicable in any high-pressure recovery situation.
+
+## Discover API Endpoints via Swagger/OpenAPI Before Re-Reading Prose Docs
+
+When an API call returns 404 or "endpoint not found", fetch the actual paths from the live spec (`/swagger.v1.json`, `/openapi.json`, `/api-docs`) with `curl + jq` before consulting prose documentation. The spec is generated from the running code; the prose drifts. Five seconds with curl beats fifteen minutes of doc spelunking. See [API Integration](api-integration.md) section 6 for the full pattern.
+
+## Grep Minified Third-Party Source to Learn Its Runtime Protocol
+
+When you must understand how a minified third-party library behaves at runtime (popup/auth detection, `postMessage` protocol, event names, expected `window.name`), grep the minified bundle for concrete string/pattern signatures (`grep -oP 'window\.opener[^;]*'`, event-name literals, magic constants) instead of guessing or reading upstream docs that may not match the shipped build. Minified code still contains the literal strings and property accesses that reveal the protocol reliably — enough to integrate against it without modifying its source.