distill: best practices from 2026-04-19 cross-project run

Adds 3 new topic files (ai-parallel-agents, api-integration,
python-patterns) and extends 21 existing topic files with new gotchas
and patterns surfaced from memory across tracked projects. Index
updated accordingly.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

ai-parallel-agents.md

@@ -0,0 +1,111 @@
+# AI Parallel Agents
+
+Patterns for orchestrating parallel AI agents inside a Claude Code (or similar) workflow — research fan-out, file-contention avoidance, tool-capability limits, context-budget discipline, and dataset-wide audits via background agents.
+
+Cross-references: [Agent Repos & Container Agent Operations](agent-repos.md) covers the container-agent execution environment (harnesses, agent repos, CP dispatch). This file covers the orchestration patterns a main-thread Claude Code session uses when spawning and coordinating subagents.
+
+---
+
+## 1. Dispatch Parallel Agents for Multi-Facet Research
+
+**Principle:** When researching a topic with multiple independent facets, dispatch parallel research agents rather than running sequential queries from the main thread.
+
+**Why it matters:** Sequential investigation delays cross-facet contradiction discovery to the point where it is expensive to address (often after a plan has been written). Scoping each agent narrowly to one facet — per API, per jurisdiction, per sub-topic, per source-type filter — both parallelises the work and improves signal-to-noise because each agent's context is tightly focused on one domain.
+
+**How to implement:**
+- Enumerate the facets of the research topic before dispatching. For an integration project, facets are usually "per external API". For a compliance question, facets are usually "per jurisdiction". For a market survey, facets are usually "per source category" (vendor docs, academic papers, blog posts).
+- Spawn one agent per facet, each with a narrow prompt: "Research X specifically in the context of Y. Do not cover adjacent topics."
+- For integration work that touches 2+ external APIs, spawn one research agent per API, plus a parallel plan-reviewer agent that checks cross-API compatibility as soon as the per-API agents report back.
+- Merge results in the main thread — deduplicate, flag contradictions, and surface cross-facet constraints.
+
+**Anti-patterns:**
+- Running a single agent with "research topics X, Y, and Z" — context dilution, and any blocker in X delays Y and Z.
+- Sequential WebFetch calls from the main thread when the facets are independent.
+- Dispatching parallel agents with overlapping scope — they return the same information and the dedup cost eats the parallelism savings.
+- Waiting for research agent N to finish before dispatching reviewer agents — run them concurrently when they have no data dependency.
+
+---
+
+## 2. File Contention: Agents Return Text, Main Thread Writes
+
+**Principle:** Never have two parallel agents edit the same file. Have agents RETURN prepared text in their final message and apply edits sequentially from the main thread.
+
+**Why it matters:** Subagent writes to a shared file silently collide — one edit wins, the other is lost, and there is no error or warning. Verified in practice during a 195-claim verification across 16 files: 4 parallel agents, each handling 3-4 files, returning their prepared edits as text, applied by the main thread — zero conflicts, one round.
+
+**How to implement:**
+- Partition the file set so each agent owns a non-overlapping subset. Never assign the same file to two agents.
+- Instruct agents explicitly: "Return your proposed edits as text in your final message. Do not write to disk." Include an example of the expected return format (e.g., file path + old/new block per edit).
+- When overlap is unavoidable (e.g., a cross-cutting change to every file), have agents RETURN their changes; apply edits from the main thread in a deterministic order.
+- For large batches, consider a two-phase pattern: parallel agents produce proposed edits as text; main thread reviews and applies.
+
+**Anti-patterns:**
+- "Each agent edits the files relevant to its findings" — guaranteed to produce silent write collisions when scopes overlap.
+- Letting agents write to a shared directory without partitioning — last write wins, earlier writes vanish.
+- Assuming filesystem-level locking will save you — Claude Code agents do not hold locks across calls.
+- Relying on git to detect the collision — an agent that reads the pre-collision state and writes after the other agent has the correct "new" content, and git sees only the last write.
+
+---
+
+## 3. Subagents Cannot WebFetch — Perform Fetches in Main Thread
+
+**Principle:** Subagents cannot use `WebFetch` or `WebSearch` (permission denied by the harness). Perform web fetches in the main conversation and delegate file processing — reading, verification, extraction, summarisation — to agents.
+
+**Why it matters:** Telling a subagent to "go fetch X from the web" fails silently from the orchestrator's perspective — the subagent returns what it "knows" (training data) rather than what's on the web today. For container agents the restriction is stricter still: no network tools at all, and training-data version claims have been measured at ~30% inaccuracy.
+
+**How to implement:**
+- Use `WebFetch` / `WebSearch` in the main thread to pull live content into files or into the context.
+- Pass the fetched content to subagents as text (in their prompt) or as file paths they can Read.
+- For research agents that need multi-source fetches, fetch everything in the main thread first, then dispatch agents to process/summarise the local content.
+- For container agents, treat all external network access as unavailable — run a separate web-validation pass from the main conversation after the container agent completes.
+
+**Anti-patterns:**
+- "Dispatch a subagent to research X on the web" — subagent has no web tools, will fabricate from training data.
+- Telling container agents to "use web search" — they have none and will not tell you so.
+- Trusting version numbers, release dates, or "actively maintained" claims from a subagent that had no network access — validate separately.
+
+---
+
+## 4. Narrow Read Instructions to Prevent Context Blowup
+
+**Principle:** Give agents narrow read instructions so each agent's context stays small. Name specific sections, grep patterns, or line ranges rather than "read the file."
+
+**Why it matters:** Parallel agents that each read every full file consume tokens without improving output quality. A 16-file, 195-claim verification run blows up if every agent reads every file; it stays cheap if each agent reads only the section relevant to its claim. Context is the scarce resource in multi-agent workflows.
+
+**How to implement:**
+- In the agent prompt, specify the exact section or line range: "Read only the `## Key Data Points` section of each file" or "Read lines 120-180 of spec/ingestion.md."
+- When the agent needs to find the relevant section itself, instruct it to grep first and Read only matching line windows — not the whole file.
+- For large source trees, supply a curated file list; do not let the agent glob an entire repo.
+- Review agent prompts for accidental "read everything" phrasing — "look through the spec" is unbounded, "read Section 3.2 of the spec" is bounded.
+
+**Anti-patterns:**
+- "Read all the files in the spec directory and tell me X" — N agents × M files = huge context burn.
+- Open-ended research prompts with no read boundaries — agents default to reading everything.
+- Long agent prompts that themselves include full file contents when a section would suffice.
+- Failing to cross-reference against an existing index file (CLAUDE.md, MEMORY.md) — forcing every agent to rediscover the repo structure.
+
+---
+
+## 5. Background Agents for Dataset-Wide Audits
+
+**Principle:** When a question requires surveying every record in a large dataset — field gap analysis, distribution across files, unknown-unknowns discovery — dispatch a background (container) agent with a single well-scoped sweep prompt and a research-document output. One complete pass produces better results than piecemeal interactive exploration.
+
+**Why it matters:** Interactive main-thread exploration of large datasets is slow, context-expensive, and prone to anchoring on whatever the operator looked at first. A background agent with a single sweep prompt can process the full dataset in one pass, produce a structured research artifact, and surface patterns the interactive session would miss.
+
+**How to implement:**
+- Frame the audit as a complete question: "For every ticket in the dataset, classify by [dimensions]. Produce a research document at `research/<audit-name>.md` with sections [X, Y, Z] and a summary table."
+- Give the agent the full dataset location (repo path, data export, dataset manifest) and explicit output instructions.
+- Prefer one deeper pass over many shallow passes — the marginal cost of one big research run is usually lower than the cumulative cost of back-and-forth.
+- Store research outputs in a dedicated folder (e.g., `research/`) separate from code and docs. Humans review, distill the useful findings, and archive or discard the raw research.
+- For recurring audits (weekly, per-milestone), template the prompt so it can be re-run with a date/scope parameter.
+
+**Anti-patterns:**
+- Running the sweep interactively in the main thread — slow and context-expensive.
+- Multiple overlapping sweeps with no consolidation — hard to reason about what's covered.
+- No dedicated output folder — research artifacts get lost alongside code.
+- Never reading the research output after dispatch — a "fire and forget" audit with no human review has no value.
+
+---
+
+## Summary
+
+Parallel agent orchestration multiplies throughput when agents have narrowly scoped work, disjoint file sets, and the right tool capabilities. The main thread holds responsibilities subagents cannot do: web fetches, file writes on contested paths, and final integration of returned text. For large datasets, a single background sweep with a research-document output usually beats interactive back-and-forth.