22d49b2c9a
Adds 3 new topic files (ai-parallel-agents, api-integration, python-patterns) and extends 21 existing topic files with new gotchas and patterns surfaced from memory across tracked projects. Index updated accordingly. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
112 lines
9.6 KiB
Markdown
112 lines
9.6 KiB
Markdown
# AI Parallel Agents
|
||
|
||
Patterns for orchestrating parallel AI agents inside a Claude Code (or similar) workflow — research fan-out, file-contention avoidance, tool-capability limits, context-budget discipline, and dataset-wide audits via background agents.
|
||
|
||
Cross-references: [Agent Repos & Container Agent Operations](agent-repos.md) covers the container-agent execution environment (harnesses, agent repos, CP dispatch). This file covers the orchestration patterns a main-thread Claude Code session uses when spawning and coordinating subagents.
|
||
|
||
---
|
||
|
||
## 1. Dispatch Parallel Agents for Multi-Facet Research
|
||
|
||
**Principle:** When researching a topic with multiple independent facets, dispatch parallel research agents rather than running sequential queries from the main thread.
|
||
|
||
**Why it matters:** Sequential investigation delays cross-facet contradiction discovery to the point where it is expensive to address (often after a plan has been written). Scoping each agent narrowly to one facet — per API, per jurisdiction, per sub-topic, per source-type filter — both parallelises the work and improves signal-to-noise because each agent's context is tightly focused on one domain.
|
||
|
||
**How to implement:**
|
||
- Enumerate the facets of the research topic before dispatching. For an integration project, facets are usually "per external API". For a compliance question, facets are usually "per jurisdiction". For a market survey, facets are usually "per source category" (vendor docs, academic papers, blog posts).
|
||
- Spawn one agent per facet, each with a narrow prompt: "Research X specifically in the context of Y. Do not cover adjacent topics."
|
||
- For integration work that touches 2+ external APIs, spawn one research agent per API, plus a parallel plan-reviewer agent that checks cross-API compatibility as soon as the per-API agents report back.
|
||
- Merge results in the main thread — deduplicate, flag contradictions, and surface cross-facet constraints.
|
||
|
||
**Anti-patterns:**
|
||
- Running a single agent with "research topics X, Y, and Z" — context dilution, and any blocker in X delays Y and Z.
|
||
- Sequential WebFetch calls from the main thread when the facets are independent.
|
||
- Dispatching parallel agents with overlapping scope — they return the same information and the dedup cost eats the parallelism savings.
|
||
- Waiting for research agent N to finish before dispatching reviewer agents — run them concurrently when they have no data dependency.
|
||
|
||
---
|
||
|
||
## 2. File Contention: Agents Return Text, Main Thread Writes
|
||
|
||
**Principle:** Never have two parallel agents edit the same file. Have agents RETURN prepared text in their final message and apply edits sequentially from the main thread.
|
||
|
||
**Why it matters:** Subagent writes to a shared file silently collide — one edit wins, the other is lost, and there is no error or warning. Verified in practice during a 195-claim verification across 16 files: 4 parallel agents, each handling 3-4 files, returning their prepared edits as text, applied by the main thread — zero conflicts, one round.
|
||
|
||
**How to implement:**
|
||
- Partition the file set so each agent owns a non-overlapping subset. Never assign the same file to two agents.
|
||
- Instruct agents explicitly: "Return your proposed edits as text in your final message. Do not write to disk." Include an example of the expected return format (e.g., file path + old/new block per edit).
|
||
- When overlap is unavoidable (e.g., a cross-cutting change to every file), have agents RETURN their changes; apply edits from the main thread in a deterministic order.
|
||
- For large batches, consider a two-phase pattern: parallel agents produce proposed edits as text; main thread reviews and applies.
|
||
|
||
**Anti-patterns:**
|
||
- "Each agent edits the files relevant to its findings" — guaranteed to produce silent write collisions when scopes overlap.
|
||
- Letting agents write to a shared directory without partitioning — last write wins, earlier writes vanish.
|
||
- Assuming filesystem-level locking will save you — Claude Code agents do not hold locks across calls.
|
||
- Relying on git to detect the collision — an agent that reads the pre-collision state and writes after the other agent has the correct "new" content, and git sees only the last write.
|
||
|
||
---
|
||
|
||
## 3. Subagents Cannot WebFetch — Perform Fetches in Main Thread
|
||
|
||
**Principle:** Subagents cannot use `WebFetch` or `WebSearch` (permission denied by the harness). Perform web fetches in the main conversation and delegate file processing — reading, verification, extraction, summarisation — to agents.
|
||
|
||
**Why it matters:** Telling a subagent to "go fetch X from the web" fails silently from the orchestrator's perspective — the subagent returns what it "knows" (training data) rather than what's on the web today. For container agents the restriction is stricter still: no network tools at all, and training-data version claims have been measured at ~30% inaccuracy.
|
||
|
||
**How to implement:**
|
||
- Use `WebFetch` / `WebSearch` in the main thread to pull live content into files or into the context.
|
||
- Pass the fetched content to subagents as text (in their prompt) or as file paths they can Read.
|
||
- For research agents that need multi-source fetches, fetch everything in the main thread first, then dispatch agents to process/summarise the local content.
|
||
- For container agents, treat all external network access as unavailable — run a separate web-validation pass from the main conversation after the container agent completes.
|
||
|
||
**Anti-patterns:**
|
||
- "Dispatch a subagent to research X on the web" — subagent has no web tools, will fabricate from training data.
|
||
- Telling container agents to "use web search" — they have none and will not tell you so.
|
||
- Trusting version numbers, release dates, or "actively maintained" claims from a subagent that had no network access — validate separately.
|
||
|
||
---
|
||
|
||
## 4. Narrow Read Instructions to Prevent Context Blowup
|
||
|
||
**Principle:** Give agents narrow read instructions so each agent's context stays small. Name specific sections, grep patterns, or line ranges rather than "read the file."
|
||
|
||
**Why it matters:** Parallel agents that each read every full file consume tokens without improving output quality. A 16-file, 195-claim verification run blows up if every agent reads every file; it stays cheap if each agent reads only the section relevant to its claim. Context is the scarce resource in multi-agent workflows.
|
||
|
||
**How to implement:**
|
||
- In the agent prompt, specify the exact section or line range: "Read only the `## Key Data Points` section of each file" or "Read lines 120-180 of spec/ingestion.md."
|
||
- When the agent needs to find the relevant section itself, instruct it to grep first and Read only matching line windows — not the whole file.
|
||
- For large source trees, supply a curated file list; do not let the agent glob an entire repo.
|
||
- Review agent prompts for accidental "read everything" phrasing — "look through the spec" is unbounded, "read Section 3.2 of the spec" is bounded.
|
||
|
||
**Anti-patterns:**
|
||
- "Read all the files in the spec directory and tell me X" — N agents × M files = huge context burn.
|
||
- Open-ended research prompts with no read boundaries — agents default to reading everything.
|
||
- Long agent prompts that themselves include full file contents when a section would suffice.
|
||
- Failing to cross-reference against an existing index file (CLAUDE.md, MEMORY.md) — forcing every agent to rediscover the repo structure.
|
||
|
||
---
|
||
|
||
## 5. Background Agents for Dataset-Wide Audits
|
||
|
||
**Principle:** When a question requires surveying every record in a large dataset — field gap analysis, distribution across files, unknown-unknowns discovery — dispatch a background (container) agent with a single well-scoped sweep prompt and a research-document output. One complete pass produces better results than piecemeal interactive exploration.
|
||
|
||
**Why it matters:** Interactive main-thread exploration of large datasets is slow, context-expensive, and prone to anchoring on whatever the operator looked at first. A background agent with a single sweep prompt can process the full dataset in one pass, produce a structured research artifact, and surface patterns the interactive session would miss.
|
||
|
||
**How to implement:**
|
||
- Frame the audit as a complete question: "For every ticket in the dataset, classify by [dimensions]. Produce a research document at `research/<audit-name>.md` with sections [X, Y, Z] and a summary table."
|
||
- Give the agent the full dataset location (repo path, data export, dataset manifest) and explicit output instructions.
|
||
- Prefer one deeper pass over many shallow passes — the marginal cost of one big research run is usually lower than the cumulative cost of back-and-forth.
|
||
- Store research outputs in a dedicated folder (e.g., `research/`) separate from code and docs. Humans review, distill the useful findings, and archive or discard the raw research.
|
||
- For recurring audits (weekly, per-milestone), template the prompt so it can be re-run with a date/scope parameter.
|
||
|
||
**Anti-patterns:**
|
||
- Running the sweep interactively in the main thread — slow and context-expensive.
|
||
- Multiple overlapping sweeps with no consolidation — hard to reason about what's covered.
|
||
- No dedicated output folder — research artifacts get lost alongside code.
|
||
- Never reading the research output after dispatch — a "fire and forget" audit with no human review has no value.
|
||
|
||
---
|
||
|
||
## Summary
|
||
|
||
Parallel agent orchestration multiplies throughput when agents have narrowly scoped work, disjoint file sets, and the right tool capabilities. The main thread holds responsibilities subagents cannot do: web fetches, file writes on contested paths, and final integration of returned text. For large datasets, a single background sweep with a research-document output usually beats interactive back-and-forth.
|