distill: best practices from 2026-04-19 cross-project run
Adds 3 new topic files (ai-parallel-agents, api-integration, python-patterns) and extends 21 existing topic files with new gotchas and patterns surfaced from memory across tracked projects. Index updated accordingly. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
@@ -413,3 +413,63 @@ curl -s -X DELETE http://localhost:8100/tasks/{id}
|
||||
## Artifact Passing via Git Branches Instead of Env Vars
|
||||
|
||||
For multi-stage workflows where downstream tasks need upstream outputs, push artifacts to branches in an agent repo rather than embedding in prompts or env vars. This avoids K8s env var size limits (~228KB), survives pod restarts, provides a git audit trail, and scales to any artifact size. The downstream task clones the branch as a reference directory.
|
||||
|
||||
## Protect Test Files from Agent Modification via Root-Owned Read-Only Clone
|
||||
|
||||
When agents run tests, they may "fix" failing tests by weakening assertions rather than fixing the underlying code. Prevent this by cloning the test suite into `/workspace/reference/tests/` as a root-owned directory (the agent gets a permission error if it tries to write). The agent's working directory gets a symlink or copy of the tests at startup, but the authoritative copy is immutable. This is the same pattern as `/workspace/reference/main/` for source code — root ownership makes modification a hard error, not a policy.
|
||||
|
||||
## Infrastructure Failures Dominate Agent Failure Modes
|
||||
|
||||
In measured agent runs, the majority of task failures are infrastructure failures, not agent reasoning failures: network timeouts, SSH key not loaded, missing package in the base image, environment variable not propagated. Before debugging agent behaviour, check whether the failure is environmental — a task that consistently fails at "git clone" is an infrastructure problem, not an agent problem.
|
||||
|
||||
**Concrete ratio:** in one measured 12-task batch, 6/12 tasks failed and all 6 were infrastructure-class (wrong harness, missing payload fields, stale images) — zero model failures. Task templates that validate payload structure and harness compatibility before dispatch eliminate the entire dominant failure mode.
|
||||
|
||||
**Pre-dispatch validation checklist:**
|
||||
- SSH key reachable from the agent harness (test clone before dispatching)
|
||||
- Required env vars present (model API keys, registry credentials)
|
||||
- Harness image has all required tools (`uv`, `ruff`, `pytest`, etc.)
|
||||
- Target repo and branch exist
|
||||
- Network egress allows required domains
|
||||
- **Task templates** — long-term remediation. Validate payload structure and harness compatibility at template-render time, not via ad-hoc per-dispatch checks.
|
||||
|
||||
Invest in pre-dispatch validation scripts that catch the top-N infrastructure failures before the first agent container starts.
|
||||
|
||||
## Cross-Model Reviews Catch ~38% More Issues Than a Single Model
|
||||
|
||||
Running the same security or spec review with two different models (e.g., Opus + MiniMax) and comparing outputs catches ~38% more issues than either alone — in measured reviews, only 62% of findings overlap. Models converge on obvious issues but diverge on edge cases and design concerns. Worth the extra cost for security-critical specs and architecture reviews; overkill for routine code review.
|
||||
|
||||
**Pattern:** dispatch parallel review tasks to different models with identical prompts, union the findings, deduplicate against a shared issue key (file + line + category). Present the merged list to the human reviewer along with per-model attribution so reviewers can see where models agreed vs. diverged.
|
||||
|
||||
## Agent Worktree Branches Contain Files, Not Commits — Copy, Don't Merge
|
||||
|
||||
**Symptom:** Orchestrator merges an agent's branch and sees "Already up to date" because the agent wrote files to its worktree but never ran `git add` / `git commit`. Downstream tasks that depend on the upstream artifact then fail or silently use stale data.
|
||||
|
||||
**Fix:** Orchestration must explicitly copy files from dependency worktrees (driven by a `writes` field in the task manifest) into the consuming worktree. Git merge is insufficient when agent output is untracked.
|
||||
|
||||
**Alternative:** Require agents to commit before exit (finalize phase auto-commits everything under `/workspace/working/`), which unlocks git-branch artifact passing. The finalize-phase auto-commit described above is the canonical implementation — enforce it for any agent whose output other tasks depend on.
|
||||
|
||||
## Include Exact Dataclass/Context Schemas in Template-Writing Agent Prompts
|
||||
|
||||
**Symptom:** Agents writing templates invent their own mock context objects (e.g., dict-style `model["fields"]`) while real code provides a different shape (e.g., dataclass `model.fields`). Templates render against the mocks but produce attribute errors against real objects during integration.
|
||||
|
||||
**Fix:** Always include exact dataclass/type definitions of the render context in the agent prompt. During review, compare each agent's mock objects against the real normalized types before merging. Treat "wrote its own mock shape" as a review-blocking issue — the mock shape is a contract the agent must follow, not invent.
|
||||
|
||||
## Never Assume Web Search in Container Agents; Validate Version-Specific Claims Separately
|
||||
|
||||
**Symptom:** Agents in Docker/K8s containers have no `WebSearch` or `WebFetch` capability even with `--dangerously-skip-permissions`. Version numbers, release dates, and "actively maintained" claims come from training data and are often wrong (~30% inaccuracy on package versions in measured runs).
|
||||
|
||||
**Fix:**
|
||||
- Never tell a container agent to "use web search" — it has none and will silently fabricate from training data.
|
||||
- Run a separate web-validation pass (Opus or similar with network access) after research agents complete.
|
||||
- Treat all version claims as hypotheses until validated.
|
||||
- Budget web validation as a distinct pipeline stage, not an afterthought.
|
||||
|
||||
## Cost-Effective Models Need Explicit Scope Boundaries
|
||||
|
||||
Smaller/cheaper models (e.g., MiniMax, Haiku) need tighter scope constraints than capable flagship models. Without explicit boundaries, they drift into scope creep, run the full test suite when asked to write tests, or attempt broad refactors. For cost-effective model tasks:
|
||||
- **No full test suite runs** — specify which test file or test ID to run
|
||||
- **Concrete patterns, not open-ended** — "Write a test matching `test_cp_*.py` naming" not "Write tests for the control plane"
|
||||
- **Longer timeouts** — cheaper models are often slower per token; set `runtime.timeout` to 2-3× what flagship models need
|
||||
- **Explicit output location** — "Write to `results/output.md`" not "Write your findings"
|
||||
|
||||
Treat scope boundaries as a harness concern, not an agent concern — encode them in the prompt template or harness context, not in ad-hoc task prompts.
|
||||
|
||||
Reference in New Issue
Block a user