distill: flush prior-session best-practice additions
Additive/refining entries left uncommitted in the working tree from an earlier distill session (found during the 2026-07 sweep commit): agent-repos 5-step plan pattern; ai-parallel-agents cheap-model scope; spec-driven spec-inversion and caller/callee cross-reference rules.
This commit is contained in:
@@ -436,10 +436,24 @@ Invest in pre-dispatch validation scripts that catch the top-N infrastructure fa
|
|||||||
|
|
||||||
## Cross-Model Reviews Catch ~38% More Issues Than a Single Model
|
## Cross-Model Reviews Catch ~38% More Issues Than a Single Model
|
||||||
|
|
||||||
Running the same security or spec review with two different models (e.g., Opus + MiniMax) and comparing outputs catches ~38% more issues than either alone — in measured reviews, only 62% of findings overlap. Models converge on obvious issues but diverge on edge cases and design concerns. Worth the extra cost for security-critical specs and architecture reviews; overkill for routine code review.
|
Running the same security or spec review with two different models (e.g., Opus + MiniMax) and comparing outputs catches ~38% more issues than either alone — in measured reviews, only 62% of findings overlap. Reconfirmed at ~40% additional findings across multiple milestones in a second independent project — the effect is consistent, not noise. Models converge on obvious issues but diverge on edge cases and design concerns. Worth the extra cost for security-critical specs and architecture reviews; overkill for routine code review.
|
||||||
|
|
||||||
**Pattern:** dispatch parallel review tasks to different models with identical prompts, union the findings, deduplicate against a shared issue key (file + line + category). Present the merged list to the human reviewer along with per-model attribution so reviewers can see where models agreed vs. diverged.
|
**Pattern:** dispatch parallel review tasks to different models with identical prompts, union the findings, deduplicate against a shared issue key (file + line + category). Present the merged list to the human reviewer along with per-model attribution so reviewers can see where models agreed vs. diverged.
|
||||||
|
|
||||||
|
**Run in parallel, not sequentially.** Sequential review wastes wall time AND biases the second reviewer if the first's output lands in the shared context. Concurrent dispatch with identical prompts is the correct shape; merge findings after both complete.
|
||||||
|
|
||||||
|
## Plan → Review → Decisions → V2 → Consistency-Check (5-Step Pattern)
|
||||||
|
|
||||||
|
For non-trivial plans (multi-subsystem features, security-critical changes, cross-team boundaries), use a named 5-step workflow rather than ad-hoc iteration:
|
||||||
|
|
||||||
|
1. **Plan** — produce the initial PLAN.md from the architecture conversation.
|
||||||
|
2. **Review** — dispatch cross-model reviews in parallel (Opus + a different family). Both reviewers receive the same plan and the same review prompt.
|
||||||
|
3. **Decisions** — write a DECISIONS.md capturing the outcome of each review finding: accepted, rejected (with rationale), or deferred. The decisions file is a permanent record of why the plan looks the way it does.
|
||||||
|
4. **V2** — produce PLAN-v2.md applying all accepted decisions. Keep V1 alongside V2; don't overwrite.
|
||||||
|
5. **Consistency-check** — automated grep/script pass to confirm: revoked phrases don't reappear, paired schema shapes match across docs, all referenced requirement IDs resolve. Skipping this lets spec inversions ship — the revoked text remains alongside the new policy and agents follow the wrong one half the time.
|
||||||
|
|
||||||
|
Phase 5 is the one most teams skip and the one that prevents the largest class of multi-day fix-up cycles. Treat it as a mandatory CI step on any PR that touches a plan or spec.
|
||||||
|
|
||||||
## Agent Worktree Branches Contain Files, Not Commits — Copy, Don't Merge
|
## Agent Worktree Branches Contain Files, Not Commits — Copy, Don't Merge
|
||||||
|
|
||||||
**Symptom:** Orchestrator merges an agent's branch and sees "Already up to date" because the agent wrote files to its worktree but never ran `git add` / `git commit`. Downstream tasks that depend on the upstream artifact then fail or silently use stale data.
|
**Symptom:** Orchestrator merges an agent's branch and sees "Already up to date" because the agent wrote files to its worktree but never ran `git add` / `git commit`. Downstream tasks that depend on the upstream artifact then fail or silently use stale data.
|
||||||
|
|||||||
@@ -104,8 +104,52 @@ Cross-references: [Agent Repos & Container Agent Operations](agent-repos.md) cov
|
|||||||
- No dedicated output folder — research artifacts get lost alongside code.
|
- No dedicated output folder — research artifacts get lost alongside code.
|
||||||
- Never reading the research output after dispatch — a "fire and forget" audit with no human review has no value.
|
- Never reading the research output after dispatch — a "fire and forget" audit with no human review has no value.
|
||||||
|
|
||||||
|
### Cheap-Model Sub-Sessions: When to Use, When to Skip
|
||||||
|
|
||||||
|
Tools that fork a cheap-model session for grunt work (`/ask-minimax`, similar MiniMax-, Qwen-, or Haiku-backed delegators) shine on a narrow band of tasks. Reach for them when **the file payload dwarfs the answer payload**.
|
||||||
|
|
||||||
|
**Use for:**
|
||||||
|
- Summarising large files (logs, dumps, generated reports >500 lines)
|
||||||
|
- Extracting specific facts from many files or long reference docs
|
||||||
|
- Generating big files (migrations, fixtures, large docs) whose content does not need to flow back
|
||||||
|
- Format conversion of large files (CSV↔JSON, XML↔YAML)
|
||||||
|
- Bulk find-and-extract where only the matches matter
|
||||||
|
|
||||||
|
**Skip for:**
|
||||||
|
- Iterative design or debugging — the parent session needs the content in context
|
||||||
|
- Small files (<500 lines) — overhead exceeds savings
|
||||||
|
- Tasks where you will immediately re-read the result to act on it
|
||||||
|
- Anything requiring tools the cheap model lacks (web fetch, MCP, browser, agent dispatch)
|
||||||
|
- Architecture or quality-sensitive output — cheap models are for grunt work, not nuanced reasoning
|
||||||
|
- **Per-test verification loops** — the LLM round-trip cost dwarfs the test's own cost; the cheap model also stalls silently on tasks requiring sustained state across iterations. Replace with an AST script.
|
||||||
|
|
||||||
|
**Delegate by reference, not content:** pass file paths, not file contents. If you read the input files yourself before invoking, you have already paid the token cost the skill exists to avoid.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 6. Cost-Effective Models: Single-File, Single-Rule Task Scope
|
||||||
|
|
||||||
|
**Principle:** Cheap/small models (MiniMax, Qwen3.x, Haiku via airouter) reliably produce output when scoped to ONE file and ONE rule per task. Multi-file or multi-rule tasks silently produce no output even when the agent's reasoning logs look correct.
|
||||||
|
|
||||||
|
**Why it matters:** Cost-effective models trade context-management ability for token cost. With multi-file scope, the model loses track of which file it has already edited and produces empty output or commits without the expected files. The failure is silent — exit code 0, reasoning logs look fine, no output file. Verified in a 9-dispatch batch: single-file scope succeeded 8/8, multi-file scope failed 2/2 with no output.
|
||||||
|
|
||||||
|
**How to implement:**
|
||||||
|
- **One file per task** — split multi-file changes into separate dispatches. Each task creates or edits exactly one file.
|
||||||
|
- **Reference the template, don't inline it** — give the agent the path to an existing similar file to read, rather than including all structure in the prompt.
|
||||||
|
- **State the file path explicitly** — "Create `<exact/path/to/file>`" at the start of the prompt, not implied by context.
|
||||||
|
- **Verify output before treating as done** — check the agent-repo branch (or output directory) for the expected file immediately after task success. Don't trust the exit code alone.
|
||||||
|
|
||||||
|
**Sequential chain for unavoidable multi-file work:** dispatch task-1 to produce file-A, wait for success, then dispatch task-2 to produce file-B referencing task-1's output (passed via git branch artifact). This bounds the agent's scope to one artifact at a time and lets each task verify the previous one's output before continuing.
|
||||||
|
|
||||||
|
**Anti-patterns:**
|
||||||
|
- Dispatching a single cheap-model task with "create files X, Y, Z" — silent no-output is the typical result.
|
||||||
|
- Trusting `exit code = 0` from a cheap-model dispatch — always verify the expected file landed on disk or the agent-repo branch.
|
||||||
|
- Using a cheap model for tasks that require sustained state across many turns — the failure mode is silent stall, not error.
|
||||||
|
|
||||||
|
Cross-reference: [Agent Repos](agent-repos.md) "Airouter Qwen3.6 — Scope Decomposition" has the production dispatch pattern and time-to-success bands.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## Summary
|
## Summary
|
||||||
|
|
||||||
Parallel agent orchestration multiplies throughput when agents have narrowly scoped work, disjoint file sets, and the right tool capabilities. The main thread holds responsibilities subagents cannot do: web fetches, file writes on contested paths, and final integration of returned text. For large datasets, a single background sweep with a research-document output usually beats interactive back-and-forth.
|
Parallel agent orchestration multiplies throughput when agents have narrowly scoped work, disjoint file sets, and the right tool capabilities. The main thread holds responsibilities subagents cannot do: web fetches, file writes on contested paths, and final integration of returned text. For large datasets, a single background sweep with a research-document output usually beats interactive back-and-forth. Cost-effective models need tighter scope still — one file, one rule, verified output.
|
||||||
|
|||||||
@@ -147,6 +147,34 @@ When reviewing a PR that touches a spec subsystem:
|
|||||||
- [ ] Cross-references still valid
|
- [ ] Cross-references still valid
|
||||||
- [ ] No requirements removed without deprecation note
|
- [ ] No requirements removed without deprecation note
|
||||||
|
|
||||||
|
### Spec Inversions Must Amend, Not Add
|
||||||
|
|
||||||
|
When a design decision reverses a prior policy in a spec, **amend the existing requirement** and explicitly revoke the inverted text — never add a new requirement that contradicts an existing one. Both rules then coexist and agents follow the wrong one half the time.
|
||||||
|
|
||||||
|
**Pattern:**
|
||||||
|
1. Locate the requirement that is being inverted (e.g., `SH-DENY-4: deny by default`)
|
||||||
|
2. Replace its body with the new policy (e.g., `SH-DENY-4: allow by default unless flagged`) — keep the ID so downstream tests still reference it
|
||||||
|
3. Add a one-line note: "Revokes prior text: 'deny by default'. See decision log entry <ID>."
|
||||||
|
4. Add a CI check (`check_absent`) that fails the build if the revoked phrase reappears in any spec file
|
||||||
|
|
||||||
|
Revoked text creeps back in via copy-paste, AI agent suggestions, or merge conflicts that resolve to "both." The `check_absent` guard catches regressions at PR time, not after deployment. Generalises beyond any single project to any spec or doc with numbered, versioned requirements.
|
||||||
|
|
||||||
|
### Cross-Reference Caller-Side and Callee-Side Spec Shapes
|
||||||
|
|
||||||
|
When two specs reference the same interface — one defines the schema, the other consumes it — run an explicit "do the inputs and outputs match?" review before implementation begins.
|
||||||
|
|
||||||
|
**What to check:**
|
||||||
|
- Field names match exactly (renames in the producer spec, not propagated to the consumer)
|
||||||
|
- Field types match (the producer says `int`, the consumer expects `str`)
|
||||||
|
- Required vs optional matches (the consumer assumes a field exists; the producer marks it optional)
|
||||||
|
- Sample shapes in both specs use the canonical paths/keys, not paraphrases
|
||||||
|
|
||||||
|
**How to run the review:**
|
||||||
|
- Dispatch a cross-spec consistency-check task explicitly: "Given spec A (producer) and spec B (consumer), do their shared interface shapes agree? List any mismatches."
|
||||||
|
- This is a different check from individual spec review — a per-spec review catches internal inconsistency but not cross-spec drift.
|
||||||
|
|
||||||
|
Single-spec review can't catch cross-spec drift; this is the dominant cause of "the integration tests passed but the components don't actually agree" failures. Add it as a named step in the spec workflow.
|
||||||
|
|
||||||
## Context Architecture for Agents
|
## Context Architecture for Agents
|
||||||
|
|
||||||
Based on the Codified Context paper (108k-line system, 283 sessions), structure project knowledge in three tiers:
|
Based on the Codified Context paper (108k-line system, 283 sessions), structure project knowledge in three tiers:
|
||||||
@@ -277,7 +305,11 @@ When changes are small per file (5-15 lines) but tightly coupled across many fil
|
|||||||
|
|
||||||
## Multi-Model Review for Security-Critical Specs
|
## Multi-Model Review for Security-Critical Specs
|
||||||
|
|
||||||
Running the same security review with two different LLM models and comparing outputs catches significantly more issues than either alone. In measured experiments, only 62% of findings overlapped — the union covered 38% more issues. For security-critical specs, the cost of a second model review is justified by the coverage improvement. When models agree, confidence is high; when they disagree, escalate to human review.
|
Running the same security review with two different LLM models and comparing outputs catches significantly more issues than either alone. In measured experiments, only 62% of findings overlapped — the union covered 38% more issues. Reconfirmed across multiple milestones in a second independent project at ~40% additional findings. For security-critical specs, the cost of a second model review is justified by the coverage improvement. When models agree, confidence is high; when they disagree, escalate to human review.
|
||||||
|
|
||||||
|
**Default pairing: different model families.** Pick one model from the Anthropic family (Opus, Sonnet) and one from a different family (MiniMax, Qwen, GPT). Both families have systematic, complementary blindspots — single-model coverage is insufficient regardless of which model. Don't skip the second model just because Claude is the driver session. For specs with substantial security or correctness risk, a third opinion from a third family is justified; for routine specs, two is sufficient.
|
||||||
|
|
||||||
|
**Run in parallel, not sequentially.** Sequential review wastes wall time and — worse — biases the second reviewer if the first reviewer's findings land in the conversation context. Dispatch both reviews concurrently with identical prompts and merge the findings only after both complete.
|
||||||
|
|
||||||
## Security Review Before Agent Implementation
|
## Security Review Before Agent Implementation
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user