distill: flush prior-session best-practice additions

Additive/refining entries left uncommitted in the working tree from an earlier
distill session (found during the 2026-07 sweep commit): agent-repos 5-step
plan pattern; ai-parallel-agents cheap-model scope; spec-driven spec-inversion
and caller/callee cross-reference rules.
This commit is contained in:
Paul O'Reilly
2026-07-02 15:58:52 +12:00
parent 7e348f5ee3
commit c8691368e4
3 changed files with 93 additions and 3 deletions

View File

@@ -436,10 +436,24 @@ Invest in pre-dispatch validation scripts that catch the top-N infrastructure fa
## Cross-Model Reviews Catch ~38% More Issues Than a Single Model ## Cross-Model Reviews Catch ~38% More Issues Than a Single Model
Running the same security or spec review with two different models (e.g., Opus + MiniMax) and comparing outputs catches ~38% more issues than either alone — in measured reviews, only 62% of findings overlap. Models converge on obvious issues but diverge on edge cases and design concerns. Worth the extra cost for security-critical specs and architecture reviews; overkill for routine code review. Running the same security or spec review with two different models (e.g., Opus + MiniMax) and comparing outputs catches ~38% more issues than either alone — in measured reviews, only 62% of findings overlap. Reconfirmed at ~40% additional findings across multiple milestones in a second independent project — the effect is consistent, not noise. Models converge on obvious issues but diverge on edge cases and design concerns. Worth the extra cost for security-critical specs and architecture reviews; overkill for routine code review.
**Pattern:** dispatch parallel review tasks to different models with identical prompts, union the findings, deduplicate against a shared issue key (file + line + category). Present the merged list to the human reviewer along with per-model attribution so reviewers can see where models agreed vs. diverged. **Pattern:** dispatch parallel review tasks to different models with identical prompts, union the findings, deduplicate against a shared issue key (file + line + category). Present the merged list to the human reviewer along with per-model attribution so reviewers can see where models agreed vs. diverged.
**Run in parallel, not sequentially.** Sequential review wastes wall time AND biases the second reviewer if the first's output lands in the shared context. Concurrent dispatch with identical prompts is the correct shape; merge findings after both complete.
## Plan → Review → Decisions → V2 → Consistency-Check (5-Step Pattern)
For non-trivial plans (multi-subsystem features, security-critical changes, cross-team boundaries), use a named 5-step workflow rather than ad-hoc iteration:
1. **Plan** — produce the initial PLAN.md from the architecture conversation.
2. **Review** — dispatch cross-model reviews in parallel (Opus + a different family). Both reviewers receive the same plan and the same review prompt.
3. **Decisions** — write a DECISIONS.md capturing the outcome of each review finding: accepted, rejected (with rationale), or deferred. The decisions file is a permanent record of why the plan looks the way it does.
4. **V2** — produce PLAN-v2.md applying all accepted decisions. Keep V1 alongside V2; don't overwrite.
5. **Consistency-check** — automated grep/script pass to confirm: revoked phrases don't reappear, paired schema shapes match across docs, all referenced requirement IDs resolve. Skipping this lets spec inversions ship — the revoked text remains alongside the new policy and agents follow the wrong one half the time.
Phase 5 is the one most teams skip and the one that prevents the largest class of multi-day fix-up cycles. Treat it as a mandatory CI step on any PR that touches a plan or spec.
## Agent Worktree Branches Contain Files, Not Commits — Copy, Don't Merge ## Agent Worktree Branches Contain Files, Not Commits — Copy, Don't Merge
**Symptom:** Orchestrator merges an agent's branch and sees "Already up to date" because the agent wrote files to its worktree but never ran `git add` / `git commit`. Downstream tasks that depend on the upstream artifact then fail or silently use stale data. **Symptom:** Orchestrator merges an agent's branch and sees "Already up to date" because the agent wrote files to its worktree but never ran `git add` / `git commit`. Downstream tasks that depend on the upstream artifact then fail or silently use stale data.

View File

@@ -104,8 +104,52 @@ Cross-references: [Agent Repos & Container Agent Operations](agent-repos.md) cov
- No dedicated output folder — research artifacts get lost alongside code. - No dedicated output folder — research artifacts get lost alongside code.
- Never reading the research output after dispatch — a "fire and forget" audit with no human review has no value. - Never reading the research output after dispatch — a "fire and forget" audit with no human review has no value.
### Cheap-Model Sub-Sessions: When to Use, When to Skip
Tools that fork a cheap-model session for grunt work (`/ask-minimax`, similar MiniMax-, Qwen-, or Haiku-backed delegators) shine on a narrow band of tasks. Reach for them when **the file payload dwarfs the answer payload**.
**Use for:**
- Summarising large files (logs, dumps, generated reports >500 lines)
- Extracting specific facts from many files or long reference docs
- Generating big files (migrations, fixtures, large docs) whose content does not need to flow back
- Format conversion of large files (CSV↔JSON, XML↔YAML)
- Bulk find-and-extract where only the matches matter
**Skip for:**
- Iterative design or debugging — the parent session needs the content in context
- Small files (<500 lines) — overhead exceeds savings
- Tasks where you will immediately re-read the result to act on it
- Anything requiring tools the cheap model lacks (web fetch, MCP, browser, agent dispatch)
- Architecture or quality-sensitive output — cheap models are for grunt work, not nuanced reasoning
- **Per-test verification loops** — the LLM round-trip cost dwarfs the test's own cost; the cheap model also stalls silently on tasks requiring sustained state across iterations. Replace with an AST script.
**Delegate by reference, not content:** pass file paths, not file contents. If you read the input files yourself before invoking, you have already paid the token cost the skill exists to avoid.
---
## 6. Cost-Effective Models: Single-File, Single-Rule Task Scope
**Principle:** Cheap/small models (MiniMax, Qwen3.x, Haiku via airouter) reliably produce output when scoped to ONE file and ONE rule per task. Multi-file or multi-rule tasks silently produce no output even when the agent's reasoning logs look correct.
**Why it matters:** Cost-effective models trade context-management ability for token cost. With multi-file scope, the model loses track of which file it has already edited and produces empty output or commits without the expected files. The failure is silent — exit code 0, reasoning logs look fine, no output file. Verified in a 9-dispatch batch: single-file scope succeeded 8/8, multi-file scope failed 2/2 with no output.
**How to implement:**
- **One file per task** — split multi-file changes into separate dispatches. Each task creates or edits exactly one file.
- **Reference the template, don't inline it** — give the agent the path to an existing similar file to read, rather than including all structure in the prompt.
- **State the file path explicitly** — "Create `<exact/path/to/file>`" at the start of the prompt, not implied by context.
- **Verify output before treating as done** — check the agent-repo branch (or output directory) for the expected file immediately after task success. Don't trust the exit code alone.
**Sequential chain for unavoidable multi-file work:** dispatch task-1 to produce file-A, wait for success, then dispatch task-2 to produce file-B referencing task-1's output (passed via git branch artifact). This bounds the agent's scope to one artifact at a time and lets each task verify the previous one's output before continuing.
**Anti-patterns:**
- Dispatching a single cheap-model task with "create files X, Y, Z" — silent no-output is the typical result.
- Trusting `exit code = 0` from a cheap-model dispatch — always verify the expected file landed on disk or the agent-repo branch.
- Using a cheap model for tasks that require sustained state across many turns — the failure mode is silent stall, not error.
Cross-reference: [Agent Repos](agent-repos.md) "Airouter Qwen3.6 — Scope Decomposition" has the production dispatch pattern and time-to-success bands.
--- ---
## Summary ## Summary
Parallel agent orchestration multiplies throughput when agents have narrowly scoped work, disjoint file sets, and the right tool capabilities. The main thread holds responsibilities subagents cannot do: web fetches, file writes on contested paths, and final integration of returned text. For large datasets, a single background sweep with a research-document output usually beats interactive back-and-forth. Parallel agent orchestration multiplies throughput when agents have narrowly scoped work, disjoint file sets, and the right tool capabilities. The main thread holds responsibilities subagents cannot do: web fetches, file writes on contested paths, and final integration of returned text. For large datasets, a single background sweep with a research-document output usually beats interactive back-and-forth. Cost-effective models need tighter scope still — one file, one rule, verified output.

View File

@@ -147,6 +147,34 @@ When reviewing a PR that touches a spec subsystem:
- [ ] Cross-references still valid - [ ] Cross-references still valid
- [ ] No requirements removed without deprecation note - [ ] No requirements removed without deprecation note
### Spec Inversions Must Amend, Not Add
When a design decision reverses a prior policy in a spec, **amend the existing requirement** and explicitly revoke the inverted text — never add a new requirement that contradicts an existing one. Both rules then coexist and agents follow the wrong one half the time.
**Pattern:**
1. Locate the requirement that is being inverted (e.g., `SH-DENY-4: deny by default`)
2. Replace its body with the new policy (e.g., `SH-DENY-4: allow by default unless flagged`) — keep the ID so downstream tests still reference it
3. Add a one-line note: "Revokes prior text: 'deny by default'. See decision log entry <ID>."
4. Add a CI check (`check_absent`) that fails the build if the revoked phrase reappears in any spec file
Revoked text creeps back in via copy-paste, AI agent suggestions, or merge conflicts that resolve to "both." The `check_absent` guard catches regressions at PR time, not after deployment. Generalises beyond any single project to any spec or doc with numbered, versioned requirements.
### Cross-Reference Caller-Side and Callee-Side Spec Shapes
When two specs reference the same interface — one defines the schema, the other consumes it — run an explicit "do the inputs and outputs match?" review before implementation begins.
**What to check:**
- Field names match exactly (renames in the producer spec, not propagated to the consumer)
- Field types match (the producer says `int`, the consumer expects `str`)
- Required vs optional matches (the consumer assumes a field exists; the producer marks it optional)
- Sample shapes in both specs use the canonical paths/keys, not paraphrases
**How to run the review:**
- Dispatch a cross-spec consistency-check task explicitly: "Given spec A (producer) and spec B (consumer), do their shared interface shapes agree? List any mismatches."
- This is a different check from individual spec review — a per-spec review catches internal inconsistency but not cross-spec drift.
Single-spec review can't catch cross-spec drift; this is the dominant cause of "the integration tests passed but the components don't actually agree" failures. Add it as a named step in the spec workflow.
## Context Architecture for Agents ## Context Architecture for Agents
Based on the Codified Context paper (108k-line system, 283 sessions), structure project knowledge in three tiers: Based on the Codified Context paper (108k-line system, 283 sessions), structure project knowledge in three tiers:
@@ -277,7 +305,11 @@ When changes are small per file (5-15 lines) but tightly coupled across many fil
## Multi-Model Review for Security-Critical Specs ## Multi-Model Review for Security-Critical Specs
Running the same security review with two different LLM models and comparing outputs catches significantly more issues than either alone. In measured experiments, only 62% of findings overlapped — the union covered 38% more issues. For security-critical specs, the cost of a second model review is justified by the coverage improvement. When models agree, confidence is high; when they disagree, escalate to human review. Running the same security review with two different LLM models and comparing outputs catches significantly more issues than either alone. In measured experiments, only 62% of findings overlapped — the union covered 38% more issues. Reconfirmed across multiple milestones in a second independent project at ~40% additional findings. For security-critical specs, the cost of a second model review is justified by the coverage improvement. When models agree, confidence is high; when they disagree, escalate to human review.
**Default pairing: different model families.** Pick one model from the Anthropic family (Opus, Sonnet) and one from a different family (MiniMax, Qwen, GPT). Both families have systematic, complementary blindspots — single-model coverage is insufficient regardless of which model. Don't skip the second model just because Claude is the driver session. For specs with substantial security or correctness risk, a third opinion from a third family is justified; for routine specs, two is sufficient.
**Run in parallel, not sequentially.** Sequential review wastes wall time and — worse — biases the second reviewer if the first reviewer's findings land in the conversation context. Dispatch both reviews concurrently with identical prompts and merge the findings only after both complete.
## Security Review Before Agent Implementation ## Security Review Before Agent Implementation