From c8691368e462eced46d0e095c66440259fd3eded Mon Sep 17 00:00:00 2001 From: Paul O'Reilly Date: Thu, 2 Jul 2026 15:58:52 +1200 Subject: [PATCH] distill: flush prior-session best-practice additions Additive/refining entries left uncommitted in the working tree from an earlier distill session (found during the 2026-07 sweep commit): agent-repos 5-step plan pattern; ai-parallel-agents cheap-model scope; spec-driven spec-inversion and caller/callee cross-reference rules. --- agent-repos.md | 16 ++++++++++++- ai-parallel-agents.md | 46 +++++++++++++++++++++++++++++++++++++- spec-driven-development.md | 34 +++++++++++++++++++++++++++- 3 files changed, 93 insertions(+), 3 deletions(-) diff --git a/agent-repos.md b/agent-repos.md index 6d5892c..b8f6d29 100644 --- a/agent-repos.md +++ b/agent-repos.md @@ -436,10 +436,24 @@ Invest in pre-dispatch validation scripts that catch the top-N infrastructure fa ## Cross-Model Reviews Catch ~38% More Issues Than a Single Model -Running the same security or spec review with two different models (e.g., Opus + MiniMax) and comparing outputs catches ~38% more issues than either alone — in measured reviews, only 62% of findings overlap. Models converge on obvious issues but diverge on edge cases and design concerns. Worth the extra cost for security-critical specs and architecture reviews; overkill for routine code review. +Running the same security or spec review with two different models (e.g., Opus + MiniMax) and comparing outputs catches ~38% more issues than either alone — in measured reviews, only 62% of findings overlap. Reconfirmed at ~40% additional findings across multiple milestones in a second independent project — the effect is consistent, not noise. Models converge on obvious issues but diverge on edge cases and design concerns. Worth the extra cost for security-critical specs and architecture reviews; overkill for routine code review. **Pattern:** dispatch parallel review tasks to different models with identical prompts, union the findings, deduplicate against a shared issue key (file + line + category). Present the merged list to the human reviewer along with per-model attribution so reviewers can see where models agreed vs. diverged. +**Run in parallel, not sequentially.** Sequential review wastes wall time AND biases the second reviewer if the first's output lands in the shared context. Concurrent dispatch with identical prompts is the correct shape; merge findings after both complete. + +## Plan → Review → Decisions → V2 → Consistency-Check (5-Step Pattern) + +For non-trivial plans (multi-subsystem features, security-critical changes, cross-team boundaries), use a named 5-step workflow rather than ad-hoc iteration: + +1. **Plan** — produce the initial PLAN.md from the architecture conversation. +2. **Review** — dispatch cross-model reviews in parallel (Opus + a different family). Both reviewers receive the same plan and the same review prompt. +3. **Decisions** — write a DECISIONS.md capturing the outcome of each review finding: accepted, rejected (with rationale), or deferred. The decisions file is a permanent record of why the plan looks the way it does. +4. **V2** — produce PLAN-v2.md applying all accepted decisions. Keep V1 alongside V2; don't overwrite. +5. **Consistency-check** — automated grep/script pass to confirm: revoked phrases don't reappear, paired schema shapes match across docs, all referenced requirement IDs resolve. Skipping this lets spec inversions ship — the revoked text remains alongside the new policy and agents follow the wrong one half the time. + +Phase 5 is the one most teams skip and the one that prevents the largest class of multi-day fix-up cycles. Treat it as a mandatory CI step on any PR that touches a plan or spec. + ## Agent Worktree Branches Contain Files, Not Commits — Copy, Don't Merge **Symptom:** Orchestrator merges an agent's branch and sees "Already up to date" because the agent wrote files to its worktree but never ran `git add` / `git commit`. Downstream tasks that depend on the upstream artifact then fail or silently use stale data. diff --git a/ai-parallel-agents.md b/ai-parallel-agents.md index 884f840..ab91fe6 100644 --- a/ai-parallel-agents.md +++ b/ai-parallel-agents.md @@ -104,8 +104,52 @@ Cross-references: [Agent Repos & Container Agent Operations](agent-repos.md) cov - No dedicated output folder — research artifacts get lost alongside code. - Never reading the research output after dispatch — a "fire and forget" audit with no human review has no value. +### Cheap-Model Sub-Sessions: When to Use, When to Skip + +Tools that fork a cheap-model session for grunt work (`/ask-minimax`, similar MiniMax-, Qwen-, or Haiku-backed delegators) shine on a narrow band of tasks. Reach for them when **the file payload dwarfs the answer payload**. + +**Use for:** +- Summarising large files (logs, dumps, generated reports >500 lines) +- Extracting specific facts from many files or long reference docs +- Generating big files (migrations, fixtures, large docs) whose content does not need to flow back +- Format conversion of large files (CSV↔JSON, XML↔YAML) +- Bulk find-and-extract where only the matches matter + +**Skip for:** +- Iterative design or debugging — the parent session needs the content in context +- Small files (<500 lines) — overhead exceeds savings +- Tasks where you will immediately re-read the result to act on it +- Anything requiring tools the cheap model lacks (web fetch, MCP, browser, agent dispatch) +- Architecture or quality-sensitive output — cheap models are for grunt work, not nuanced reasoning +- **Per-test verification loops** — the LLM round-trip cost dwarfs the test's own cost; the cheap model also stalls silently on tasks requiring sustained state across iterations. Replace with an AST script. + +**Delegate by reference, not content:** pass file paths, not file contents. If you read the input files yourself before invoking, you have already paid the token cost the skill exists to avoid. + +--- + +## 6. Cost-Effective Models: Single-File, Single-Rule Task Scope + +**Principle:** Cheap/small models (MiniMax, Qwen3.x, Haiku via airouter) reliably produce output when scoped to ONE file and ONE rule per task. Multi-file or multi-rule tasks silently produce no output even when the agent's reasoning logs look correct. + +**Why it matters:** Cost-effective models trade context-management ability for token cost. With multi-file scope, the model loses track of which file it has already edited and produces empty output or commits without the expected files. The failure is silent — exit code 0, reasoning logs look fine, no output file. Verified in a 9-dispatch batch: single-file scope succeeded 8/8, multi-file scope failed 2/2 with no output. + +**How to implement:** +- **One file per task** — split multi-file changes into separate dispatches. Each task creates or edits exactly one file. +- **Reference the template, don't inline it** — give the agent the path to an existing similar file to read, rather than including all structure in the prompt. +- **State the file path explicitly** — "Create ``" at the start of the prompt, not implied by context. +- **Verify output before treating as done** — check the agent-repo branch (or output directory) for the expected file immediately after task success. Don't trust the exit code alone. + +**Sequential chain for unavoidable multi-file work:** dispatch task-1 to produce file-A, wait for success, then dispatch task-2 to produce file-B referencing task-1's output (passed via git branch artifact). This bounds the agent's scope to one artifact at a time and lets each task verify the previous one's output before continuing. + +**Anti-patterns:** +- Dispatching a single cheap-model task with "create files X, Y, Z" — silent no-output is the typical result. +- Trusting `exit code = 0` from a cheap-model dispatch — always verify the expected file landed on disk or the agent-repo branch. +- Using a cheap model for tasks that require sustained state across many turns — the failure mode is silent stall, not error. + +Cross-reference: [Agent Repos](agent-repos.md) "Airouter Qwen3.6 — Scope Decomposition" has the production dispatch pattern and time-to-success bands. + --- ## Summary -Parallel agent orchestration multiplies throughput when agents have narrowly scoped work, disjoint file sets, and the right tool capabilities. The main thread holds responsibilities subagents cannot do: web fetches, file writes on contested paths, and final integration of returned text. For large datasets, a single background sweep with a research-document output usually beats interactive back-and-forth. +Parallel agent orchestration multiplies throughput when agents have narrowly scoped work, disjoint file sets, and the right tool capabilities. The main thread holds responsibilities subagents cannot do: web fetches, file writes on contested paths, and final integration of returned text. For large datasets, a single background sweep with a research-document output usually beats interactive back-and-forth. Cost-effective models need tighter scope still — one file, one rule, verified output. diff --git a/spec-driven-development.md b/spec-driven-development.md index 714458e..f84b318 100644 --- a/spec-driven-development.md +++ b/spec-driven-development.md @@ -147,6 +147,34 @@ When reviewing a PR that touches a spec subsystem: - [ ] Cross-references still valid - [ ] No requirements removed without deprecation note +### Spec Inversions Must Amend, Not Add + +When a design decision reverses a prior policy in a spec, **amend the existing requirement** and explicitly revoke the inverted text — never add a new requirement that contradicts an existing one. Both rules then coexist and agents follow the wrong one half the time. + +**Pattern:** +1. Locate the requirement that is being inverted (e.g., `SH-DENY-4: deny by default`) +2. Replace its body with the new policy (e.g., `SH-DENY-4: allow by default unless flagged`) — keep the ID so downstream tests still reference it +3. Add a one-line note: "Revokes prior text: 'deny by default'. See decision log entry ." +4. Add a CI check (`check_absent`) that fails the build if the revoked phrase reappears in any spec file + +Revoked text creeps back in via copy-paste, AI agent suggestions, or merge conflicts that resolve to "both." The `check_absent` guard catches regressions at PR time, not after deployment. Generalises beyond any single project to any spec or doc with numbered, versioned requirements. + +### Cross-Reference Caller-Side and Callee-Side Spec Shapes + +When two specs reference the same interface — one defines the schema, the other consumes it — run an explicit "do the inputs and outputs match?" review before implementation begins. + +**What to check:** +- Field names match exactly (renames in the producer spec, not propagated to the consumer) +- Field types match (the producer says `int`, the consumer expects `str`) +- Required vs optional matches (the consumer assumes a field exists; the producer marks it optional) +- Sample shapes in both specs use the canonical paths/keys, not paraphrases + +**How to run the review:** +- Dispatch a cross-spec consistency-check task explicitly: "Given spec A (producer) and spec B (consumer), do their shared interface shapes agree? List any mismatches." +- This is a different check from individual spec review — a per-spec review catches internal inconsistency but not cross-spec drift. + +Single-spec review can't catch cross-spec drift; this is the dominant cause of "the integration tests passed but the components don't actually agree" failures. Add it as a named step in the spec workflow. + ## Context Architecture for Agents Based on the Codified Context paper (108k-line system, 283 sessions), structure project knowledge in three tiers: @@ -277,7 +305,11 @@ When changes are small per file (5-15 lines) but tightly coupled across many fil ## Multi-Model Review for Security-Critical Specs -Running the same security review with two different LLM models and comparing outputs catches significantly more issues than either alone. In measured experiments, only 62% of findings overlapped — the union covered 38% more issues. For security-critical specs, the cost of a second model review is justified by the coverage improvement. When models agree, confidence is high; when they disagree, escalate to human review. +Running the same security review with two different LLM models and comparing outputs catches significantly more issues than either alone. In measured experiments, only 62% of findings overlapped — the union covered 38% more issues. Reconfirmed across multiple milestones in a second independent project at ~40% additional findings. For security-critical specs, the cost of a second model review is justified by the coverage improvement. When models agree, confidence is high; when they disagree, escalate to human review. + +**Default pairing: different model families.** Pick one model from the Anthropic family (Opus, Sonnet) and one from a different family (MiniMax, Qwen, GPT). Both families have systematic, complementary blindspots — single-model coverage is insufficient regardless of which model. Don't skip the second model just because Claude is the driver session. For specs with substantial security or correctness risk, a third opinion from a third family is justified; for routine specs, two is sufficient. + +**Run in parallel, not sequentially.** Sequential review wastes wall time and — worse — biases the second reviewer if the first reviewer's findings land in the conversation context. Dispatch both reviews concurrently with identical prompts and merge the findings only after both complete. ## Security Review Before Agent Implementation