distill: flush prior-session best-practice additions

Additive/refining entries left uncommitted in the working tree from an earlier
distill session (found during the 2026-07 sweep commit): agent-repos 5-step
plan pattern; ai-parallel-agents cheap-model scope; spec-driven spec-inversion
and caller/callee cross-reference rules.
This commit is contained in:
Paul O'Reilly
2026-07-02 15:58:52 +12:00
parent 7e348f5ee3
commit c8691368e4
3 changed files with 93 additions and 3 deletions

View File

@@ -147,6 +147,34 @@ When reviewing a PR that touches a spec subsystem:
- [ ] Cross-references still valid
- [ ] No requirements removed without deprecation note
### Spec Inversions Must Amend, Not Add
When a design decision reverses a prior policy in a spec, **amend the existing requirement** and explicitly revoke the inverted text — never add a new requirement that contradicts an existing one. Both rules then coexist and agents follow the wrong one half the time.
**Pattern:**
1. Locate the requirement that is being inverted (e.g., `SH-DENY-4: deny by default`)
2. Replace its body with the new policy (e.g., `SH-DENY-4: allow by default unless flagged`) — keep the ID so downstream tests still reference it
3. Add a one-line note: "Revokes prior text: 'deny by default'. See decision log entry <ID>."
4. Add a CI check (`check_absent`) that fails the build if the revoked phrase reappears in any spec file
Revoked text creeps back in via copy-paste, AI agent suggestions, or merge conflicts that resolve to "both." The `check_absent` guard catches regressions at PR time, not after deployment. Generalises beyond any single project to any spec or doc with numbered, versioned requirements.
### Cross-Reference Caller-Side and Callee-Side Spec Shapes
When two specs reference the same interface — one defines the schema, the other consumes it — run an explicit "do the inputs and outputs match?" review before implementation begins.
**What to check:**
- Field names match exactly (renames in the producer spec, not propagated to the consumer)
- Field types match (the producer says `int`, the consumer expects `str`)
- Required vs optional matches (the consumer assumes a field exists; the producer marks it optional)
- Sample shapes in both specs use the canonical paths/keys, not paraphrases
**How to run the review:**
- Dispatch a cross-spec consistency-check task explicitly: "Given spec A (producer) and spec B (consumer), do their shared interface shapes agree? List any mismatches."
- This is a different check from individual spec review — a per-spec review catches internal inconsistency but not cross-spec drift.
Single-spec review can't catch cross-spec drift; this is the dominant cause of "the integration tests passed but the components don't actually agree" failures. Add it as a named step in the spec workflow.
## Context Architecture for Agents
Based on the Codified Context paper (108k-line system, 283 sessions), structure project knowledge in three tiers:
@@ -277,7 +305,11 @@ When changes are small per file (5-15 lines) but tightly coupled across many fil
## Multi-Model Review for Security-Critical Specs
Running the same security review with two different LLM models and comparing outputs catches significantly more issues than either alone. In measured experiments, only 62% of findings overlapped — the union covered 38% more issues. For security-critical specs, the cost of a second model review is justified by the coverage improvement. When models agree, confidence is high; when they disagree, escalate to human review.
Running the same security review with two different LLM models and comparing outputs catches significantly more issues than either alone. In measured experiments, only 62% of findings overlapped — the union covered 38% more issues. Reconfirmed across multiple milestones in a second independent project at ~40% additional findings. For security-critical specs, the cost of a second model review is justified by the coverage improvement. When models agree, confidence is high; when they disagree, escalate to human review.
**Default pairing: different model families.** Pick one model from the Anthropic family (Opus, Sonnet) and one from a different family (MiniMax, Qwen, GPT). Both families have systematic, complementary blindspots — single-model coverage is insufficient regardless of which model. Don't skip the second model just because Claude is the driver session. For specs with substantial security or correctness risk, a third opinion from a third family is justified; for routine specs, two is sufficient.
**Run in parallel, not sequentially.** Sequential review wastes wall time and — worse — biases the second reviewer if the first reviewer's findings land in the conversation context. Dispatch both reviews concurrently with identical prompts and merge the findings only after both complete.
## Security Review Before Agent Implementation