distill: flush prior-session best-practice additions
Additive/refining entries left uncommitted in the working tree from an earlier distill session (found during the 2026-07 sweep commit): agent-repos 5-step plan pattern; ai-parallel-agents cheap-model scope; spec-driven spec-inversion and caller/callee cross-reference rules.
This commit is contained in:
@@ -147,6 +147,34 @@ When reviewing a PR that touches a spec subsystem:
|
||||
- [ ] Cross-references still valid
|
||||
- [ ] No requirements removed without deprecation note
|
||||
|
||||
### Spec Inversions Must Amend, Not Add
|
||||
|
||||
When a design decision reverses a prior policy in a spec, **amend the existing requirement** and explicitly revoke the inverted text — never add a new requirement that contradicts an existing one. Both rules then coexist and agents follow the wrong one half the time.
|
||||
|
||||
**Pattern:**
|
||||
1. Locate the requirement that is being inverted (e.g., `SH-DENY-4: deny by default`)
|
||||
2. Replace its body with the new policy (e.g., `SH-DENY-4: allow by default unless flagged`) — keep the ID so downstream tests still reference it
|
||||
3. Add a one-line note: "Revokes prior text: 'deny by default'. See decision log entry <ID>."
|
||||
4. Add a CI check (`check_absent`) that fails the build if the revoked phrase reappears in any spec file
|
||||
|
||||
Revoked text creeps back in via copy-paste, AI agent suggestions, or merge conflicts that resolve to "both." The `check_absent` guard catches regressions at PR time, not after deployment. Generalises beyond any single project to any spec or doc with numbered, versioned requirements.
|
||||
|
||||
### Cross-Reference Caller-Side and Callee-Side Spec Shapes
|
||||
|
||||
When two specs reference the same interface — one defines the schema, the other consumes it — run an explicit "do the inputs and outputs match?" review before implementation begins.
|
||||
|
||||
**What to check:**
|
||||
- Field names match exactly (renames in the producer spec, not propagated to the consumer)
|
||||
- Field types match (the producer says `int`, the consumer expects `str`)
|
||||
- Required vs optional matches (the consumer assumes a field exists; the producer marks it optional)
|
||||
- Sample shapes in both specs use the canonical paths/keys, not paraphrases
|
||||
|
||||
**How to run the review:**
|
||||
- Dispatch a cross-spec consistency-check task explicitly: "Given spec A (producer) and spec B (consumer), do their shared interface shapes agree? List any mismatches."
|
||||
- This is a different check from individual spec review — a per-spec review catches internal inconsistency but not cross-spec drift.
|
||||
|
||||
Single-spec review can't catch cross-spec drift; this is the dominant cause of "the integration tests passed but the components don't actually agree" failures. Add it as a named step in the spec workflow.
|
||||
|
||||
## Context Architecture for Agents
|
||||
|
||||
Based on the Codified Context paper (108k-line system, 283 sessions), structure project knowledge in three tiers:
|
||||
@@ -277,7 +305,11 @@ When changes are small per file (5-15 lines) but tightly coupled across many fil
|
||||
|
||||
## Multi-Model Review for Security-Critical Specs
|
||||
|
||||
Running the same security review with two different LLM models and comparing outputs catches significantly more issues than either alone. In measured experiments, only 62% of findings overlapped — the union covered 38% more issues. For security-critical specs, the cost of a second model review is justified by the coverage improvement. When models agree, confidence is high; when they disagree, escalate to human review.
|
||||
Running the same security review with two different LLM models and comparing outputs catches significantly more issues than either alone. In measured experiments, only 62% of findings overlapped — the union covered 38% more issues. Reconfirmed across multiple milestones in a second independent project at ~40% additional findings. For security-critical specs, the cost of a second model review is justified by the coverage improvement. When models agree, confidence is high; when they disagree, escalate to human review.
|
||||
|
||||
**Default pairing: different model families.** Pick one model from the Anthropic family (Opus, Sonnet) and one from a different family (MiniMax, Qwen, GPT). Both families have systematic, complementary blindspots — single-model coverage is insufficient regardless of which model. Don't skip the second model just because Claude is the driver session. For specs with substantial security or correctness risk, a third opinion from a third family is justified; for routine specs, two is sufficient.
|
||||
|
||||
**Run in parallel, not sequentially.** Sequential review wastes wall time and — worse — biases the second reviewer if the first reviewer's findings land in the conversation context. Dispatch both reviews concurrently with identical prompts and merge the findings only after both complete.
|
||||
|
||||
## Security Review Before Agent Implementation
|
||||
|
||||
|
||||
Reference in New Issue
Block a user