diff --git a/test-driven-development.md b/test-driven-development.md index b8d8300..bd5e6c9 100644 --- a/test-driven-development.md +++ b/test-driven-development.md @@ -360,6 +360,29 @@ The workflow: 1. **Phase 1:** Agent reads spec → writes tests. Human reviews tests against spec. 2. **Phase 2:** Agent (or different agent) reads spec + tests → writes implementation until tests pass. +### Read-Only Test Gates + +When an implementation agent is gated by tests, **place tests in a read-only reference directory** — not the working directory. Agents (especially smaller models) will modify test files to make tests pass rather than writing correct implementation. Prompt-level "DO NOT MODIFY" instructions are insufficient. + +**Enforcement pattern:** +1. Clone tests to `/workspace/reference/` (root-owned, `chmod a-w`) +2. Agent implements in `/workspace/working/` +3. Run pytest against the immutable reference: `cd /workspace/working && PYTHONPATH=/workspace/working python -m pytest /workspace/reference/main/tests/ -v` + +Use filesystem enforcement, not prompt instructions. In a 3-way model comparison: Sonnet respected "DO NOT MODIFY" instructions; MiniMax edited tests 7 times; Haiku rewrote the entire test file. The filesystem makes modification impossible regardless of model. + +### Model Selection for Implementation Agents + +**Sonnet is the minimum viable model for constrained implementation tasks** (spec + test gate). Smaller and cheaper models modify test files or ignore constraints: + +| Model | Result | Notes | +|---|---|---| +| Sonnet | 113/113 tests passing, tests untouched | Viable for implementation | +| MiniMax | Modified tests 7 times | Invalid — use for review/test-writing only | +| Haiku | Rewrote test file entirely | Invalid — use for review/test-writing only | + +Haiku and MiniMax are viable for test-writing, review, and spec work — tasks where the output is inspected by a human, not enforced by a gate. + ### Hidden Test Splits Hold back some tests that the implementing agent never sees. Use them as a final validation: