best-practices/test-driven-development.md

# Test-Driven Development for Spec-Based Projects

Best practices for writing comprehensive, regression-catching tests in projects that use structured specifications. Focuses on maximising test value (catching real bugs) rather than test volume (inflating coverage numbers). Extracted from industry research, academic papers (TDAD, Codified Context), and practitioner experience.

## Core Principle: Tests Are the Spec's Enforcement Layer

In a spec-driven project, the spec defines *what* and the tests *prove it*. A requirement without a test is an aspiration. A test without a requirement is undocumented behaviour. Keep them tightly coupled:

- Every numbered requirement (P-1, E-3) has at least one test
- Every test function name includes its requirement ID: `test_e3_preaction_failure_exits_1`
- Spec changes and test changes ship in the same commit

## Deriving Tests from Specs

### Requirements to Tests

Each spec requirement becomes one or more test functions. The mapping isn't always 1:1 — a requirement like "must respect timeout" needs tests for: default timeout, explicit timeout, timeout=0 (no limit), timeout exceeded.

```python
# From spec: R-4: Runners must respect runtime.timeout.
# Default 3600s. Value of 0 means no timeout.

def test_r4_default_timeout_is_3600():
    """R-4: When timeout not specified, default is 3600s."""

def test_r4_explicit_timeout_is_honoured():
    """R-4: When timeout=60, process killed after 60s."""

def test_r4_zero_timeout_means_no_limit():
    """R-4: When timeout=0, no timeout is applied."""

def test_r4_timeout_returns_exit_code_124():
    """R-4 + R-5: Timeout produces exit code 124."""
```

### Scenarios to Tests

GIVEN/WHEN/THEN scenarios translate directly to Arrange/Act/Assert:

```python
def test_scenario_preaction_failure_runs_on_error():
    """Given clone fails, on_error runs and exits 1. Runner never executes."""
    # GIVEN — arrange
    payload = make_payload(pre_actions=[{"action": "clone", "repo": "bad-url"}])
    mock_clone = Mock(side_effect=subprocess.CalledProcessError(128, "git"))

    # WHEN — act
    exit_code = run_entrypoint(payload, clone_handler=mock_clone)

    # THEN — assert
    assert exit_code == 1
    mock_runner.assert_not_called()
    mock_on_error.assert_called_once()
```

### Parameterised Tests from Spec Enumerations

When a spec lists multiple valid values, use `@pytest.mark.parametrize`:

```python
# From spec: task states are pending, assigned, running, succeeded, failed, timed_out, cancelled
@pytest.mark.parametrize("terminal_state", ["succeeded", "failed", "timed_out", "cancelled"])
def test_cp_terminal_state_cannot_be_overwritten(terminal_state):
    """CP: Terminal states reject further transitions with 409."""
```

## Systematic Edge Case Discovery

~80% of bugs cluster at boundaries. Use these techniques to find edge cases systematically rather than by intuition.

### Boundary Value Analysis

For every input parameter, test at the edges of its valid range:

| Input type | Test values |
|---|---|
| Integer (range 1-100) | 0, 1, 2, 99, 100, 101, -1, MAX_INT |
| String | `""`, `"a"`, max-length string, max+1, unicode (`"\u0000"`, emoji), whitespace-only |
| List/Array | `[]`, `[single]`, many items, duplicates, `None` |
| Dict/Map | `{}`, missing required keys, extra unknown keys, `None` values |
| Timeout (seconds) | 0, 1, -1, very large (999999), `None`/missing |
| Base64 | valid, invalid chars, empty, padding variants (`=`, `==`, none) |

### Equivalence Partitioning

Group inputs into classes where all members should behave identically. Test one from each class:

```python
# Payload validation: prompt field
# Class 1: valid string → accepted
# Class 2: empty string → rejected (spec says prompt is required)
# Class 3: missing key → rejected
# Class 4: wrong type (int, list, None) → rejected
# Class 5: very long string → accepted (no length limit in spec)

@pytest.mark.parametrize("prompt,should_pass", [
    ("Fix the bug", True),          # Class 1: valid
    ("", False),                     # Class 2: empty
    (None, False),                   # Class 3: missing/None
    (42, False),                     # Class 4: wrong type
    ("x" * 100_000, True),          # Class 5: long string
])
def test_p_prompt_validation(prompt, should_pass):
    ...
```

### State Transition Coverage

For state machines (task states, dispatcher states), test:

1. **Every valid transition:** `pending → assigned → running → succeeded`
2. **Every invalid transition:** `succeeded → running` (should be rejected)
3. **Initial state:** newly created tasks start in `pending`
4. **Terminal states:** `succeeded`, `failed`, `timed_out`, `cancelled` cannot transition further
5. **Re-entrant transitions:** same state → same state (should be idempotent or rejected, per spec)

```python
VALID_TRANSITIONS = [
    ("pending", "assigned"),
    ("assigned", "running"),
    ("running", "succeeded"),
    ("running", "failed"),
    ("running", "timed_out"),
    ("assigned", "cancelled"),
    ("running", "cancelled"),
]

INVALID_TRANSITIONS = [
    ("succeeded", "failed"),
    ("failed", "running"),
    ("cancelled", "pending"),
    ("timed_out", "running"),
]

@pytest.mark.parametrize("from_state,to_state", VALID_TRANSITIONS)
def test_valid_state_transition(from_state, to_state):
    ...

@pytest.mark.parametrize("from_state,to_state", INVALID_TRANSITIONS)
def test_invalid_state_transition_rejected(from_state, to_state):
    ...
```

### The Edge Case Checklist

Walk through this for every function under test:

1. **Empty/null inputs** — what happens when required fields are missing?
2. **Boundary values** — min, max, zero, negative, off-by-one
3. **Type mismatches** — string where int expected, list where dict expected
4. **Malformed input** — invalid JSON, bad base64, truncated data
5. **Concurrent operations** — two tasks claiming the same resource
6. **Ordering** — actions that depend on sequence (pre-action before runner)
7. **Idempotency** — calling the same operation twice (kill an already-killed container)
8. **Resource exhaustion** — at capacity, disk full, timeout expired
9. **Partial failure** — first action succeeds, second fails (cleanup?)

## Property-Based Testing with Hypothesis

Instead of specifying individual test cases, define *properties* that must hold for all inputs. Hypothesis generates hundreds of inputs including edge cases you'd never think of.

### When to Use Property-Based Testing

- **Serialisation roundtrips:** encode → decode returns original
- **Parsers:** should never crash on any input
- **Data transformations:** invariants that hold regardless of input
- **Validators:** valid inputs accepted, invalid inputs rejected (never crash)

### When NOT to Use It

- Tests where generating valid inputs is harder than the code itself
- Tests where the "property" just restates the implementation
- UI or integration tests

### Patterns

```python
from hypothesis import given, strategies as st, assume, settings
from hypothesis import example

# Roundtrip: base64 encode/decode preserves payload
@given(st.text())
def test_base64_roundtrip(payload_str):
    encoded = base64.b64encode(payload_str.encode()).decode()
    decoded = base64.b64decode(encoded).decode()
    assert decoded == payload_str

# Invariant: payload validation never crashes (may reject, never exception)
@given(st.dictionaries(st.text(), st.text() | st.integers() | st.none()))
def test_payload_validation_never_crashes(raw_payload):
    # Should return True/False or raise ValidationError — never unhandled exception
    try:
        validate_payload(raw_payload)
    except ValidationError:
        pass  # Expected for invalid input

# Pin known edge cases alongside random generation
@example("")           # empty string
@example("\x00")       # null byte
@example("a" * 10**6)  # very long
@given(st.text())
def test_prompt_handling(prompt):
    ...

# Composite strategies for domain objects
@st.composite
def valid_payloads(draw):
    return {
        "task_id": draw(st.uuids()).hex,
        "prompt": draw(st.text(min_size=1)),
        "runtime": {"cli": draw(st.sampled_from(["claude", "codex"]))},
    }

@given(valid_payloads())
def test_valid_payload_always_accepted(payload):
    assert validate_payload(payload) is True
```

### Stateful Testing for State Machines

Hypothesis can generate sequences of operations and check invariants after each step:

```python
from hypothesis.stateful import RuleBasedStateMachine, rule, precondition

class TaskStateMachine(RuleBasedStateMachine):
    def __init__(self):
        super().__init__()
        self.task = Task(state="pending")

    @rule()
    @precondition(lambda self: self.task.state == "pending")
    def assign(self):
        self.task.transition("assigned")
        assert self.task.state == "assigned"

    @rule()
    @precondition(lambda self: self.task.state == "running")
    def complete(self):
        self.task.transition("succeeded")
        assert self.task.state == "succeeded"

    # Invariant: terminal states never change
    @invariant()
    def terminal_states_are_final(self):
        if self.task.state in ("succeeded", "failed", "cancelled"):
            with pytest.raises(InvalidTransition):
                self.task.transition("running")

TestTaskStates = TaskStateMachine.TestCase
```

## Mutation Testing

Mutation testing answers: "If someone introduced a bug, would our tests catch it?"

Tools make small code changes (replacing `>` with `>=`, `True` with `False`, deleting statements) and check if tests still pass. Surviving mutants = test gaps.

### Setup with mutmut

```toml
# pyproject.toml
[tool.mutmut]
paths_to_mutate = "entrypoint/"
tests_dir = "tests/"
runner = "python -m pytest tests/ -x -q"
```

```bash
# Run mutation testing
mutmut run

# See surviving mutants
mutmut results

# Inspect a specific mutant
mutmut show 42
```

### Practical Guidance

- **Target: mutation score above 80%.** Scores above 90% have diminishing returns (equivalent mutants).
- **Focus on business logic** — validators, state machines, parsers. Skip glue code.
- **Use mutation testing to audit AI-generated tests.** This is the most powerful combination: AI writes tests from spec, mutation testing verifies those tests catch real faults.
- **Run on changed files only in CI** (full suite is slow). Full run nightly or pre-release.

## Test Architecture

### The Testing Pyramid for Spec-Driven Projects

| Layer | Proportion | Speed | What it catches |
|---|---|---|---|
| Unit tests | 60-70% | <1ms each | Logic errors, boundary violations, state machine bugs |
| Property-based | 10-15% | ~10ms each | Edge cases humans miss, roundtrip failures, crash inputs |
| Integration | 15-20% | ~100ms each | Component interaction bugs, mock/reality divergence |
| E2E / acceptance | 5-10% | ~1s+ each | Full-chain failures, deployment config issues |

### Test Isolation Principles

- **No test depends on another test's state.** Each test sets up its own preconditions.
- **No test depends on execution order.** `pytest-randomly` catches order dependencies.
- **No test touches the real filesystem outside `tmp_path`.** Monkeypatch paths that default to production locations (like `/workspace`).
- **No test makes network calls.** Mock HTTP, subprocess, and socket calls.
- **Integration tests are marked** (`@pytest.mark.integration`) and excluded by default.

### Fixture Architecture

```python
# conftest.py — shared fixtures, not test logic

@pytest.fixture
def minimal_payload():
    """Smallest valid payload — tests shouldn't need more unless testing specific fields."""
    return {"task_id": "test-123", "prompt": "do something", "runtime": {"cli": "claude"}}

@pytest.fixture
def encode_payload():
    """Helper: dict → base64 string (how the dispatcher passes payloads)."""
    def _encode(d):
        return base64.b64encode(json.dumps(d).encode()).decode()
    return _encode

# Per-module conftest for module-specific fixtures
# tests/test_dispatcher/conftest.py
@pytest.fixture
def mock_backend():
    """Fake container backend that records calls without Docker."""
    ...
```

### Negative Tests Are as Important as Positive Tests

For every "this works" test, write at least one "this fails correctly" test:

```python
# Positive: valid payload accepted
def test_p1_valid_payload_loads():
    ...

# Negative: missing required field rejected
def test_p3_missing_prompt_raises():
    ...

# Negative: wrong type rejected
def test_p_prompt_wrong_type_raises():
    ...

# Negative: extra unknown fields are ignored (not rejected)
def test_p_unknown_fields_ignored():
    ...
```

## AI Agent Testing Patterns

### The Two-Phase Rule

**Never let the same agent write both tests and implementation in one pass.** An agent that writes tests and code together will unconsciously write tests that verify its own broken assumptions.

The workflow:
1. **Phase 1:** Agent reads spec → writes tests. Human reviews tests against spec.
2. **Phase 2:** Agent (or different agent) reads spec + tests → writes implementation until tests pass.

### Read-Only Test Gates

When an implementation agent is gated by tests, **place tests in a read-only reference directory** — not the working directory. Agents (especially smaller models) will modify test files to make tests pass rather than writing correct implementation. Prompt-level "DO NOT MODIFY" instructions are insufficient.

**Enforcement pattern:**
1. Clone tests to `/workspace/reference/` (root-owned, `chmod a-w`)
2. Agent implements in `/workspace/working/`
3. Run pytest against the immutable reference: `cd /workspace/working && PYTHONPATH=/workspace/working python -m pytest /workspace/reference/main/tests/ -v`

Use filesystem enforcement, not prompt instructions. In a 3-way model comparison: Sonnet respected "DO NOT MODIFY" instructions; MiniMax edited tests 7 times; Haiku rewrote the entire test file. The filesystem makes modification impossible regardless of model.

### Test Infrastructure Files Must Be Protected from Agent Modification

Read-only protection must extend beyond test files to include **test infrastructure**: `conftest.py`, `pyproject.toml`, `pytest.ini`, `setup.cfg`, `tox.ini`. An agent can satisfy tests by adding pytest hooks in a writable `conftest.py` — for example, a `pytest_collection_modifyitems` hook that skips failures, or an autouse fixture that monkeypatches the system under test. Security reviews of agent gate implementations have repeatedly found `conftest.py` as a CRITICAL bypass vector.

**Rule:** the read-only reference directory must contain all test-discovery and test-configuration files, not just `test_*.py`. At gate-enforcement time, run pytest with `--rootdir` / `--confcutdir` pointed at the read-only tree so writable copies of these files in the working directory cannot override the authoritative configuration.

### Stand Up Real Test Infrastructure Early

Deferring a real test database/service (Docker Compose, testcontainers, ephemeral Postgres, etc.) pushes integration tests into a "deselected" bucket that nobody runs. Set up the test backing service in the first phase that touches it, so integration tests execute from day one instead of accumulating as tech debt. The cost of standing up ephemeral infrastructure is almost always lower than the cost of letting integration coverage rot.

### Model Selection for Implementation Agents

**Sonnet is the minimum viable model for constrained implementation tasks** (spec + test gate). Smaller and cheaper models modify test files or ignore constraints:

| Model | Result | Notes |
|---|---|---|
| Sonnet | 113/113 tests passing, tests untouched | Viable for implementation |
| MiniMax | Modified tests 7 times | Invalid — use for review/test-writing only |
| Haiku | Rewrote test file entirely | Invalid — use for review/test-writing only |

Haiku and MiniMax are viable for test-writing, review, and spec work — tasks where the output is inspected by a human, not enforced by a gate.

### Hidden Test Splits

Hold back some tests that the implementing agent never sees. Use them as a final validation:

```python
# tests/test_payload.py — agent sees these during development
def test_p1_load_from_env_var(): ...
def test_p2_missing_payload_exits_1(): ...

# tests/test_payload_hidden.py — agent never sees these, run post-implementation
# (Marked with a custom marker, excluded from default run)
@pytest.mark.hidden
def test_p1_load_from_file_fallback(): ...
@pytest.mark.hidden
def test_p_concurrent_payload_loads(): ...
```

### Regression Tests from Real Bugs

Every bug found in production or during integration testing becomes a permanent test case:

```python
def test_regression_crlf_corruption():
    """Regression: smtp-oauth-relay converted \\r\\n to \\n, breaking quoted-printable.
    Fixed by as_bytes(policy=email_policy.SMTP). See memory/gotchas-email-relay.md."""
    ...
```

These are the highest-value tests because they catch proven failure modes.

## Test Quality Metrics

### What to Measure

| Metric | Target | Why |
|---|---|---|
| Spec coverage | 100% | Every numbered requirement has at least one test |
| Mutation score | >80% | Tests catch real faults, not just inflate coverage |
| Line coverage | >90% | Baseline hygiene (necessary but not sufficient) |
| Test speed | <10s total | Fast enough for pre-commit hooks |
| Assertion density | >1 per test | Tests that don't assert don't catch anything |

### What NOT to Measure

- **100% line coverage as a goal.** Chasing 100% leads to tests that exercise code paths without meaningful assertions.
- **Test count.** 50 well-targeted tests beat 200 shallow ones.
- **Test-to-code ratio.** The ratio depends on the module's complexity, not a universal number.

## CI Integration

### Pre-commit (Every Commit)
```bash
pytest tests/ -x -q --tb=short -m "not integration"
```

### PR Validation (Every Push)
```bash
# Unit + property-based tests
pytest tests/ -q --tb=short -m "not integration"
# Mutation testing on changed files only
mutmut run --paths-to-mutate="$(git diff --name-only main... | grep '.py$' | tr '\n' ',')"
```

### Nightly
```bash
# Full mutation testing
mutmut run
# Integration tests (requires Docker)
pytest tests/ -m integration
# Hidden test validation
pytest tests/ -m hidden
```

## Python Testing Gotchas

### `subprocess.run(check=True)` Is Invisible to Mocks

When you mock `subprocess.run`, the mock replaces the entire function — including the `check=True` logic that raises `CalledProcessError`. A mock returning `CompletedProcess(returncode=1)` won't trigger the exception even though the real code uses `check=True`. To test failure paths, use `side_effect=CalledProcessError(...)` explicitly.

### Use Routing Callables for Multi-Call Subprocess Mocks

When a function calls `subprocess.run` multiple times (e.g., git config, add, diff, commit, push), a fixed `side_effect` list is fragile and breaks when call order changes. Instead, use a routing callable that inspects the command: `mock_run.side_effect = lambda cmd, **kw: route_by_command(cmd)`. Clearer, more maintainable, and self-documenting.

### Pydantic v2 `@field_validator` Doesn't Fire for Default Values

`@field_validator('field_name')` never runs when the field takes its default value (e.g., `None`). Cross-field validation logic (e.g., "if type is X then field Y is required") silently passes when the dependent field is omitted. Use `@model_validator(mode='after')` for any validation that depends on multiple fields or needs to fire even when fields take defaults.

### Never `sys.exit()` at Module Level

`sys.exit()` in an `except ImportError` block at module level kills pytest collection entirely — all tests fail, not just the ones for that module. Use a flag pattern instead: `_HAS_DEPENDENCY = False` in the except block, then check `if not _HAS_DEPENDENCY: return 1` inside the function. This allows the module to be imported and mocked even when the optional dependency is missing.

### Use `pytest.importorskip` for Optional Dependency Tests

When test files import optional packages (e.g., `sqlalchemy`, `psycopg`) at module level, pytest collection fails for the entire test suite — not just the tests that need that package. Use `mod = pytest.importorskip("sqlalchemy")` and then attribute access (`mod.text`). Also guard transitive imports: `pytest.importorskip("myapp.db.postgres_store")` if the module itself imports the optional package at module level.

### Patch Individual Functions, Not Whole Modules

Patching an entire module (e.g., `patch("mod.kubernetes.config")`) replaces exception classes with MagicMock objects. `except SomeException` then catches `MagicMock` instead of the real exception, causing tests to pass the wrong code path. Patch individual functions (`load_incluster_config`, `load_kube_config`) and leave exception classes intact so `except` clauses work correctly.

### MagicMock Returns Truthy in Controller Loops — Always Set Boolean Defaults

`MagicMock()` return values are truthy by default. If a controller loop calls `mock.should_stop()` or `mock.reconcile_triggered()` and the mock has no explicit `return_value`, the loop never exits — or never sleeps — because every call returns a truthy MagicMock. In one real incident this pattern consumed 40GB RAM before OOM.

**Always set `mock.method.return_value = False`** for boolean-returning methods used in loop predicates. Combine with `pytest-timeout` (e.g., `timeout = 10` in `pyproject.toml`) on any project with async loops, signal handlers, or sleep patterns so runaway tests die fast instead of starving the test host.

### Async Migration Requires Full Test Conversion

When migrating a codebase from sync to async, helper functions get converted but test functions are often left as sync `def`. Every test that calls an async function needs `async def` + `@pytest.mark.asyncio` + `await`. After any async migration, run tests and grep for `RuntimeWarning: coroutine '...' was never awaited` to find remaining sync-to-async gaps.

## Anti-Patterns

### Tests that mirror implementation
**Symptom:** Test asserts that function calls happen in a specific order, using mock.assert_has_calls with exact sequences. Breaks on any refactor.
**Fix:** Test behaviour (inputs → outputs), not implementation details.

### Tests without assertions
**Symptom:** `test_it_runs()` calls the function and checks it doesn't crash. No assertion on the result.
**Fix:** Every test must assert something specific about the output, side effects, or raised exceptions.

### Overmocking
**Symptom:** Every dependency is mocked. Tests pass but integration fails because mocks don't match real behaviour.
**Fix:** Mock at the boundary (subprocess, HTTP, filesystem), not between your own modules. Use real objects for internal dependencies.

### Fragile tests
**Symptom:** Tests break when unrelated code changes. Usually caused by asserting on implementation details, shared mutable state, or execution order.
**Fix:** Test the public interface. Use fixtures for setup. Isolate each test completely.

### Testing private methods
**Symptom:** Tests import `_internal_helper` and test it directly. These break on any refactor.
**Fix:** Test through the public API. If a private method is complex enough to need its own tests, it should probably be a separate module with a public interface.