# Test-Driven Development for Spec-Based Projects

Best practices for writing comprehensive, regression-catching tests in projects that use structured specifications. Focuses on maximising test value (catching real bugs) rather than test volume (inflating coverage numbers). Extracted from industry research, academic papers (TDAD, Codified Context), and practitioner experience.

## Core Principle: Tests Are the Spec's Enforcement Layer

In a spec-driven project, the spec defines *what* and the tests *prove it*. A requirement without a test is an aspiration. A test without a requirement is undocumented behaviour. Keep them tightly coupled:

- Every numbered requirement (P-1, E-3) has at least one test
- Every test function name includes its requirement ID: `test_e3_preaction_failure_exits_1`
- Spec changes and test changes ship in the same commit

## Deriving Tests from Specs

### Requirements to Tests

Each spec requirement becomes one or more test functions. The mapping isn't always 1:1 — a requirement like "must respect timeout" needs tests for: default timeout, explicit timeout, timeout=0 (no limit), timeout exceeded.

```python
# From spec: R-4: Runners must respect runtime.timeout.
# Default 3600s. Value of 0 means no timeout.

def test_r4_default_timeout_is_3600():
    """R-4: When timeout not specified, default is 3600s."""

def test_r4_explicit_timeout_is_honoured():
    """R-4: When timeout=60, process killed after 60s."""

def test_r4_zero_timeout_means_no_limit():
    """R-4: When timeout=0, no timeout is applied."""

def test_r4_timeout_returns_exit_code_124():
    """R-4 + R-5: Timeout produces exit code 124."""
```

### Scenarios to Tests

GIVEN/WHEN/THEN scenarios translate directly to Arrange/Act/Assert:

```python
def test_scenario_preaction_failure_runs_on_error():
    """Given clone fails, on_error runs and exits 1. Runner never executes."""
    # GIVEN — arrange
    payload = make_payload(pre_actions=[{"action": "clone", "repo": "bad-url"}])
    mock_clone = Mock(side_effect=subprocess.CalledProcessError(128, "git"))

    # WHEN — act
    exit_code = run_entrypoint(payload, clone_handler=mock_clone)

    # THEN — assert
    assert exit_code == 1
    mock_runner.assert_not_called()
    mock_on_error.assert_called_once()
```

### Parameterised Tests from Spec Enumerations

When a spec lists multiple valid values, use `@pytest.mark.parametrize`:

```python
# From spec: task states are pending, assigned, running, succeeded, failed, timed_out, cancelled
@pytest.mark.parametrize("terminal_state", ["succeeded", "failed", "timed_out", "cancelled"])
def test_cp_terminal_state_cannot_be_overwritten(terminal_state):
    """CP: Terminal states reject further transitions with 409."""
```

## Systematic Edge Case Discovery

~80% of bugs cluster at boundaries. Use these techniques to find edge cases systematically rather than by intuition.

### Boundary Value Analysis

For every input parameter, test at the edges of its valid range:

| Input type | Test values |
|---|---|
| Integer (range 1-100) | 0, 1, 2, 99, 100, 101, -1, MAX_INT |
| String | `""`, `"a"`, max-length string, max+1, unicode (`"\u0000"`, emoji), whitespace-only |
| List/Array | `[]`, `[single]`, many items, duplicates, `None` |
| Dict/Map | `{}`, missing required keys, extra unknown keys, `None` values |
| Timeout (seconds) | 0, 1, -1, very large (999999), `None`/missing |
| Base64 | valid, invalid chars, empty, padding variants (`=`, `==`, none) |

### Equivalence Partitioning

Group inputs into classes where all members should behave identically. Test one from each class:

```python
# Payload validation: prompt field
# Class 1: valid string → accepted
# Class 2: empty string → rejected (spec says prompt is required)
# Class 3: missing key → rejected
# Class 4: wrong type (int, list, None) → rejected
# Class 5: very long string → accepted (no length limit in spec)

@pytest.mark.parametrize("prompt,should_pass", [
    ("Fix the bug", True),          # Class 1: valid
    ("", False),                     # Class 2: empty
    (None, False),                   # Class 3: missing/None
    (42, False),                     # Class 4: wrong type
    ("x" * 100_000, True),          # Class 5: long string
])
def test_p_prompt_validation(prompt, should_pass):
    ...
```

### State Transition Coverage

For state machines (task states, dispatcher states), test:

1. **Every valid transition:** `pending → assigned → running → succeeded`
2. **Every invalid transition:** `succeeded → running` (should be rejected)
3. **Initial state:** newly created tasks start in `pending`
4. **Terminal states:** `succeeded`, `failed`, `timed_out`, `cancelled` cannot transition further
5. **Re-entrant transitions:** same state → same state (should be idempotent or rejected, per spec)

```python
VALID_TRANSITIONS = [
    ("pending", "assigned"),
    ("assigned", "running"),
    ("running", "succeeded"),
    ("running", "failed"),
    ("running", "timed_out"),
    ("assigned", "cancelled"),
    ("running", "cancelled"),
]

INVALID_TRANSITIONS = [
    ("succeeded", "failed"),
    ("failed", "running"),
    ("cancelled", "pending"),
    ("timed_out", "running"),
]

@pytest.mark.parametrize("from_state,to_state", VALID_TRANSITIONS)
def test_valid_state_transition(from_state, to_state):
    ...

@pytest.mark.parametrize("from_state,to_state", INVALID_TRANSITIONS)
def test_invalid_state_transition_rejected(from_state, to_state):
    ...
```

### The Edge Case Checklist

Walk through this for every function under test:

1. **Empty/null inputs** — what happens when required fields are missing?
2. **Boundary values** — min, max, zero, negative, off-by-one
3. **Type mismatches** — string where int expected, list where dict expected
4. **Malformed input** — invalid JSON, bad base64, truncated data
5. **Concurrent operations** — two tasks claiming the same resource
6. **Ordering** — actions that depend on sequence (pre-action before runner)
7. **Idempotency** — calling the same operation twice (kill an already-killed container)
8. **Resource exhaustion** — at capacity, disk full, timeout expired
9. **Partial failure** — first action succeeds, second fails (cleanup?)

## Property-Based Testing with Hypothesis

Instead of specifying individual test cases, define *properties* that must hold for all inputs. Hypothesis generates hundreds of inputs including edge cases you'd never think of.

### When to Use Property-Based Testing

- **Serialisation roundtrips:** encode → decode returns original
- **Parsers:** should never crash on any input
- **Data transformations:** invariants that hold regardless of input
- **Validators:** valid inputs accepted, invalid inputs rejected (never crash)

### When NOT to Use It

- Tests where generating valid inputs is harder than the code itself
- Tests where the "property" just restates the implementation
- UI or integration tests

### Patterns

```python
from hypothesis import given, strategies as st, assume, settings
from hypothesis import example

# Roundtrip: base64 encode/decode preserves payload
@given(st.text())
def test_base64_roundtrip(payload_str):
    encoded = base64.b64encode(payload_str.encode()).decode()
    decoded = base64.b64decode(encoded).decode()
    assert decoded == payload_str

# Invariant: payload validation never crashes (may reject, never exception)
@given(st.dictionaries(st.text(), st.text() | st.integers() | st.none()))
def test_payload_validation_never_crashes(raw_payload):
    # Should return True/False or raise ValidationError — never unhandled exception
    try:
        validate_payload(raw_payload)
    except ValidationError:
        pass  # Expected for invalid input

# Pin known edge cases alongside random generation
@example("")           # empty string
@example("\x00")       # null byte
@example("a" * 10**6)  # very long
@given(st.text())
def test_prompt_handling(prompt):
    ...

# Composite strategies for domain objects
@st.composite
def valid_payloads(draw):
    return {
        "task_id": draw(st.uuids()).hex,
        "prompt": draw(st.text(min_size=1)),
        "runtime": {"cli": draw(st.sampled_from(["claude", "codex"]))},
    }

@given(valid_payloads())
def test_valid_payload_always_accepted(payload):
    assert validate_payload(payload) is True
```

### Stateful Testing for State Machines

Hypothesis can generate sequences of operations and check invariants after each step:

```python
from hypothesis.stateful import RuleBasedStateMachine, rule, precondition

class TaskStateMachine(RuleBasedStateMachine):
    def __init__(self):
        super().__init__()
        self.task = Task(state="pending")

    @rule()
    @precondition(lambda self: self.task.state == "pending")
    def assign(self):
        self.task.transition("assigned")
        assert self.task.state == "assigned"

    @rule()
    @precondition(lambda self: self.task.state == "running")
    def complete(self):
        self.task.transition("succeeded")
        assert self.task.state == "succeeded"

    # Invariant: terminal states never change
    @invariant()
    def terminal_states_are_final(self):
        if self.task.state in ("succeeded", "failed", "cancelled"):
            with pytest.raises(InvalidTransition):
                self.task.transition("running")

TestTaskStates = TaskStateMachine.TestCase
```

## Mutation Testing

Mutation testing answers: "If someone introduced a bug, would our tests catch it?"

Tools make small code changes (replacing `>` with `>=`, `True` with `False`, deleting statements) and check if tests still pass. Surviving mutants = test gaps.

### Setup with mutmut

```toml
# pyproject.toml
[tool.mutmut]
paths_to_mutate = "entrypoint/"
tests_dir = "tests/"
runner = "python -m pytest tests/ -x -q"
```

```bash
# Run mutation testing
mutmut run

# See surviving mutants
mutmut results

# Inspect a specific mutant
mutmut show 42
```

### Practical Guidance

- **Target: mutation score above 80%.** Scores above 90% have diminishing returns (equivalent mutants).
- **Focus on business logic** — validators, state machines, parsers. Skip glue code.
- **Use mutation testing to audit AI-generated tests.** This is the most powerful combination: AI writes tests from spec, mutation testing verifies those tests catch real faults.
- **Run on changed files only in CI** (full suite is slow). Full run nightly or pre-release.

## Test Architecture

### The Testing Pyramid for Spec-Driven Projects

| Layer | Proportion | Speed | What it catches |
|---|---|---|---|
| Unit tests | 60-70% | <1ms each | Logic errors, boundary violations, state machine bugs |
| Property-based | 10-15% | ~10ms each | Edge cases humans miss, roundtrip failures, crash inputs |
| Integration | 15-20% | ~100ms each | Component interaction bugs, mock/reality divergence |
| E2E / acceptance | 5-10% | ~1s+ each | Full-chain failures, deployment config issues |

### Test Isolation Principles

- **No test depends on another test's state.** Each test sets up its own preconditions.
- **No test depends on execution order.** `pytest-randomly` catches order dependencies.
- **No test touches the real filesystem outside `tmp_path`.** Monkeypatch paths that default to production locations (like `/workspace`).
- **No test makes network calls.** Mock HTTP, subprocess, and socket calls.
- **Integration tests are marked** (`@pytest.mark.integration`) and excluded by default.

### Fixture Architecture

```python
# conftest.py — shared fixtures, not test logic

@pytest.fixture
def minimal_payload():
    """Smallest valid payload — tests shouldn't need more unless testing specific fields."""
    return {"task_id": "test-123", "prompt": "do something", "runtime": {"cli": "claude"}}

@pytest.fixture
def encode_payload():
    """Helper: dict → base64 string (how the dispatcher passes payloads)."""
    def _encode(d):
        return base64.b64encode(json.dumps(d).encode()).decode()
    return _encode

# Per-module conftest for module-specific fixtures
# tests/test_dispatcher/conftest.py
@pytest.fixture
def mock_backend():
    """Fake container backend that records calls without Docker."""
    ...
```

### Negative Tests Are as Important as Positive Tests

For every "this works" test, write at least one "this fails correctly" test:

```python
# Positive: valid payload accepted
def test_p1_valid_payload_loads():
    ...

# Negative: missing required field rejected
def test_p3_missing_prompt_raises():
    ...

# Negative: wrong type rejected
def test_p_prompt_wrong_type_raises():
    ...

# Negative: extra unknown fields are ignored (not rejected)
def test_p_unknown_fields_ignored():
    ...
```

## AI Agent Testing Patterns

### The Two-Phase Rule

**Never let the same agent write both tests and implementation in one pass.** An agent that writes tests and code together will unconsciously write tests that verify its own broken assumptions.

The workflow:
1. **Phase 1:** Agent reads spec → writes tests. Human reviews tests against spec.
2. **Phase 2:** Agent (or different agent) reads spec + tests → writes implementation until tests pass.

### Hidden Test Splits

Hold back some tests that the implementing agent never sees. Use them as a final validation:

```python
# tests/test_payload.py — agent sees these during development
def test_p1_load_from_env_var(): ...
def test_p2_missing_payload_exits_1(): ...

# tests/test_payload_hidden.py — agent never sees these, run post-implementation
# (Marked with a custom marker, excluded from default run)
@pytest.mark.hidden
def test_p1_load_from_file_fallback(): ...
@pytest.mark.hidden
def test_p_concurrent_payload_loads(): ...
```

### Regression Tests from Real Bugs

Every bug found in production or during integration testing becomes a permanent test case:

```python
def test_regression_crlf_corruption():
    """Regression: smtp-oauth-relay converted \\r\\n to \\n, breaking quoted-printable.
    Fixed by as_bytes(policy=email_policy.SMTP). See memory/gotchas-email-relay.md."""
    ...
```

These are the highest-value tests because they catch proven failure modes.

## Test Quality Metrics

### What to Measure

| Metric | Target | Why |
|---|---|---|
| Spec coverage | 100% | Every numbered requirement has at least one test |
| Mutation score | >80% | Tests catch real faults, not just inflate coverage |
| Line coverage | >90% | Baseline hygiene (necessary but not sufficient) |
| Test speed | <10s total | Fast enough for pre-commit hooks |
| Assertion density | >1 per test | Tests that don't assert don't catch anything |

### What NOT to Measure

- **100% line coverage as a goal.** Chasing 100% leads to tests that exercise code paths without meaningful assertions.
- **Test count.** 50 well-targeted tests beat 200 shallow ones.
- **Test-to-code ratio.** The ratio depends on the module's complexity, not a universal number.

## CI Integration

### Pre-commit (Every Commit)
```bash
pytest tests/ -x -q --tb=short -m "not integration"
```

### PR Validation (Every Push)
```bash
# Unit + property-based tests
pytest tests/ -q --tb=short -m "not integration"
# Mutation testing on changed files only
mutmut run --paths-to-mutate="$(git diff --name-only main... | grep '.py$' | tr '\n' ',')"
```

### Nightly
```bash
# Full mutation testing
mutmut run
# Integration tests (requires Docker)
pytest tests/ -m integration
# Hidden test validation
pytest tests/ -m hidden
```

## Anti-Patterns

### Tests that mirror implementation
**Symptom:** Test asserts that function calls happen in a specific order, using mock.assert_has_calls with exact sequences. Breaks on any refactor.
**Fix:** Test behaviour (inputs → outputs), not implementation details.

### Tests without assertions
**Symptom:** `test_it_runs()` calls the function and checks it doesn't crash. No assertion on the result.
**Fix:** Every test must assert something specific about the output, side effects, or raised exceptions.

### Overmocking
**Symptom:** Every dependency is mocked. Tests pass but integration fails because mocks don't match real behaviour.
**Fix:** Mock at the boundary (subprocess, HTTP, filesystem), not between your own modules. Use real objects for internal dependencies.

### Fragile tests
**Symptom:** Tests break when unrelated code changes. Usually caused by asserting on implementation details, shared mutable state, or execution order.
**Fix:** Test the public interface. Use fixtures for setup. Isolate each test completely.

### Testing private methods
**Symptom:** Tests import `_internal_helper` and test it directly. These break on any refactor.
**Fix:** Test through the public API. If a private method is complex enough to need its own tests, it should probably be a separate module with a public interface.