best-practices/test-driven-development.md

Test-Driven Development for Spec-Based Projects

Best practices for writing comprehensive, regression-catching tests in projects that use structured specifications. Focuses on maximising test value (catching real bugs) rather than test volume (inflating coverage numbers). Extracted from industry research, academic papers (TDAD, Codified Context), and practitioner experience.

Core Principle: Tests Are the Spec's Enforcement Layer

In a spec-driven project, the spec defines what and the tests prove it. A requirement without a test is an aspiration. A test without a requirement is undocumented behaviour. Keep them tightly coupled:

• Every numbered requirement (P-1, E-3) has at least one test
• Every test function name includes its requirement ID: test_e3_preaction_failure_exits_1
• Spec changes and test changes ship in the same commit

Deriving Tests from Specs

Requirements to Tests

Each spec requirement becomes one or more test functions. The mapping isn't always 1:1 — a requirement like "must respect timeout" needs tests for: default timeout, explicit timeout, timeout=0 (no limit), timeout exceeded.

# From spec: R-4: Runners must respect runtime.timeout.
# Default 3600s. Value of 0 means no timeout.

def test_r4_default_timeout_is_3600():
    """R-4: When timeout not specified, default is 3600s."""

def test_r4_explicit_timeout_is_honoured():
    """R-4: When timeout=60, process killed after 60s."""

def test_r4_zero_timeout_means_no_limit():
    """R-4: When timeout=0, no timeout is applied."""

def test_r4_timeout_returns_exit_code_124():
    """R-4 + R-5: Timeout produces exit code 124."""

Scenarios to Tests

GIVEN/WHEN/THEN scenarios translate directly to Arrange/Act/Assert:

def test_scenario_preaction_failure_runs_on_error():
    """Given clone fails, on_error runs and exits 1. Runner never executes."""
    # GIVEN — arrange
    payload = make_payload(pre_actions=[{"action": "clone", "repo": "bad-url"}])
    mock_clone = Mock(side_effect=subprocess.CalledProcessError(128, "git"))

    # WHEN — act
    exit_code = run_entrypoint(payload, clone_handler=mock_clone)

    # THEN — assert
    assert exit_code == 1
    mock_runner.assert_not_called()
    mock_on_error.assert_called_once()

Parameterised Tests from Spec Enumerations

When a spec lists multiple valid values, use @pytest.mark.parametrize:

# From spec: task states are pending, assigned, running, succeeded, failed, timed_out, cancelled
@pytest.mark.parametrize("terminal_state", ["succeeded", "failed", "timed_out", "cancelled"])
def test_cp_terminal_state_cannot_be_overwritten(terminal_state):
    """CP: Terminal states reject further transitions with 409."""

Systematic Edge Case Discovery

~80% of bugs cluster at boundaries. Use these techniques to find edge cases systematically rather than by intuition.

Boundary Value Analysis

For every input parameter, test at the edges of its valid range:

Input type	Test values
Integer (range 1-100)	0, 1, 2, 99, 100, 101, -1, MAX_INT
String	"", "a", max-length string, max+1, unicode ("\u0000", emoji), whitespace-only
List/Array	[], [single], many items, duplicates, None
Dict/Map	{}, missing required keys, extra unknown keys, None values
Timeout (seconds)	0, 1, -1, very large (999999), None/missing
Base64	valid, invalid chars, empty, padding variants (=, ==, none)

Equivalence Partitioning

Group inputs into classes where all members should behave identically. Test one from each class:

# Payload validation: prompt field
# Class 1: valid string → accepted
# Class 2: empty string → rejected (spec says prompt is required)
# Class 3: missing key → rejected
# Class 4: wrong type (int, list, None) → rejected
# Class 5: very long string → accepted (no length limit in spec)

@pytest.mark.parametrize("prompt,should_pass", [
    ("Fix the bug", True),          # Class 1: valid
    ("", False),                     # Class 2: empty
    (None, False),                   # Class 3: missing/None
    (42, False),                     # Class 4: wrong type
    ("x" * 100_000, True),          # Class 5: long string
])
def test_p_prompt_validation(prompt, should_pass):
    ...

State Transition Coverage

For state machines (task states, dispatcher states), test:

1. Every valid transition: pending → assigned → running → succeeded
2. Every invalid transition: succeeded → running (should be rejected)
3. Initial state: newly created tasks start in pending
4. Terminal states: succeeded, failed, timed_out, cancelled cannot transition further
5. Re-entrant transitions: same state → same state (should be idempotent or rejected, per spec)

VALID_TRANSITIONS = [
    ("pending", "assigned"),
    ("assigned", "running"),
    ("running", "succeeded"),
    ("running", "failed"),
    ("running", "timed_out"),
    ("assigned", "cancelled"),
    ("running", "cancelled"),
]

INVALID_TRANSITIONS = [
    ("succeeded", "failed"),
    ("failed", "running"),
    ("cancelled", "pending"),
    ("timed_out", "running"),
]

@pytest.mark.parametrize("from_state,to_state", VALID_TRANSITIONS)
def test_valid_state_transition(from_state, to_state):
    ...

@pytest.mark.parametrize("from_state,to_state", INVALID_TRANSITIONS)
def test_invalid_state_transition_rejected(from_state, to_state):
    ...

The Edge Case Checklist

Walk through this for every function under test:

1. Empty/null inputs — what happens when required fields are missing?
2. Boundary values — min, max, zero, negative, off-by-one
3. Type mismatches — string where int expected, list where dict expected
4. Malformed input — invalid JSON, bad base64, truncated data
5. Concurrent operations — two tasks claiming the same resource
6. Ordering — actions that depend on sequence (pre-action before runner)
7. Idempotency — calling the same operation twice (kill an already-killed container)
8. Resource exhaustion — at capacity, disk full, timeout expired
9. Partial failure — first action succeeds, second fails (cleanup?)

Property-Based Testing with Hypothesis

Instead of specifying individual test cases, define properties that must hold for all inputs. Hypothesis generates hundreds of inputs including edge cases you'd never think of.

When to Use Property-Based Testing

• Serialisation roundtrips: encode → decode returns original
• Parsers: should never crash on any input
• Data transformations: invariants that hold regardless of input
• Validators: valid inputs accepted, invalid inputs rejected (never crash)

When NOT to Use It

• Tests where generating valid inputs is harder than the code itself
• Tests where the "property" just restates the implementation
• UI or integration tests

Patterns

from hypothesis import given, strategies as st, assume, settings
from hypothesis import example

# Roundtrip: base64 encode/decode preserves payload
@given(st.text())
def test_base64_roundtrip(payload_str):
    encoded = base64.b64encode(payload_str.encode()).decode()
    decoded = base64.b64decode(encoded).decode()
    assert decoded == payload_str

# Invariant: payload validation never crashes (may reject, never exception)
@given(st.dictionaries(st.text(), st.text() | st.integers() | st.none()))
def test_payload_validation_never_crashes(raw_payload):
    # Should return True/False or raise ValidationError — never unhandled exception
    try:
        validate_payload(raw_payload)
    except ValidationError:
        pass  # Expected for invalid input

# Pin known edge cases alongside random generation
@example("")           # empty string
@example("\x00")       # null byte
@example("a" * 10**6)  # very long
@given(st.text())
def test_prompt_handling(prompt):
    ...

# Composite strategies for domain objects
@st.composite
def valid_payloads(draw):
    return {
        "task_id": draw(st.uuids()).hex,
        "prompt": draw(st.text(min_size=1)),
        "runtime": {"cli": draw(st.sampled_from(["claude", "codex"]))},
    }

@given(valid_payloads())
def test_valid_payload_always_accepted(payload):
    assert validate_payload(payload) is True

Stateful Testing for State Machines

Hypothesis can generate sequences of operations and check invariants after each step:

from hypothesis.stateful import RuleBasedStateMachine, rule, precondition

class TaskStateMachine(RuleBasedStateMachine):
    def __init__(self):
        super().__init__()
        self.task = Task(state="pending")

    @rule()
    @precondition(lambda self: self.task.state == "pending")
    def assign(self):
        self.task.transition("assigned")
        assert self.task.state == "assigned"

    @rule()
    @precondition(lambda self: self.task.state == "running")
    def complete(self):
        self.task.transition("succeeded")
        assert self.task.state == "succeeded"

    # Invariant: terminal states never change
    @invariant()
    def terminal_states_are_final(self):
        if self.task.state in ("succeeded", "failed", "cancelled"):
            with pytest.raises(InvalidTransition):
                self.task.transition("running")

TestTaskStates = TaskStateMachine.TestCase

Mutation Testing

Mutation testing answers: "If someone introduced a bug, would our tests catch it?"

Tools make small code changes (replacing > with >=, True with False, deleting statements) and check if tests still pass. Surviving mutants = test gaps.

Setup with mutmut

# pyproject.toml
[tool.mutmut]
paths_to_mutate = "entrypoint/"
tests_dir = "tests/"
runner = "python -m pytest tests/ -x -q"

# Run mutation testing
mutmut run

# See surviving mutants
mutmut results

# Inspect a specific mutant
mutmut show 42

Practical Guidance

• Target: mutation score above 80%. Scores above 90% have diminishing returns (equivalent mutants).
• Focus on business logic — validators, state machines, parsers. Skip glue code.
• Use mutation testing to audit AI-generated tests. This is the most powerful combination: AI writes tests from spec, mutation testing verifies those tests catch real faults.
• Run on changed files only in CI (full suite is slow). Full run nightly or pre-release.

Test Architecture

The Testing Pyramid for Spec-Driven Projects

Layer	Proportion	Speed	What it catches
Unit tests	60-70%	<1ms each	Logic errors, boundary violations, state machine bugs
Property-based	10-15%	~10ms each	Edge cases humans miss, roundtrip failures, crash inputs
Integration	15-20%	~100ms each	Component interaction bugs, mock/reality divergence
E2E / acceptance	5-10%	~1s+ each	Full-chain failures, deployment config issues

Test Isolation Principles

• No test depends on another test's state. Each test sets up its own preconditions.
• No test depends on execution order. pytest-randomly catches order dependencies.
• No test touches the real filesystem outside tmp_path. Monkeypatch paths that default to production locations (like /workspace).
• No test makes network calls. Mock HTTP, subprocess, and socket calls.
• Integration tests are marked (@pytest.mark.integration) and excluded by default.

Fixture Architecture

# conftest.py — shared fixtures, not test logic

@pytest.fixture
def minimal_payload():
    """Smallest valid payload — tests shouldn't need more unless testing specific fields."""
    return {"task_id": "test-123", "prompt": "do something", "runtime": {"cli": "claude"}}

@pytest.fixture
def encode_payload():
    """Helper: dict → base64 string (how the dispatcher passes payloads)."""
    def _encode(d):
        return base64.b64encode(json.dumps(d).encode()).decode()
    return _encode

# Per-module conftest for module-specific fixtures
# tests/test_dispatcher/conftest.py
@pytest.fixture
def mock_backend():
    """Fake container backend that records calls without Docker."""
    ...

Negative Tests Are as Important as Positive Tests

For every "this works" test, write at least one "this fails correctly" test:

# Positive: valid payload accepted
def test_p1_valid_payload_loads():
    ...

# Negative: missing required field rejected
def test_p3_missing_prompt_raises():
    ...

# Negative: wrong type rejected
def test_p_prompt_wrong_type_raises():
    ...

# Negative: extra unknown fields are ignored (not rejected)
def test_p_unknown_fields_ignored():
    ...

AI Agent Testing Patterns

The Two-Phase Rule

Never let the same agent write both tests and implementation in one pass. An agent that writes tests and code together will unconsciously write tests that verify its own broken assumptions.

The workflow:

1. Phase 1: Agent reads spec → writes tests. Human reviews tests against spec.
2. Phase 2: Agent (or different agent) reads spec + tests → writes implementation until tests pass.

Hidden Test Splits

Hold back some tests that the implementing agent never sees. Use them as a final validation:

# tests/test_payload.py — agent sees these during development
def test_p1_load_from_env_var(): ...
def test_p2_missing_payload_exits_1(): ...

# tests/test_payload_hidden.py — agent never sees these, run post-implementation
# (Marked with a custom marker, excluded from default run)
@pytest.mark.hidden
def test_p1_load_from_file_fallback(): ...
@pytest.mark.hidden
def test_p_concurrent_payload_loads(): ...

Regression Tests from Real Bugs

Every bug found in production or during integration testing becomes a permanent test case:

def test_regression_crlf_corruption():
    """Regression: smtp-oauth-relay converted \\r\\n to \\n, breaking quoted-printable.
    Fixed by as_bytes(policy=email_policy.SMTP). See memory/gotchas-email-relay.md."""
    ...

These are the highest-value tests because they catch proven failure modes.

Test Quality Metrics

What to Measure

Metric	Target	Why
Spec coverage	100%	Every numbered requirement has at least one test
Mutation score	>80%	Tests catch real faults, not just inflate coverage
Line coverage	>90%	Baseline hygiene (necessary but not sufficient)
Test speed	<10s total	Fast enough for pre-commit hooks
Assertion density	>1 per test	Tests that don't assert don't catch anything

What NOT to Measure

• 100% line coverage as a goal. Chasing 100% leads to tests that exercise code paths without meaningful assertions.
• Test count. 50 well-targeted tests beat 200 shallow ones.
• Test-to-code ratio. The ratio depends on the module's complexity, not a universal number.

CI Integration

Pre-commit (Every Commit)

pytest tests/ -x -q --tb=short -m "not integration"

PR Validation (Every Push)

# Unit + property-based tests
pytest tests/ -q --tb=short -m "not integration"
# Mutation testing on changed files only
mutmut run --paths-to-mutate="$(git diff --name-only main... | grep '.py$' | tr '\n' ',')"

Nightly

# Full mutation testing
mutmut run
# Integration tests (requires Docker)
pytest tests/ -m integration
# Hidden test validation
pytest tests/ -m hidden

Python Testing Gotchas

subprocess.run(check=True) Is Invisible to Mocks

When you mock subprocess.run, the mock replaces the entire function — including the check=True logic that raises CalledProcessError. A mock returning CompletedProcess(returncode=1) won't trigger the exception even though the real code uses check=True. To test failure paths, use side_effect=CalledProcessError(...) explicitly.

Use Routing Callables for Multi-Call Subprocess Mocks

When a function calls subprocess.run multiple times (e.g., git config, add, diff, commit, push), a fixed side_effect list is fragile and breaks when call order changes. Instead, use a routing callable that inspects the command: mock_run.side_effect = lambda cmd, **kw: route_by_command(cmd). Clearer, more maintainable, and self-documenting.

Pydantic v2 @field_validator Doesn't Fire for Default Values

@field_validator('field_name') never runs when the field takes its default value (e.g., None). Cross-field validation logic (e.g., "if type is X then field Y is required") silently passes when the dependent field is omitted. Use @model_validator(mode='after') for any validation that depends on multiple fields or needs to fire even when fields take defaults.

Never sys.exit() at Module Level

sys.exit() in an except ImportError block at module level kills pytest collection entirely — all tests fail, not just the ones for that module. Use a flag pattern instead: _HAS_DEPENDENCY = False in the except block, then check if not _HAS_DEPENDENCY: return 1 inside the function. This allows the module to be imported and mocked even when the optional dependency is missing.

Use pytest.importorskip for Optional Dependency Tests

When test files import optional packages (e.g., sqlalchemy, psycopg) at module level, pytest collection fails for the entire test suite — not just the tests that need that package. Use mod = pytest.importorskip("sqlalchemy") and then attribute access (mod.text). Also guard transitive imports: pytest.importorskip("myapp.db.postgres_store") if the module itself imports the optional package at module level.

Patch Individual Functions, Not Whole Modules

Patching an entire module (e.g., patch("mod.kubernetes.config")) replaces exception classes with MagicMock objects. except SomeException then catches MagicMock instead of the real exception, causing tests to pass the wrong code path. Patch individual functions (load_incluster_config, load_kube_config) and leave exception classes intact so except clauses work correctly.

Async Migration Requires Full Test Conversion

When migrating a codebase from sync to async, helper functions get converted but test functions are often left as sync def. Every test that calls an async function needs async def + @pytest.mark.asyncio + await. After any async migration, run tests and grep for RuntimeWarning: coroutine '...' was never awaited to find remaining sync-to-async gaps.

Anti-Patterns

Tests that mirror implementation

Symptom: Test asserts that function calls happen in a specific order, using mock.assert_has_calls with exact sequences. Breaks on any refactor. Fix: Test behaviour (inputs → outputs), not implementation details.

Tests without assertions

Symptom: test_it_runs() calls the function and checks it doesn't crash. No assertion on the result. Fix: Every test must assert something specific about the output, side effects, or raised exceptions.

Overmocking

Symptom: Every dependency is mocked. Tests pass but integration fails because mocks don't match real behaviour. Fix: Mock at the boundary (subprocess, HTTP, filesystem), not between your own modules. Use real objects for internal dependencies.

Fragile tests

Symptom: Tests break when unrelated code changes. Usually caused by asserting on implementation details, shared mutable state, or execution order. Fix: Test the public interface. Use fixtures for setup. Isolate each test completely.

Testing private methods

Symptom: Tests import _internal_helper and test it directly. These break on any refactor. Fix: Test through the public API. If a private method is complex enough to need its own tests, it should probably be a separate module with a public interface.