best-practices/spec-driven-development.md

Spec-Driven Development with AI Agents

Best practices for using structured specifications to coordinate AI agent implementation work. Extracted from real project experience (agent-runtimes) and industry research (OpenSpec, Codified Context paper, Addy Osmani's workflow guides).

Why Specs Matter for AI Agents

AI agents trust documentation absolutely. A well-written spec gives an agent everything it needs to implement a subsystem without reading the entire codebase. A stale or vague spec causes silent failures where agents generate code that is structurally valid but architecturally wrong.

Specs serve three functions that CLAUDE.md alone cannot:

1. Compressed context — an agent reads one spec, not 300 lines of mixed concerns
2. Testable contracts — numbered requirements and scenarios translate directly to pytest
3. Handoff boundaries — an agent working on the dispatcher doesn't need to understand the entrypoint internals, just the interface between them

Spec Structure

Each spec follows a consistent template. Sections are ordered so an agent can read top-down and build understanding progressively.

Required Sections

1. Overview — What this subsystem does. 2-3 sentences. An agent should know if this spec is relevant after reading this.
2. Responsibilities — What this subsystem owns and what it delegates. Prevents scope creep during implementation.
3. Dependencies — Which other specs to read first. Keeps the reading list minimal.
4. Data Model — Types, schemas, state machines, interfaces. The concrete contract.
5. Requirements — Numbered functional requirements (e.g., E-1, E-2). Each must be independently testable.
6. Scenarios — Concrete given/when/then examples. These become test functions.

Optional Sections

7. Interface — API surface, function signatures, HTTP endpoints. Include when the subsystem has an external-facing API.
8. Extension Points — How to add new capabilities without modifying existing code. Step-by-step instructions.
9. Error Handling — Failure modes and expected behaviour. Prevents agents from inventing their own error strategies.

Writing Guidelines

• Be specific, not comprehensive. A spec that says "handle errors appropriately" is useless. A spec that says "return exit code 124 on timeout" is testable.
• Include the why. Design intent and constraints prevent agents from making structurally valid but architecturally wrong changes. Requirements without rationale are followed mechanically — agents can't judge edge cases or make trade-offs. Every constraint needs a "Why:" line. Example: "Secrets never in payload" needs "because payloads may be logged and stored in task history".
• Use concrete examples. Every data model section should include a realistic JSON/code example, not just a schema.
• Cross-reference, don't duplicate. If two specs share a concept (e.g., the payload schema), one spec owns it and the other links to it.
• Keep each spec self-contained. An agent should be able to implement a subsystem by reading the target spec plus its listed dependencies. If it needs to read CLAUDE.md, the spec is incomplete.

Requirement Numbering

Each spec uses a short prefix derived from its name, followed by a sequential number:

Spec	Prefix	Example
payload.md	P	P-1, P-2
entrypoint.md	E	E-1, E-2
actions.md	A	A-1, A-2
runners.md	R	R-1, R-2
dispatcher.md	D	D-1, D-2
control-plane.md	CP	CP-1, CP-2

Requirements must be:

• Independently testable — each maps to one or more test functions
• Unambiguous — an agent can determine pass/fail without human judgement
• Stable — changing a requirement number invalidates tests, so avoid renumbering

Scenarios as Test Blueprints

Every scenario in a spec should be directly translatable to a test function. Use this format:

### Scenario: Pre-action failure
**Given:** Payload with clone pre-action (invalid repo URL)
**When:** Clone fails (git returns non-zero)
**Then:** on_error actions run, container exits 1. Runner never executes.

This becomes:

def test_scenario_preaction_failure_runs_on_error_and_exits_1(...):
    """Given clone fails, on_error runs and exits 1. Runner never executes."""

Guidelines:

• Each scenario tests one behaviour, not a combination
• Include both happy paths and error paths
• Name the scenario descriptively — it becomes the test function's docstring
• Include enough setup detail that an agent can write the test without guessing

Plan Mode Produces Architecture, Not Contracts

Plans and specs answer different questions:

• Plans answer "what we'll build" — milestones, tech choices, deployment shape, high-level architecture
• Specs answer "how it must behave" — numbered requirements, scenarios, data models, interfaces

Skipping specs and jumping straight from plan to code forces retroactive spec writing once behaviour questions surface — and then tests written against the implementation have to be rewritten against the spec. This has been measured at ~30% of a session in one case.

Rule: for any multi-milestone coding project, enforce Plan → Spec → Test → Code. The plan is not a substitute for the spec.

The Spec → Test → Code Workflow

This is the core development loop. Tests are written from the spec before code exists.

1. Write or Update the Spec

Define requirements and scenarios. Get them reviewed. The spec is the source of truth for what the system should do.

2. Write Tests from the Spec

Translate requirements and scenarios into pytest functions. Tests should:

• Map to requirement IDs in their names: test_e4_preaction_failure_skips_remaining
• Use the scenario's given/when/then as the test body structure
• Mock external dependencies (subprocess, HTTP, filesystem)
• Run fast (no Docker, no network, no real APIs)

3. Run the Tests — They Should All Fail

This confirms the tests are actually testing something. If a test passes before implementation, it's either testing the wrong thing or the feature already exists.

4. Implement Until Tests Pass

Write the minimum code to make tests pass. The spec defines what, the tests verify it, the code implements it.

5. Update Spec if Implementation Reveals Issues

Sometimes implementation reveals that a requirement is unworkable or incomplete. Update the spec, update the test, then update the code. The spec stays authoritative.

Spec Maintenance

Preventing Drift

Specs drift from code when they're treated as planning documents that are "done" after implementation. They must be treated as living contracts.

Rules:

• Spec changes require test changes. If a requirement changes, its test must change in the same commit.
• Code changes that affect interfaces require spec changes. If a function signature, API endpoint, or data schema changes, the relevant spec must be updated in the same commit.
• New features require spec-first. Add the requirement and scenario to the spec, write the test, then implement.

CI Enforcement

Enforce spec hygiene with automated checks:

1. Pre-commit hook — run pytest, block commit on failure (already implemented)
2. Spec coverage check — a script that verifies every numbered requirement has at least one test function referencing it
3. Orphan test detection — tests referencing requirement IDs that no longer exist in specs

Before completing any milestone, manually walk through every requirement ID (e.g., CP-1..CP-20, TH-1..TH-13) and verify a corresponding test exists. Automated spec coverage checks catch this in CI, but a manual audit before milestone completion catches gaps that the automation might miss (stubs, placeholder tests, tests that reference the ID but don't actually test the requirement).

Review Checklist

When reviewing a PR that touches a spec subsystem:

• Spec updated if interface or behaviour changed
• Test added/updated for new/changed requirements
• Cross-references still valid
• No requirements removed without deprecation note

Context Architecture for Agents

Based on the Codified Context paper (108k-line system, 283 sessions), structure project knowledge in three tiers:

Tier 1: Hot Context (Always Loaded)

CLAUDE.md — conventions, env vars, repo structure, scripts. Loaded every session. Keep under ~300 lines by linking to details elsewhere.

Tier 2: Spec Context (Per-Task)

spec/ files — loaded based on what the agent is working on. An agent implementing a new action reads spec/actions.md + spec/payload.md. An agent working on the dispatcher reads spec/dispatcher.md + spec/container-backends.md.

The spec index (SPEC.md) has a "read this when..." column to guide selection.

Tier 3: Cold Context (On-Demand)

memory/ files — gotchas, reflections, decisions. Loaded only when relevant. An agent hitting a weird Cilium issue checks memory/gotchas-cilium.md.

Routing Context to Agents

When launching an agent to work on a subsystem:

1. Point it at the relevant spec(s) via its prompt
2. Include CLAUDE.md for conventions
3. Let it pull from memory/ on-demand if it hits issues

Don't load everything — agents perform better with focused context than with a 50-page dump.

Testing Depth

The spec→test→code workflow defines when to write tests. For how to write comprehensive tests — edge case discovery, property-based testing, mutation testing, AI agent testing patterns — see Test-Driven Development.

Post-Write Spec Audit

After writing specs, audit them against best practices before implementation. Common gap categories:

1. Missing rationale — Constraints without "Why:" lines. Agents follow them mechanically but can't judge edge cases.
2. Missing error/failure scenarios — Happy paths are covered but failure modes aren't specified.
3. Cross-spec interface misalignment — Two specs describe the same interface differently.
4. Vague requirements — "Handle errors appropriately" instead of specific error codes and behaviours.
5. Missing specs for discovered subsystems — Implementation reveals components that weren't planned for.

Write-then-audit is more productive than trying to get specs perfect on the first pass. The audit step catches systematic gaps across all specs at once.

Planning Session Limits

Architecture decisions, infrastructure research, and spec refinement each get one planning session. After three sessions of planning, start implementation. Specs are hypotheses that need code to validate them — extended planning without implementation produces diminishing returns and theoretical designs that don't survive contact with reality.

Categorize Findings Before Acting

When a spec review or audit produces many findings, categorize them by priority (high/medium/low) before making changes. Present the categorized list for alignment before editing. Starting edits without prioritization leads to scope creep — low-priority cosmetic fixes consume time that should go to high-priority structural gaps.

Multi-Agent Orchestration Practices

Commit WIP Before Decomposing Tasks

Untracked and uncommitted files are NOT available in git worktrees. If agents work in worktrees (or container-mounted worktrees), they won't see specs, plans, or dependency outputs that haven't been committed. Commit to a staging branch before decomposition — this eliminates the dominant overhead of manually copying files into each worktree.

Agents Must Self-Verify with Tests

Add "Run tests and fix any failures" to every implementation agent prompt. Agents that write code without running tests produce bugs that only surface during assembly. Self-verification catches issues while the agent still has full context of what it wrote.

State Import and Style Conventions Explicitly

Agents default to standard language conventions (e.g., relative Python imports, standard packaging). If the project uses non-standard patterns (bare imports, specific naming conventions, module-level structure), state them explicitly in the prompt. A single line like "Use from harness import X, not from .harness import X" prevents import mismatches during assembly.

Budget for Assembly Fixups

Parallel agent work produces ~3 fixups per orchestration run, each under 5 minutes. Common fixup categories: import conventions, module-level side effects, SDK exception constructor signatures, validator patterns. This is the expected cost of parallel work, not a failure. Budget 15-20 minutes for assembly and fixup after each orchestration run.

Two-Phase Orchestration: Specs First, Then Implementation

When orchestrating multi-agent work for a milestone, decompose in two phases:

1. Phase 1: Spec-writing agents produce the contracts (using the plan as input).
2. Review: Human reviews specs for cross-spec consistency before proceeding.
3. Phase 2: Implementation agents receive actual spec files (not plan descriptions).

This works significantly better than defining all tasks upfront because spec agents validate the plan against reality, the review step catches cross-spec inconsistencies, and implementation agents work from concrete contracts rather than plan summaries.

Include an Integration Verification Task After Orchestration

Agent orchestration leaves integration gaps at component boundaries. Each agent completes its assigned scope correctly, but nobody owns the integration points between them (e.g., stub comments, ORM mapping methods not updated for new fields). After every orchestration run, include an explicit integration verification step that checks cross-component contracts — call sites, shared data models, and handoff points.

Integration failures fall into a taxonomy of root causes. Categorize failures before fixing and address them category-by-category:

1. New required fields on shared dataclasses without defaults — An agent adds a field to a shared data model without a default value, breaking every other agent's code that constructs that model.
2. Mock targets that don't match actual code structure — Agents patch "module.ClassName.method" but the actual code uses a different import path or method name, so tests pass against mocks but fail against real code.
3. Tests written before implementation is finalized — Tests assume behaviour that changed during implementation. The spec said one thing, the implementation diverged, and the test was never updated.
4. Tests with clear=True on os.environ missing required env vars — Tests that clear the environment forget to set variables the code requires at import time or during setup, causing failures unrelated to the tested behaviour.

Decompose Along File Boundaries

When splitting work into parallel agent tasks, ensure each task writes to distinct files. When two agents must modify the same file, make the shared changes small and predictable — identify the conflict point upfront so the merge is trivial. File-boundary decomposition produces zero-conflict assemblies.

Parallel Agent Patterns: File Contention, WebFetch Limits, Narrow Reads

When using parallel subagents (Task tool with multiple concurrent invocations):

1. Never have two agents edit the same file concurrently. Subagent writes silently collide — one agent's edit wins and the other is lost with no error. Instead, have each agent RETURN prepared text in its response and apply the edits sequentially from the main thread. The main thread owns writes; agents produce content.
2. Subagents cannot use WebFetch (permission denied in the subagent sandbox). Perform web fetches in the main conversation and pass the retrieved content to agents as input. Delegate file processing and analysis to agents, not network IO.
3. Give agents narrow read instructions (e.g., "read only the Key Data Points section of finding X") to prevent expensive full-file reads that blow their context budget.

Verified pattern: 4 parallel agents splitting files 3-4 each verified 195 claims across 16 files in a single round, returning prepared findings to the main thread for sequential application.

Choose Manual Implementation for Tightly-Coupled Cross-Component Work

When changes are small per file (5-15 lines) but tightly coupled across many files (each change depends on the previous), skip agent orchestration and implement manually. The assembly overhead exceeds the implementation time. Agent orchestration excels when tasks are independent and substantial; manual implementation excels when work is sequential and interconnected.

Anti-Patterns

Specs as documentation, not contracts

Symptom: Specs describe what was built, updated after the fact. Tests don't reference spec IDs. Fix: Write specs before code. Tests reference requirement IDs. Specs are the input, not the output.

Mega-spec

Symptom: One large spec covering the entire system. Agents must read thousands of lines to find what they need. Fix: Split by subsystem. Each spec should be readable in under 5 minutes.

Spec without scenarios

Symptom: Requirements are abstract ("handle errors gracefully"). No concrete examples. Fix: Every requirement needs at least one given/when/then scenario with specific inputs and outputs.

Implementation details in specs

Symptom: Spec dictates variable names, algorithm choices, internal data structures. Fix: Specs define what and why, not how. The interface is specified; the implementation is free.

Untested requirements

Symptom: Requirements exist in the spec but no test references them. They drift without anyone noticing. Fix: Spec coverage check in CI. Every requirement ID must appear in at least one test function name.

Multi-Model Review for Security-Critical Specs

Running the same security review with two different LLM models and comparing outputs catches significantly more issues than either alone. In measured experiments, only 62% of findings overlapped — the union covered 38% more issues. For security-critical specs, the cost of a second model review is justified by the coverage improvement. When models agree, confidence is high; when they disagree, escalate to human review.

Security Review Before Agent Implementation

Run a security review of the plan before decomposing into implementation tasks. A 15-minute review before coding catches real vulnerabilities that would otherwise ship to production. Example: a source injection vulnerability in a wrapper script was caught during plan review that would have been a production security hole if caught only after implementation.

Forward-Looking Annotations Must Be Labelled

RBAC annotations and other forward-looking requirements in specs (e.g., "requires admin role" when RBAC isn't implemented yet) must be prefixed with "(Future MN)" to indicate they describe future enforcement, not current behaviour. Without the label, agents may implement access checks prematurely or build infrastructure not needed for the current milestone.

New Fields on Shared Data Models Must Have Defaults

When adding fields to shared data models (dataclasses, Pydantic models, Protobuf messages), new fields must always have default values. Code in other branches, agents, or callers constructs instances without the new field. A required field without a default breaks every existing caller. Use field(default_factory=list) for collections and None or sentinel values for optionals.

Agent Prompts Must Include Mock Targets and Import Conventions

Agents working in isolated worktrees or containers cannot discover mock targets or import conventions from sibling test files. Every implementation prompt must explicitly state: the exact function paths to mock (e.g., patch.object(instance, "_method_name") not patch("module.function")), the project's import convention, and test fixture patterns.

Plan-First Approach Eliminates Fix Cycles for Cross-Cutting Changes

For changes touching 5+ files across multiple subsystems, invest 30-45 minutes in exploration and planning before writing code. Measured results: sessions with plan-first had 0 fix commits; sessions with code-first had 7:1 fix:forward ratios. Use parallel exploration agents to cover different dimensions of the problem space.

Wave-Based TDD Dispatch

For large milestones with many subsystems, dispatch test-writing and implementation as two explicit waves rather than interleaving them per agent:

Wave 1 — Test agents: Each agent receives the spec for one subsystem and writes all tests. No implementation code yet. Tests must all fail (or be skipped) at the end of Wave 1. Commit the test files to the agents branch.

Wave 2 — Implementation agents: Each agent receives the spec + the failing tests written by Wave 1. The agent's success criterion is "make your tests pass without modifying the test file." This hard separation prevents the common failure mode where an agent makes a test pass by weakening it.

Human review gate between waves: Before starting Wave 2, review the Wave 1 test files for coverage gaps and assert quality. It's cheaper to fix tests before implementation than after. Check that tests are genuinely failing (not just skipped), that assertions are specific, and that edge cases from the spec scenarios are covered.