api-design: add API-first methodology, DX, and contract testing

Extends api-design.md beyond its security/operations focus with three new dimensions: - §0 API-First Design Process — OpenAPI 3.1 as single source of truth, Spectral governance, dogfooding (UIs consume the public API, no privileged backdoors), auth-required-by-default as a design stance. - §7 Documentation and Developer Experience — Scalar/Mintlify, RFC 9457 Problem Details error envelope, interactive playgrounds, generated SDKs (Stainless, Speakeasy, Fern), RFC 9745 deprecation signals and changelog UX. - §8 Contract Testing and API Quality — schema validation in the test suite, Pact CDC vs provider verification, Schemathesis property-based fuzzing, oasdiff drift detection in CI, the API test pyramid. Intro, cross-refs in §3.1/§3.3/§4.1, and Sources block reorganised by topic. Index entry in BESTPRACTICES.md updated. PLAN file included for traceability. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-25 13:42:03 +12:00
parent 22d49b2c9a
commit 6154031cf9
3 changed files with 431 additions and 6 deletions
--- a/BESTPRACTICES.md
+++ b/BESTPRACTICES.md
@@ -23,7 +23,7 @@ Generalised best practices extracted from real project work via the `/distill-be
 - [Docker UID Matching](docker-uid-matching.md) — UID wrapper entrypoint for mounted volumes, gosu pattern, when to use vs K8s securityContext
 - [Database Selection](database-selection.md) — SQLite is not a production database; always use PostgreSQL for services with FQDNs, multiple consumers, or concurrent access
 - [Docker](docker.md) — gosu PID 1, GIT_SSH_COMMAND scope, slim image health checks, buildx local images, Compose networking/restart gotchas, volume paths, override merge behaviour, init script privilege order, payload size limits, bind-mount rm gotcha
- [API Design](api-design.md) — Transport security, auth (OAuth2/JWT/mTLS), versioning, pagination, error handling, idempotency, rate limiting, input validation, zero-trust patterns
+- [API Design](api-design.md) — API-first methodology (OpenAPI 3.1 source of truth, Spectral governance, dogfooding, auth-by-default); transport, auth (OAuth2/JWT/mTLS), versioning, errors, idempotency, rate limiting, validation, zero-trust; docs/DX (Scalar/Mintlify, RFC 9457 errors, generated SDKs, RFC 9745 deprecation); contract testing (Schemathesis, Pact, oasdiff drift detection, test pyramid)
 - [API Integration](api-integration.md) — Client-side third-party API integration: capability verification, app-layer compensation, git+SOPS polling sync, bidirectional SoR
 - [Octopus Process Templates](octopus-process-templates.md) — OCL syntax, step template references, channel scoping, parameters, versioning, Platform Hub patterns
 - [LLM Code Security](llm-code-security.md) — Security vulnerabilities in AI-generated code: injection flaws, hardcoded secrets, hallucinated packages, over-permissive defaults, IaC risks, crypto mistakes, operational vulnerabilities (idempotency, CI/CD integrity, supply chain provenance, concurrent access), review checklists
--- a/api-design.md
+++ b/api-design.md
@@ -1,8 +1,103 @@
 # API Design
-Best practices for REST/HTTP APIs in internal microservices and platform services. Focused on practical defaults -- not aspirational ideals. Sourced from OWASP API Security Top 10 (2023), RFC 9700 (OAuth 2.0 Security BCP, January 2025), Google AIP, and production experience.
+Best practices for REST/HTTP APIs in internal microservices and platform services. Covers four dimensions: **methodology** (design-first, dogfooding, governance), **security and operations** (transport, auth, validation, service mesh), **developer experience** (docs, SDKs, deprecation signals), and **quality** (contract testing, drift detection). Focused on practical defaults -- not aspirational ideals. Sourced from OWASP API Security Top 10 (2023), RFC 9700 (OAuth 2.0 Security BCP, January 2025), RFC 9457 (Problem Details, 2023), RFC 9745 (Deprecation header, 2024), Google AIP, and production experience.
-Cross-references: [Security Architecture](security-architecture.md) covers the server boundary rule and proxy patterns. [Secrets Management](secrets-management.md) covers credential storage and rotation.
+Cross-references: [Security Architecture](security-architecture.md) covers the server boundary rule and proxy patterns. [Secrets Management](secrets-management.md) covers credential storage and rotation. [Test-Driven Development](test-driven-development.md) covers the testing principles that §8 extends.
 ---
 ## 0. API-First Design Process
 This section covers methodology -- *how* APIs get designed and governed, not *what* goes in them. The mechanical sections (§1-§6) assume an API-first workflow. If your team is code-first, start here.
 ### 0.1 Design the contract before writing code
 **Principle:** The OpenAPI document is authored, reviewed, and committed before any handler code is written. The spec drives mocks, SDKs, docs, validation middleware, and contract tests in parallel -- not as artifacts generated after the fact.
 **Why it matters:** Code-first specs (annotations on handlers exporting OpenAPI) describe how the API was implemented, not how it should be used. They rebake internal types, drift the moment someone refactors, and miss design flaws because the spec inherits them. The Postman 2024 State of the API report puts API-first adoption at 74% (83% including partial adopters).
 **How to implement:**
 - Treat the OpenAPI file as source code -- in the repo, in PRs, code-reviewed, versioned.
 - Backend, frontend, and partners build against the same spec from day one. Use Prism (or any OpenAPI mock server) to unblock parallel work before the service exists.
 - Run a brief design review before merging the spec -- focus on resource modelling, lifecycle, and breaking-change risk; let the linter (§0.3) catch mechanical issues.
 - For greenfield, consider TypeSpec (Microsoft) for spec authoring -- it compiles to OpenAPI and is faster to write than raw YAML.
 **Anti-patterns:**
 - Spec generated from code annotations and never reviewed independently -- drifts within months.
 - "We'll document it after v1 ships" -- guarantees a v2 rewrite once you discover the design flaws.
 - OpenAPI file treated as build output (not in PRs, not reviewed).
 - Specs that mirror the database schema 1:1 instead of designing the consumer-facing contract.
 ### 0.2 OpenAPI 3.1 as the single source of truth
 **Principle:** Standardize on OpenAPI 3.1 (not 3.0). One document drives docs, SDKs, mocks, validation, and tests across the entire API surface.
 **Why it matters:** OpenAPI 3.1 is a superset of JSON Schema Draft 2020-12; 3.0 was a near-but-not-quite subset that forced tooling to maintain two parallel schema engines. Standardizing on 3.1 lets you use one schema language across REST APIs, AsyncAPI 3.0 events, and validation libraries -- no more "validation schema" / "docs schema" split.
 **How to implement:**
 - Migrate from 3.0 → 3.1: `nullable: true` is gone (use `type: ["string", "null"]`); `exclusiveMinimum`/`exclusiveMaximum` take values not booleans; `example` becomes `examples` (array); file uploads use `contentMediaType`/`contentEncoding`; the spec gains first-class `webhooks`.
 - Adopt AsyncAPI 3.0 for event payloads -- same JSON Schema dialect, one mental model.
 - Pin to a specific minor (`openapi: 3.1.2`) to avoid silent tooling drift.
 **Anti-patterns:**
 - Staying on 3.0 to avoid the migration -- locks you out of conditional schemas (`if`/`then`/`else`), tuple validation, and tool consolidation.
 - Maintaining separate "schemas for validation" and "schemas for docs" -- they will diverge.
 - Using `nullable: true` in a 3.1 doc -- silently ignored by some tools, causes subtle validation gaps.
 - Treating OpenAPI as docs only while runtime validation is implemented separately and drifts.
 ### 0.3 Governance through linting
 **Principle:** API style guides are enforced as code. A Spectral ruleset is checked into the repo and runs in CI on every spec change, blocking merges on errors.
 **Why it matters:** Style guides written as wiki pages get ignored. Lint rules don't. Mechanical enforcement also frees design reviews to focus on intent and edge cases instead of bikeshedding naming.
 **How to implement:**
 - Adopt Spectral 6.x. Extend the default `oas` ruleset and layer the [Spectral OWASP ruleset](https://github.com/stoplightio/spectral-owasp-ruleset) on top -- it codifies the OWASP API Security Top 10 (2023) at the spec level (e.g. flags any operation lacking `security` as API2:2023).
 - Enforce: resource naming (plural nouns, kebab-case paths, camelCase fields); pagination shape; canonical error envelope (RFC 9457, see §7.3); required `operationId` for SDK gen; mandatory `security` block on every operation; response schemas on every documented status code.
 - Layer rulesets by maturity -- a base ruleset for all APIs, stricter rules for partner/public APIs.
 - Pair with `oasdiff` for breaking-change detection (covered in §8.4).
 **Anti-patterns:**
 - Style guide as a wiki page nobody reads -- encode every rule that *can* be linted.
 - One monolithic ruleset applied identically to a prototype and a public API.
 - Per-team error envelopes -- every service inventing its own error shape.
 - Treating every Spectral warning as equally severe -- tune severities or the team will start ignoring all of them.
 ### 0.4 Dogfood your own API
 **Principle:** Internal UIs, admin tools, mobile clients, and partner integrations all consume the same public API surface as third-party developers. No privileged backdoors. No internal-only fields. No hidden endpoints.
 **Why it matters:** If your own dashboard cannot authenticate, paginate, or recover from a 429, neither can your customers. Dogfooding is the forcing function that keeps the API actually usable -- and it surfaces auth gaps, rate-limit gaps, and missing affordances before customers find them. The Bezos 2002 mandate at Amazon is the canonical formulation: "no direct linking, no direct reads of another team's data store, no shared-memory model, no back-doors whatsoever."
 **How to implement:**
 - Public and internal clients live in the same repo where possible -- code review catches API shortcuts.
 - Internal traffic hits the same gateway, auth, and rate limits as external traffic. No separate "internal" tier.
 - Spectral rule that flags `x-internal: true` operations on the public spec -- prove the absence of backdoors mechanically.
 - Stripe-style "friction logging" -- when teams build new abstractions, document every snag before the abstraction reaches GA. The snags become the next batch of API improvements.
 **Anti-patterns:**
 - Admin/internal endpoints that bypass auth or rate limits "because it's just us."
 - Internal-only fields on shared schemas leaking sensitive data, or worse -- external consumers come to depend on them.
 - A separate `/v1-internal` API that diverges from the public one, doubling maintenance.
 - UI that talks directly to the database while customers go through the API -- every UI feature becomes a customer feature request the API can't satisfy.
 ### 0.5 Auth required by default -- as a design stance
 **Principle:** Every operation in the spec has a `security` requirement at design time. Unauthenticated endpoints (health checks, public OIDC discovery) are the rare, deliberately-justified exception, tagged so an auditor can list them in seconds.
 **Why it matters:** This is "default deny" applied at API design time, not at the firewall. There is no internal network in a zero-trust model -- every request, including service-to-service, proves identity. Mechanics live in §2; the *design stance* in §0.5 is what decides whether your spec ever reaches a security reviewer with anonymous endpoints in it.
 **How to implement:**
 - Spec-level: every operation has a `security` block. Spectral OWASP rule `owasp:api2` enforces it.
 - Code-level: middleware rejects any request to an unauthenticated route unless that route is on an explicit allowlist.
 - For service-to-service calls, identity is mTLS (SPIFFE) or a short-lived JWT -- see §2.3.
 - Track unauthenticated endpoints in a single inventory file. Auditors review it; security review is required to add to it.
 **Anti-patterns:**
 - Operations with no `security` block ("we'll add it later" -- you won't).
 - IP allowlist or VPN as the only control between services -- collapses the moment someone runs the service in a different environment.
 - An "internal" tier with no auth because "it's behind the load balancer."
 - API keys as the only credential, shared across services, never rotated.
 ---
@@ -143,7 +238,7 @@ Cross-references: [Security Architecture](security-architecture.md) covers the s
 ### 3.1 Versioning
-**Principle:** Version your API from day one using URL path versioning (`/v1/`). Support at most two versions simultaneously.
+**Principle:** Version your API from day one using URL path versioning (`/v1/`). Support at most two versions simultaneously. See §7.5 for the deprecation-comms counterpart (`Deprecation` and `Sunset` headers, changelog UX) and the alternative versioning models (Stripe-style dated, GitHub-style header).
 **Why it matters:** Breaking changes without versioning cause cascading failures across all consumers simultaneously. Supporting too many versions creates maintenance burden and security risk (old versions may lack patches).
@@ -182,7 +277,7 @@ Cross-references: [Security Architecture](security-architecture.md) covers the s
 ### 3.3 Error handling
-**Principle:** Return structured, machine-readable errors with stable error codes, human-readable messages, and consistent shape across all endpoints.
+**Principle:** Return structured, machine-readable errors with stable error codes, human-readable messages, and consistent shape across all endpoints. Standardise on RFC 9457 Problem Details (`application/problem+json`) -- see §7.3 for the spec-side definition and error catalog pattern.
 **Why it matters:** Inconsistent error formats force every consumer to write custom parsing logic. Missing error codes make automated retry decisions impossible. Leaking stack traces exposes internals to attackers.
@@ -253,7 +348,7 @@ Cross-references: [Security Architecture](security-architecture.md) covers the s
 ### 4.1 Schema validation at the edge
-**Principle:** Validate all request bodies against a schema (OpenAPI/JSON Schema) at the API gateway or middleware layer. Reject requests that don't conform before they reach business logic.
+**Principle:** Validate all request bodies against a schema (OpenAPI/JSON Schema) at the API gateway or middleware layer. Reject requests that don't conform before they reach business logic. The same OpenAPI document drives runtime validation here, in-test validation in §8.1, and the docs in §7.1 -- one source of truth (§0.2).
 **Why it matters:** Invalid input that reaches business logic causes unpredictable behavior -- crashes, data corruption, injection attacks. Edge validation is the first line of defense.
@@ -427,6 +522,213 @@ Cross-references: [Security Architecture](security-architecture.md) covers the s
 ---
 ## 7. API Documentation and Developer Experience
 The OpenAPI document from §0.2 is also the input to docs, SDKs, and changelogs. Hand-written variants drift.
 ### 7.1 Render docs from the OpenAPI spec
 **Principle:** Documentation is rendered automatically from the committed OpenAPI document by an open-source or SaaS tool -- never hand-written prose that drifts from code.
 **Why it matters:** Hand-written docs are wrong within weeks of any active API. Spec-driven docs cannot drift because they are recompiled on every spec change. The 2024-2025 doc-rendering market shifted decisively: Scalar (open source, MIT, framework-native middleware) is the default new pick; Mintlify dominates the SaaS tier (markdown + OpenAPI, used by Anthropic, Microsoft, Coinbase). Swagger UI still works but is no longer the default new-project choice; Stoplight Elements development slowed sharply after the SmartBear acquisition.
 **How to implement:**
 - New projects -- pick Scalar for OSS, Mintlify or ReadMe.com for SaaS. Use Redoc if you want a single-page reference style.
 - Docs build and deploy on every spec change; pipeline fails closed if the spec doesn't lint.
 - Host docs at a stable URL (`docs.<domain>` or `<api-domain>/docs`).
 - Validate examples against their schemas in CI -- Spectral rule `oas3-valid-schema-example`.
 **Anti-patterns:**
 - Defaulting to Swagger UI in 2026 by habit -- Scalar is a near drop-in with better UX.
 - Hand-written docs maintained alongside the spec -- they always drift.
 - Picking Stoplight Elements expecting active development -- verify the roadmap.
 - SaaS doc vendor whose OpenAPI ingestion is brittle -- test with your real spec before committing.
 - Rendering docs from a build-time snapshot that is never re-validated against the deployed API.
 ### 7.2 Interactive playgrounds with real auth
 **Principle:** Every endpoint in the docs is callable from the page. Auth is collected once at the top of the page; all subsequent requests sign automatically. Examples come from the spec, not from `"string"` / `0` placeholders.
 **Why it matters:** The fastest path to a developer's first successful request is the only DX metric that matters. Code samples in their language reduce friction further. Playgrounds that proxy requests through the docs vendor leak credentials and break for CORS-restricted APIs.
 **How to implement:**
 - Every endpoint shows code samples in cURL plus 4-6 SDK languages (TypeScript, Python, Go, Java, Ruby, PHP cover ~95% of demand). Generate samples from the same SDK pipeline so they cannot drift.
 - Persist credentials in browser session -- never proxy through the docs vendor.
 - Environment switcher (prod / staging / sandbox) is a first-class control.
 - Every operation has at least 2-3 named examples in the spec (`examples: { minimal: ..., withMetadata: ... }`); the playground pre-fills bodies from them.
 - Provide a Postman public workspace as a complement -- useful for forking and sharing collections, but not a substitute for the embedded portal.
 **Anti-patterns:**
 - "Try it" that proxies requests through the docs vendor -- leaks creds, breaks CORS, hides the real network call.
 - Re-prompting for credentials per endpoint instead of persisting a session.
 - Auto-filled bodies as `"string"` / `0` placeholders -- developer must hand-build every payload.
 - Code samples written by hand, drifting behind the SDK; or only cURL shown, forcing developers to translate.
 ### 7.3 Standard error envelope: RFC 9457 Problem Details
 **Principle:** Every error response uses the same envelope, defined once under `components.schemas.Problem` following **RFC 9457 Problem Details for HTTP APIs**. An error catalog page enumerates each `type` URI with its meaning, retryability, and remediation.
 **Why it matters:** RFC 9457 (2023) replaced RFC 7807 with a clearer link between `type` URIs and the extension fields a client can expect. Per-endpoint error shapes force every consumer to write custom parsing logic. The catalog turns errors into documentation a developer can search. Cross-reference: §3.3 covers error handling generally; §7.3 is the docs-side counterpart.
 **How to implement:**
 - Define one `Problem` schema with `type` (URI), `title`, `status`, `detail`, `instance`, plus your extensions (`code`, `correlation_id`, `errors[]` for validation). Use `application/problem+json` content type on error responses.
 - Every error response in the spec references `#/components/schemas/Problem` (or a refinement of it).
 - Error catalog page enumerates each `type` URI -- meaning, HTTP status, retryability, remediation, link to migration if deprecated.
 - Examples on every error response, with at least these named cases per endpoint where applicable: success, business-rule failure, validation failure, auth failure.
 **Anti-patterns:**
 - Per-endpoint bespoke error shapes (`{error: "..."}` here, `{message, code}` there) -- clients can't write one error handler.
 - Single example per endpoint when the realistic case has 4+ shapes.
 - Documenting only HTTP status codes without an application-level `type` URI taxonomy -- clients branch on prose `message` strings.
 - Examples that don't validate against the schema -- run `oas3-valid-schema-example` in CI.
 ### 7.4 Generated SDKs
 **Principle:** SDKs are generated from the OpenAPI spec on every change. The generator opens a PR against the SDK repo so changes can be reviewed and released on a deliberate cadence.
 **Why it matters:** Hand-written SDKs maintained by the API team always fall behind the spec. A generated SDK cannot drift. Stripe's pipeline -- a single internal definition fanning out to ~10 SDKs on a daily release cadence -- is the reference architecture.
 **How to implement:**
 - For public/customer-facing SDKs -- Stainless (used by Stripe, Anthropic, Cloudflare, OpenAI) or Speakeasy (10 languages, single-runtime-dependency TS output, ships as a self-contained binary). Fern is a credible alternative.
 - For internal stubs and prototypes -- openapi-generator (open source, 50+ languages); generated quality is uneven, fine for internal use but weak for public SDKs without heavy template customisation.
 - A good SDK includes: idiomatic per-language style; automatic retry with exponential backoff + jitter on 429/503; transparent pagination (iterator/async-iterator hides cursor mechanics); typed errors as a discriminated union; built-in auth helpers (OAuth refresh, key rotation); webhook signature verification; small dependency footprint.
 - Pin SDK versions to dated API versions (Stripe model) -- see §7.5.
 **Anti-patterns:**
 - Hand-written SDKs maintained by the API team -- always behind the spec.
 - openapi-generator default templates shipped as a public SDK -- surfaces deprecated framework calls, looks non-idiomatic.
 - SDKs with no retry, no pagination helper, untyped errors -- every consumer rebuilds the same plumbing.
 - SDKs pulling 25-40 transitive dependencies for a thin HTTP wrapper.
 ### 7.5 Changelog, versioning UX, and deprecation signals
 **Principle:** API changes are communicated both in docs and on the wire. Wire signals are RFC-defined headers (`Deprecation`, `Sunset`, `Link`); the docs side is a machine-readable changelog with migration guides linked from each entry.
 **Why it matters:** Deprecating an endpoint in docs only means clients on old SDKs never learn. Sunset dates without migration guides leave developers with nowhere to go. Mature APIs use both channels. Cross-reference: §3.1 covers versioning models; §7.5 is the deprecation-comms counterpart.
 **How to implement:**
 - Wire signals -- `Deprecation: @<unix-ts>` (RFC 9745, finalized 2024); `Sunset: <http-date>` (RFC 8594, 2019); `Link: <migration-url>; rel="deprecation"` plus `rel="successor-version"`.
 - Versioning models -- pick one consciously:
  - **Stripe-style dated versions** (`2026-04-22.<codename>`) -- SDK clients pin a version; minor releases backward-compatible; breaking changes cluster into named majors. Best for fine-grained evolution.
  - **GitHub-style `X-API-Version` header** -- calendar-dated, opt-in. Lighter weight than Stripe's model.
  - **URL-path versioning** (`/v1/`, `/v2/`) -- coarse-grained, simple. Acceptable for small APIs; awkward for fine-grained evolution. Used by Twilio and SendGrid for major boundaries.
 - Generate the changelog from `oasdiff` output, filterable by API version, linking every entry to a migration guide.
 - Provide an RSS/Atom or JSON feed of changes so customers can wire alerts.
 - Deprecation lead time -- 12 months minimum for public APIs (Twilio policy), 6+ for partner APIs.
 **Anti-patterns:**
 - Deprecating in docs only with nothing on the wire -- clients on old SDKs never learn.
 - Sunset dates without migration guides.
 - Major-version-bump-only versioning (`/v1` → `/v2`) for small additive changes -- forces clients to choose between staying on a frozen API or rewriting everything.
 - Changelog as a hand-written prose blog with no machine-readable feed and no link to API version.
 - "Breaking changes" buried in release notes without explicit replacement field/endpoint pointers.
 ---
 ## 8. Contract Testing and API Quality
 Schema validation, contract tests, and drift detection turn the OpenAPI document from §0.2 into a executable contract that the running service must obey.
 ### 8.1 Schema validation in the existing test suite
 **Principle:** Every test that exercises an HTTP handler validates both the request and response against the committed OpenAPI spec, in-process. This is the foundation of contract testing -- cheap, fast, integrated with whatever test framework you already use.
 **Why it matters:** Spec drift kills consumers. A middleware that asserts every request and response against the spec catches "spec lies, code is right" drift the moment it appears. Validating only requests (and trusting framework parsing for responses) is the most common gap -- response drift is the more common production bug.
 **How to implement:**
 - Per language:
  - **Python** -- `openapi-core` (Flask, Django, Falcon, Starlette, Werkzeug, Requests integrations).
  - **Node** -- `express-openapi-validator` (auto-validates requests, responses, security).
  - **Ruby** -- `committee` (Rack middleware + `Committee::Test::Methods` test helpers).
  - **Java/Kotlin** -- `springdoc-openapi` + Spring REST Docs, or `atlassian/swagger-request-validator` for MockMvc/RestAssured.
  - **Go** -- `kin-openapi` (`openapi3filter`).
 - Validate both directions in tests, not just at runtime.
 - Reload the spec on every test run -- never cache it across runs in CI.
 - Validation failures must be build-failing; warning-level is ignored within a sprint.
 **Anti-patterns:**
 - Validating only requests, not responses -- response drift is the more common bug.
 - Hand-written JSON Schema next to the OpenAPI doc -- they will diverge.
 - Validation failures as warnings instead of build failures.
 - Loading the spec once at app startup and never re-loading in tests, so spec edits don't reach the validator.
 ### 8.2 Provider verification vs consumer-driven contracts
 **Principle:** Provider-side spec compliance (Schemathesis, Dredd) is the default. Consumer-driven contracts (Pact) are a deliberate add-on for APIs with a small, known set of internal consumers.
 **Why it matters:** CDC with Pact is high-value when you have a mobile app + web SPA + a couple of internal services that talk to one provider -- each consumer publishes a contract describing what it actually calls; the provider verifies against all of them and uses `can-i-deploy` to gate releases. CDC is overkill (and frequently abandoned) when the provider has many unknown consumers, when teams are organisationally distant, or when an OpenAPI-first workflow already gives you provider compliance via §8.1 and §8.3.
 **How to implement:**
 - Default -- Schemathesis (provider-side, OpenAPI-driven, see §8.3) plus the schema validation in §8.1.
 - Add Pact when -- you publish SDKs you control, you have a known set of internal consumers, or consumer teams want guarantees independent of the provider's tests.
 - For Pact -- store contracts in PactFlow / Pact Broker. Gate provider deploys on `can-i-deploy`. The contract is not advisory -- the broker is part of the release pipeline.
 - For mixed worlds (some consumers known, many not) -- bi-directional contract testing in PactFlow accepts the OpenAPI document as the provider contract; gives you provider compliance without per-consumer Pacts.
 **Anti-patterns:**
 - Adopting Pact organisation-wide for an API with public/unknown consumers -- you can't enumerate the contracts.
 - Consumer Pacts using random data -- every run produces a "new" contract; the broker fills with noise.
 - Provider verification as advisory (not gating deploys) -- the contract becomes documentation only.
 - Conflating CDC with end-to-end testing -- they answer different questions, and CDC should not stand in for E2E nor vice-versa.
 ### 8.3 Property-based fuzz testing the spec
 **Principle:** Run Schemathesis against the running service in CI. It generates conformant requests from the OpenAPI document, exercises endpoints in stateful sequences derived from `links`, and asserts behavioural properties -- no 5xx where 4xx is expected, response bodies match the declared response schema, status codes are documented, security boundaries hold.
 **Why it matters:** Unit tests probe values you thought of; property-based fuzzing probes values the schema *says are legal* but you never tried -- Unicode, edge integers, deeply nested optional fields, missing-but-valid combinations. Schemathesis caught 1.4×-4.5× more defects than competing API fuzzers in published evaluations; production users include Spotify, JetBrains, Red Hat, WordPress.
 **How to implement:**
 - Adopt Schemathesis 4.x. Use stateful mode (`st fuzz`) -- most real bugs are sequence-dependent (create → read returns wrong shape) and stateless fuzzing can't find them.
 - Run against a real service backed by a real database, not mocks -- many bugs only surface with persistent state.
 - Wire reports into CI -- Allure, JUnit, or HAR for replaying failures.
 - Use `after_validate` hooks for custom invariants beyond what the schema declares (e.g. "the same `id` must reappear on subsequent GETs").
 - For dynamic auth (OAuth, refresh tokens), Schemathesis 4.13+ has first-class config -- no Python glue code needed.
 **Anti-patterns:**
 - Running Schemathesis only against a mocked dev server -- misses bugs that depend on real DB state and auth.
 - Excluding 5xx checks because they're "flaky" -- a 5xx on a schema-valid request is the most important signal Schemathesis produces. Fix the bug, don't suppress the check.
 - Running it once and suppressing all failures with `--exclude` -- hides drift forever.
 - Skipping stateful mode -- most real bugs are sequence-dependent.
 ### 8.4 Drift detection in CI
 **Principle:** Three artefacts must stay in sync -- (a) the spec in the repo, (b) the spec the running service exposes, (c) the behaviour of the deployed production service. CI must detect drift between any pair.
 **Why it matters:** A code-first project without a "regenerate-and-diff" CI step *will* drift -- the only question is when. Runtime drift between deployed-API behaviour and the documented spec is the most expensive failure because consumer SDKs were generated against the documented version.
 **How to implement:**
 - **Spec-vs-spec on PR** -- `oasdiff` (Go CLI + GitHub Action) checks 450+ categories of breaking changes; non-zero exit gates the PR. Pair with Spectral for style/governance.
 - **Code-vs-spec in CI** -- for code-first stacks, regenerate the spec from code in CI and `oasdiff` against the committed spec -- fail the build if non-empty.
 - **Runtime-vs-spec** -- sample real traffic and validate against the spec. Optic captures HTTP traffic and diffs against the spec; Speakeasy offers SDK-driven runtime drift detection.
 - Breaking-change detection must be a required PR check, not a Slack notification.
 **Anti-patterns:**
 - Code-first project with no spec regeneration step -- the committed spec rots silently.
 - Breaking-change detection as a Slack notification rather than a required PR check.
 - Linting only on a schedule rather than per-PR -- drift accumulates between runs.
 - No production sampling -- the deployed API can diverge from spec without anyone noticing until a consumer breaks.
 ### 8.5 The API test pyramid
 **Principle:** Layer tests as -- unit → schema-validated handler tests (§8.1) → contract tests (§8.2 / §8.3) → integration (real DB, real downstreams) → smoke tests of the deployed environment. Contract tests sit between unit and integration: faster than integration, broader in scope than unit.
 **Why it matters:** 100% mocked tests with no contract layer encode developer assumptions, not the actual provider's behaviour. The classic failure mode -- "all green in CI, broken in prod" -- comes from this gap. A contract layer breaks the dependency on slow, flaky integration environments while still catching real interface drift.
 **How to implement:**
 - Unit tests cover pure logic with no I/O -- fastest, most numerous.
 - Schema-validated handler tests (§8.1) cover most contract-shape concerns in-process.
 - Contract tests (§8.2 / §8.3) hit the running service, no mocks. Run on every PR.
 - Integration tests cover real DB + real downstream stubs (or testcontainers). Slower; run on PR but fewer in number.
 - Post-deploy smoke tests are a tiny set of canary checks (login, list, create one resource, delete it) against the deployed environment -- catch infra/config drift (TLS, auth proxy, env var) that contract tests cannot.
 **Anti-patterns:**
 - 100% mocked tests with no contract layer -- mocks encode assumptions, drift is invisible until prod.
 - Contract tests against a stale spec -- the suite passes but the spec doesn't match deployed behaviour. Pair with §8.4.
 - Skipping post-deploy smoke tests because "we have contract tests" -- contract tests don't catch infra drift.
 - Inverted pyramid (UI-test-heavy "ice cream cone") -- slow, flaky, brittle.
 - Treating Schemathesis as integration tests -- it's contract verification, not business-flow testing; you still need a small set of curated end-to-end scenarios.
 ---
 ## OWASP API Security Top 10 (2023) Quick Reference
 For context, the current OWASP API Security Top 10 maps to the practices above:
@@ -448,6 +750,7 @@ For context, the current OWASP API Security Top 10 maps to the practices above:
 ## Sources
 ### Security and operations (§1-§6)
 - [OWASP API Security Top 10](https://owasp.org/API-Security/)
 - [RFC 9700 - OAuth 2.0 Security Best Current Practice (January 2025)](https://datatracker.ietf.org/doc/rfc9700/)
 - [OAuth best practices: RFC 9700 summary -- WorkOS](https://workos.com/blog/oauth-best-practices)
@@ -461,3 +764,44 @@ For context, the current OWASP API Security Top 10 maps to the practices above:
 - [Buoyant: Zero Trust, mTLS, and the Service Mesh](https://www.buoyant.io/blog/zero-trust-mtls-and-the-service-mesh-explained)
 - [Kong: Zero Trust with Service Mesh](https://konghq.com/blog/engineering/zero-trust-service-mesh-security)
 - [Microsoft Azure: Web API Design Best Practices](https://learn.microsoft.com/en-us/azure/architecture/best-practices/api-design)
 ### API-first methodology (§0)
 - [APIs You Won't Hate: A Developer's Guide to API Design-First](https://apisyouwonthate.com/blog/a-developers-guide-to-api-design-first/)
 - [Generating OpenAPI from Code is an Anti-Pattern -- Craig Drabik](https://medium.com/@craigdrabik/generating-openapi-or-swagger-from-code-is-an-anti-pattern-and-heres-why-bfb27aa98e2d)
 - [How Stripe Builds APIs -- Postman Blog](https://blog.postman.com/how-stripe-builds-apis/)
 - [Postman 2024 State of the API Report](https://www.postman.com/state-of-api/2024)
 - [TypeSpec: API Design-First with Microsoft TypeSpec](https://devblogs.microsoft.com/ise/design-api-first-with-typespec/)
 - [OpenAPI 3.0 → 3.1 Upgrade Guide](https://learn.openapis.org/upgrading/v3.0-to-v3.1.html)
 - [OpenAPI Specification v3.1.2](https://spec.openapis.org/oas/v3.1.2.html)
 - [Spectral OpenAPI Linter](https://github.com/stoplightio/spectral)
 - [Spectral OWASP API 2023 Ruleset](https://github.com/stoplightio/spectral-owasp-ruleset)
 - [Spectral API Governance -- Kin Lane (Jan 2025)](https://apievangelist.com/2025/01/21/the-properties-of-spectral-api-governance-rules/)
 - [Dogfooding -- Australian Government API Design Guide](https://apiguide.readthedocs.io/en/latest/principles/dogfood.html)
 - [Dogfooding Your API -- Zapier Engineering](https://zapier.com/engineering/api-dogfooding)
 ### Documentation and DX (§7)
 - [RFC 9457 -- Problem Details for HTTP APIs (2023)](https://datatracker.ietf.org/doc/html/rfc9457)
 - [RFC 9745 -- The Deprecation HTTP Response Header Field (2024)](https://www.rfc-editor.org/rfc/rfc9745.html)
 - [RFC 8594 -- The Sunset HTTP Header Field (2019)](https://www.rfc-editor.org/rfc/rfc8594.html)
 - [Speakeasy: Choosing a docs vendor (2025)](https://www.speakeasy.com/blog/choosing-a-docs-vendor)
 - [APIs You Won't Hate: Top 5 API docs tools 2025](https://apisyouwonthate.com/blog/top-5-best-api-docs-tools/)
 - [Scalar API Reference (open source)](https://github.com/scalar/scalar)
 - [Speakeasy: SDK Generator Comparison](https://www.speakeasy.com/blog/comparison-sdk-generators-openapi)
 - [Stainless: Cloudflare SDK pipeline case study](https://www.stainless.com/customers/cloudflare)
 - [Nordic APIs: 8 SDK generators reviewed (2025)](https://nordicapis.com/review-of-8-sdk-generators-for-apis-in-2025/)
 - [Stripe: API versioning](https://stripe.com/blog/api-versioning)
 - [Zalando RESTful API Guidelines: Deprecation](https://github.com/zalando/restful-api-guidelines/blob/main/chapters/deprecation.adoc)
 ### Contract testing and quality (§8)
 - [openapi-core (Python)](https://github.com/python-openapi/openapi-core)
 - [express-openapi-validator (Node)](https://github.com/cdimascio/express-openapi-validator)
 - [Pact -- Consumer-Driven Contract Testing](https://docs.pact.io/)
 - [PactFlow: Bi-directional vs Consumer-Driven Contract Testing](https://pactflow.io/difference-between-consumer-driven-contract-testing-and-bi-directional-contract-testing/)
 - [PactFlow: The Curious Case for the Provider-Driven Contract](https://pactflow.io/blog/the-curious-case-for-the-provider-driven-contract/)
 - [Tweag: Contract Testing (January 2025)](https://www.tweag.io/blog/2025-01-23-contract-testing/)
 - [Schemathesis](https://schemathesis.io/)
 - [Schemathesis on GitHub](https://github.com/schemathesis/schemathesis)
 - [Capital One: API Testing with Schemathesis](https://www.capitalone.com/tech/software-engineering/api-testing-schemathesis/)
 - [oasdiff -- OpenAPI breaking-change detector](https://www.oasdiff.com/)
 - [Speakeasy: OpenAPI Drift Detection](https://www.speakeasy.com/blog/openapi-spec-drift-detection)
 - [Martin Fowler: The Practical Test Pyramid](https://martinfowler.com/articles/practical-test-pyramid.html)
--- a/enhance-api-design-PLAN.md
+++ b/enhance-api-design-PLAN.md
@@ -0,0 +1,81 @@
 # Enhance api-design.md — API-First Design, DX, and Contract Testing
 ## Goal
 Extend the existing `api-design.md` (security/operational focus) with three new sections covering the **methodology**, **developer experience**, and **testing** dimensions of building APIs. The user's framing: "API-first design" — robust testing, auth required by default, UIs using the same API as the public surface, OpenAPI/Swagger as a humans-friendly contract, and self-documenting features.
 We are **enhancing the existing file**, not creating a new one. Same file, new sections.
 ## Scope
 ### New Section 0 — API-First Design Process (top of file, before Transport Security)
 - Design the contract before writing code
 - OpenAPI 3.x as the single source of truth
 - Style guides + Spectral linting / API governance
 - Design reviews
 - Dogfood your own API: UIs and internal tools consume the same public API surface
 - No privileged "internal-only" backdoors that bypass auth or expose extra fields
 - Auth required by default — design stance, not a bolt-on (mechanics live in §2)
 - Anti-patterns: code-first OpenAPI exports as an afterthought, separate "internal" and "public" APIs that drift, anonymous endpoints "because it's internal"
 ### New Section 7 — API Documentation & Developer Experience (after Service Mesh)
 - OpenAPI-driven docs: Swagger UI, Redoc, Scalar, Stoplight Elements
 - Interactive playgrounds with example requests
 - Examples and error catalogs embedded in the spec
 - Generated SDKs (openapi-generator, Speakeasy, Fern, Stainless)
 - Public changelog and deprecation signals (`Deprecation` / `Sunset` headers tied to docs)
 - Anti-patterns: hand-written docs that drift from code, no examples, no error reference
 ### New Section 8 — Contract Testing & API Quality
 - Schema validation in tests (request and response conformance)
 - Contract tests: provider-side (does the API match the spec?) and consumer-driven (Pact)
 - Property/fuzz testing the spec (Schemathesis, Dredd) to find edge cases the spec implies but the implementation breaks
 - Drift detection: spec-vs-implementation and spec-vs-deployed-API in CI
 - The API test pyramid: unit → contract → integration → smoke
 - Anti-patterns: tests that mock the API away, contract tests that only validate happy path, no consumer tests when SDKs are published
 ### Light edits to existing sections
 - §3.1 (Versioning) — cross-ref new §7 deprecation patterns
 - §4.1 (Schema validation) — cross-ref new §0 (OpenAPI as source of truth) and §8 (drift detection)
 - Update the file overview at the top to mention all three new dimensions
 ## Process
 1. ✅ Confirm scope with user (done)
 2. Write PLAN.md (this file)
 3. Dispatch 3 parallel WebSearch research agents:
   - Agent A: API-first methodology, OpenAPI as source of truth, governance/Spectral, dogfooding, design reviews (2024–2025)
   - Agent B: API documentation tooling — Swagger UI / Redoc / Scalar / Stoplight, SDK generation, examples-in-spec, deprecation UX
   - Agent C: Contract testing — Pact, Schemathesis, Dredd, OpenAPI-driven test generation, drift detection, the test pyramid for APIs
 4. Synthesize research into the three new sections, matching the existing **Principle / Why it matters / How to implement / Anti-patterns** structure
 5. Edit `api-design.md` — insert Section 0 at top, Sections 7+8 after current §6, light cross-refs in existing sections, update Sources block
 6. Update `BESTPRACTICES.md` index entry to include the new dimensions
 7. Validate: file renders cleanly, internal links resolve, no duplication with existing sections
 8. Stop. Do not commit unless user explicitly asks.
 ## Style requirements
 - Match the existing file's voice: direct, opinionated, no hedge words
 - Each numbered subsection: **Principle:** → **Why it matters:** → **How to implement:** (bulleted) → **Anti-patterns:** (bulleted)
 - Cite RFCs / OWASP / vendor docs in the Sources block at the bottom of the file
 - Keep concrete: name actual tools (Spectral, Schemathesis, Pact, Redoc, etc.), not generic "use a linter"
 ## Out of scope
 - Creating a separate `api-first-design.md` file
 - Rewriting or restructuring existing sections (1–6)
 - GraphQL or gRPC specifics — this file is REST/HTTP focused
 - A standalone tooling comparison matrix (briefly mention top tools per category, don't review them)
 ## Open questions
 - Whether to add a small "API-first vs. code-first" tradeoff note in §0 — leaning yes, brief
 - Whether SDK generation deserves its own §7.x subsection or fits inside doc tooling — leaning own subsection
 ## Verification
 After writing, re-read the file end-to-end and check:
 - The three new sections fit the existing tone
 - No duplication with §1–6
 - Cross-references work in both directions
 - Sources block updated