agent-runtime-framework/harnesses/contexts/spec-writing/v1/spec-exemplar.md

# Spec Exemplar: Rate-Limit Config Validator

This is a compact worked example of a spec written for **mechanical test generation**
(see `mechanical-test-generation.md` in `/workspace/best-practices/`). It covers a small,
self-contained subsystem end-to-end so every section can be read in full. Use it as the
structural template for real specs — same section order, same requirement shape, same
level of concreteness. The subsystem itself (a rate-limit config validator) is invented
for this exemplar; it is not part of any real codebase in this project.

A test-writer agent given ONLY this document should be able to produce correct imports,
correct assertion values, and correct mock boundaries with no other input — no reading
the codebase, no guessing module paths, no inventing error message text.

---

## Overview

The rate-limit config validator loads a service's rate-limit policy from a YAML file,
validates it against structural and semantic rules, and produces a `RateLimitPolicy`
object that the gateway middleware consumes at startup. It exists to catch
misconfiguration (overlapping rules, invalid windows, negative limits) before the
gateway starts serving traffic, rather than failing silently at request time.

## Responsibilities

- Parse a rate-limit policy YAML file into typed Pydantic models
- Validate structural correctness (required fields, types, value ranges)
- Validate semantic correctness (no two rules match the same route + method with
  different limits; window units are one of a fixed set)
- Produce a single validation report listing every error found (not just the first)
- Expose `load_policy(path) -> RateLimitPolicy` as the sole public entry point

## Dependencies

- `pydantic` v2 (data models and field validation)
- `pyyaml` (YAML parsing) — errors from malformed YAML are caught and re-raised as
  `PolicyLoadError`, never allowed to propagate as raw `yaml.YAMLError`
- No network calls, no filesystem writes — this is a pure load-and-validate module

## Module Layout

| Module | Location | Key exports |
|---|---|---|
| Policy models | `gateway/ratelimit/models.py` | `RateLimitRule`, `RateLimitPolicy`, `WindowUnit` (enum: `second`, `minute`, `hour`) |
| Validator | `gateway/ratelimit/validator.py` | `load_policy(path: str) -> RateLimitPolicy`, `PolicyLoadError`, `PolicyValidationError` |
| Overlap check | `gateway/ratelimit/overlap.py` | `find_overlapping_rules(rules: list[RateLimitRule]) -> list[tuple[RateLimitRule, RateLimitRule]]` |

| Test file | Spec requirements covered |
|---|---|
| `tests/ratelimit/test_models.py` | RLV-1, RLV-2 |
| `tests/ratelimit/test_validator.py` | RLV-3, RLV-4, RLV-5 |
| `tests/ratelimit/test_overlap.py` | RLV-6 |

## Requirements

- **RLV-1:** `RateLimitRule.limit` MUST be a positive integer (`>= 1`). A value of `0`
  or negative raises `pydantic.ValidationError` at model construction time.
  - Why: A zero or negative limit is not a valid rate limit — it either blocks
    everything or is meaningless. Catching it at the model layer means every caller
    gets the same guarantee for free, without re-checking in the validator.
  - **Scenario:** GIVEN a rule dict `{"route": "/api/orders", "method": "POST", "limit": 0, "window": 60, "window_unit": "second"}`, WHEN `RateLimitRule(**rule)` is constructed, THEN `pydantic.ValidationError` is raised mentioning field `limit`.

- **RLV-2:** `RateLimitRule.window_unit` MUST be one of `WindowUnit.second`,
  `WindowUnit.minute`, `WindowUnit.hour`. Any other string value raises
  `pydantic.ValidationError` at construction time — no case-insensitive matching,
  no aliasing (`"secs"`, `"s"` are rejected).
  - Why: Silent unit aliasing is how a `window: 5, window_unit: "s"` rule quietly
    becomes a 5-hour window instead of 5 seconds. Reject anything not in the enum.
  - **Scenario:** GIVEN a rule dict with `"window_unit": "seconds"` (plural, not in the enum), WHEN `RateLimitRule(**rule)` is constructed, THEN `pydantic.ValidationError` is raised mentioning field `window_unit`.

- **RLV-3:** `load_policy(path)` MUST raise `PolicyLoadError` with the message
  `"failed to parse YAML: <path>"` when the file exists but contains malformed YAML.
  The underlying `yaml.YAMLError` MUST NOT propagate directly.
  - Why: The gateway's startup code catches `PolicyLoadError` specifically to produce
    a clean "refusing to start: bad rate-limit config" message. Letting a raw
    `yaml.YAMLError` through breaks that error handling and dumps a parser traceback
    on operators instead.
  - **Scenario:** GIVEN a file containing `route: [unclosed`, WHEN `load_policy(path)` is called, THEN `PolicyLoadError` is raised with message `"failed to parse YAML: <path>"`.

- **RLV-4:** `load_policy(path)` MUST raise `FileNotFoundError` (not `PolicyLoadError`)
  when `path` does not exist on disk.
  - Why: Missing file and malformed file are different failure classes for an
    operator — missing file usually means a deploy/mount problem, malformed file
    means a config authoring problem. Callers need to tell them apart.
  - **Scenario:** GIVEN `path = "/etc/gateway/does-not-exist.yaml"`, WHEN `load_policy(path)` is called, THEN `FileNotFoundError` is raised.

- **RLV-5:** `load_policy(path)` MUST raise `PolicyValidationError` collecting ALL
  semantic validation failures (not just the first) when the parsed rules contain
  overlapping routes (see RLV-6) or duplicate `(route, method)` pairs with identical
  limits. `PolicyValidationError.errors` is a `list[str]`, one entry per problem
  found, each in the form `"<route> <method>: <reason>"`.
  - Why: Config authors iterate faster when they see every problem in one pass
    instead of fixing one error, rerunning, hitting the next error.
  - **Scenario:** GIVEN a policy YAML with two rules both matching `POST /api/orders` at different limits, and a third rule with `window_unit: "hour"` and `window: 0`, WHEN `load_policy(path)` is called, THEN `PolicyValidationError` is raised with `len(errors) == 2`.

- **RLV-6:** `find_overlapping_rules(rules)` MUST return every pair of rules that
  share the same `route` and `method` but have a different `limit`, `window`, or
  `window_unit`. Rules with identical `(route, method)` AND identical
  `(limit, window, window_unit)` are NOT considered overlapping (harmless
  duplication, not a conflict) — implementation note: dedupe on the full tuple
  before pairing, not just `(route, method)`.
  - Why: Two rules for the same route+method with different limits is ambiguous —
    which one applies at request time is undefined. Two identical rules are just a
    copy-paste no-op and shouldn't block a deploy.
  - **Scenario:** GIVEN rules `[{route: "/x", method: "GET", limit: 10, window: 60, window_unit: "second"}, {route: "/x", method: "GET", limit: 20, window: 60, window_unit: "second"}]`, WHEN `find_overlapping_rules(rules)` is called, THEN it returns one pair containing both rules.

## Pattern Table (parametrize matrix)

| `window_unit` input | Valid? | Why |
|---|---|---|
| `"second"` | ✓ | matches `WindowUnit.second` |
| `"minute"` | ✓ | matches `WindowUnit.minute` |
| `"hour"` | ✓ | matches `WindowUnit.hour` |
| `"seconds"` | ✗ | plural not aliased (RLV-2) |
| `"SECOND"` | ✗ | no case-insensitive matching (RLV-2) |
| `"s"` | ✗ | no abbreviation aliasing (RLV-2) |
| `"day"` | ✗ | not a supported unit |
| `""` | ✗ | empty string is not a valid enum member |

## Scenarios

### Happy path: valid policy loads cleanly
**GIVEN** a policy YAML with three non-overlapping rules, all fields valid.
**WHEN** `load_policy(path)` is called.
**THEN** it returns a `RateLimitPolicy` with `len(policy.rules) == 3` and no exception
is raised.

### Atomic failure: partial policy never returned on validation error
**GIVEN** a policy YAML where rule 1 is valid and rule 2 has `limit: -5`.
**WHEN** `load_policy(path)` is called.
**THEN** `pydantic.ValidationError` propagates from rule 2's construction and
`load_policy` does not return a partially-built `RateLimitPolicy` — the caller gets
either a complete, fully-valid policy or an exception, never a partial object.