From 47d616996c389f7c8f8d7d6f5e2d7ecd53dfad80 Mon Sep 17 00:00:00 2001 From: Paul O'Reilly Date: Sun, 26 Apr 2026 13:19:43 +1200 Subject: [PATCH] feat: add mechanical-test-generation best practice Eight practices for writing specs that serve as direct input to automated test-generation agents, extracted from the M15 Mechanical Process Nodes milestone review: module layout tables, integration boundary marking, explicit library semantics, concrete interfaces over "implementation detail", error messages as test data, pattern tables as parametric matrices, scenario selection, and pre-dispatch testability assessment. Co-Authored-By: Claude Sonnet 4.6 --- BESTPRACTICES.md | 1 + mechanical-test-generation.md | 187 ++++++++++++++++++++++++++++++++++ 2 files changed, 188 insertions(+) create mode 100644 mechanical-test-generation.md diff --git a/BESTPRACTICES.md b/BESTPRACTICES.md index 83b279d..cea75ef 100644 --- a/BESTPRACTICES.md +++ b/BESTPRACTICES.md @@ -31,3 +31,4 @@ Generalised best practices extracted from real project work via the `/distill-be - [Agent Repos & Container Agents](agent-repos.md) — Task submission, harnesses, monitoring, multi-model workflows, agent repo forks, workspace layout, artifact passing via git branches, read-only test protection, infrastructure failure modes, cost-effective model scope boundaries - [AI Parallel Agents](ai-parallel-agents.md) — Parallel agent orchestration: multi-facet research dispatch, file contention, WebFetch limits, narrow reads, dataset-wide audits - [Python Patterns](python-patterns.md) — Non-reentrant Lock deadlocks, Pydantic v2 extra='ignore' silent drops, subprocess routing callables for mocking, model_validator for cross-field validation +- [Mechanical Test Generation](mechanical-test-generation.md) — Spec properties that enable automated test writing: module layout tables, integration boundary marking, explicit library semantics, concrete interfaces over "implementation detail", error messages as test data, pattern tables as parametric matrices, pre-dispatch testability review diff --git a/mechanical-test-generation.md b/mechanical-test-generation.md new file mode 100644 index 0000000..26adcb8 --- /dev/null +++ b/mechanical-test-generation.md @@ -0,0 +1,187 @@ +# Mechanical Test Generation from Specs + +Practices for writing specs that can be used as direct input to automated test-writing agents or deterministic test-generation pipelines — i.e., workflows where a script or LLM reads a spec and produces a test file with minimal human intervention. Extracted from the M15 Mechanical Process Nodes milestone review. + +## Core Principle: Specs Should Drive Tests Without Side-Channels + +A test-generation step should need only the spec file to produce correct import statements, assertion values, and mock boundaries. If the agent needs to read the PLAN, the codebase, or guess at module paths, the spec has gaps. Close them in the spec, not in the prompt. + +--- + +## 1. Include a Module Layout Table + +**Problem:** Specs name functions (`validate_image`, `_expand_script_node`) but not their module paths. A test generator writes `from ? import validate_image` and either guesses wrong or falls back to reading the implementation. + +**Practice:** Add a "Module Layout" section to every spec that introduces new code. Map each group of requirements to its module path and exported symbols. + +```markdown +## Module Layout + +| Module | Location | Key exports | +|---|---|---| +| Image policy | `controlplane/image_policy.py` | `ImagePattern`, `ImagePolicy`, `validate_image(image, policy) → bool` | +| Process loader | `controlplane/process_loader.py` | `ProcessTemplate`, `load_template(name, version)` | +``` + +Also list the corresponding test file for each module: + +```markdown +| Test file | Spec requirements covered | +|---|---| +| `tests/test_image_policy.py` | MP-C-1..MP-C-6 | +| `tests/test_process_loader.py` | MP-A-1..MP-A-11 | +``` + +**Why:** Without this, test generators either infer the wrong path or require the PLAN as a second input, which creates coupling between planning and testing artifacts. + +--- + +## 2. Mark the Integration Boundary Explicitly + +**Problem:** Some requirements (volume lifecycle, Docker container creation, K8s API calls) cannot be tested without a live backend. A test generator that doesn't know this boundary produces integration-class tests mixed with unit tests, breaking the standard `pytest` run. + +**Practice:** State in the spec which requirements require real backends and use the `@pytest.mark.integration` marker: + +```markdown +Tests requiring a live Docker daemon or K8s API MUST be marked +`@pytest.mark.integration` and are excluded from the standard `pytest` run. +Unit tests use mocks only. +``` + +Then, within requirements, call it out: + +```markdown +- **MP-E-2:** [...] On Docker: [requires Docker daemon — integration test]. + On K8s: [K8s Job init container — assert Job spec structure; unit-testable]. +``` + +**Why:** The integration boundary determines which tests a generator should write as `unittest.mock.MagicMock`-based unit tests and which it should scaffold as `@pytest.mark.integration`. Getting this wrong produces either false-passing unit tests (mocks that don't reflect reality) or a test suite that only runs in CI with full infrastructure. + +--- + +## 3. State Library Semantics Explicitly — Don't Just Name the Library + +**Problem:** `"Semver patterns parse as packaging.specifiers.SpecifierSet"` tells a test generator which library to use but not what it does with edge cases. A test generator doesn't know that `python:3.12-slim` never matches `python:>=3.10.0` because `3.12-slim` isn't a valid PEP 440 version. + +**Practice:** When a spec requirement depends on a library's behavior, state the behavior — don't assume the reader knows the library: + +```markdown +- **MP-C-2:** [...] if the image's tag cannot be parsed as a PEP 440 version + (e.g., `:slim`, `:alpine`, `:3.12-slim`) it never matches a semver-range + pattern — only exact-string or glob patterns can match it. +``` + +**Why:** This directly produces a test case: `assert not validate_image("python:3.12-slim", ImagePolicy(whitelist=[ImagePattern("python:>=3.10.0")]))`. Without the explicit statement, this edge case is invisible. + +**General rule:** For any external library used in a requirement, add one sentence covering the non-obvious edge case (empty input, non-matching type, error path). This sentence is the test case. + +--- + +## 4. Replace "Implementation Detail" with a Concrete Interface + +**Problem:** `"The CP MUST query the dispatcher pool (implementation detail)"` is untestable. A test generator can't mock an interface it doesn't know exists. + +**Practice:** Every requirement must specify enough of the interface to write a mock. If the implementation is genuinely flexible, pick one concrete approach and say "implementation note: may be done via X or Y, but tests should use X": + +Before (untestable): +```markdown +- **MP-D-3:** The CP validates secrets at expansion time against the + dispatcher pool (implementation detail). +``` + +After (testable): +```markdown +- **MP-D-3:** `DispatcherRecord` gains `available_secrets: list[str] = []`. + At expansion time, CP computes `advertised = {s for d in active_dispatchers + for s in d.available_secrets}`. Tests: inject mock `DispatcherRecord` + objects with known secret sets and assert expansion succeeds or fails. +``` + +**Why:** "Implementation detail" in a spec is a deferred decision that the test generator can't resolve. Deferred decisions produce skipped or wrong tests. + +--- + +## 5. Make Error Messages First-Class Spec Artifacts + +**Problem:** A test generator can only write `pytest.raises(ValueError)` without knowing what message to match. That produces weak tests that pass on the wrong exception. + +**Practice:** Put exact error message templates in the requirement itself (not just the error-handling table): + +```markdown +- **MP-A-6:** [...] load-time error with the message + `"unsupported pre_action type for script step: (M15 supports only 'clone')"`. +``` + +And maintain a consolidated error-handling table as a second artifact: + +```markdown +| Condition | Error | Where raised | +|---|---|---| +| Pre-action type other than `clone` | `ValueError("unsupported pre_action type...")` | `process_loader` at load | +``` + +The test generator uses the requirement text for the test body and the table as a cross-reference index. Together they produce: + +```python +def test_mp_a6_unsupported_preaction_type(): + with pytest.raises(ValueError, match="unsupported pre_action type for script step: commit_pr"): + load_template(template_with_commit_pr_preaction) +``` + +**Why:** Exact message matching catches regressions where the exception type is right but the message has changed (e.g., a refactor that weakens the error context). Without message-level matching, tests let silent regressions through. + +--- + +## 6. Pattern Tables Are Parametric Test Matrices + +**Problem:** Specs list pattern syntax in a table (glob, semver, exact, digest). A test generator produces one test per form if it can't see the table as test data. + +**Practice:** Write pattern tables so they directly map to `@pytest.mark.parametrize` arguments. Include at least one positive and one negative example per form, plus the key edge case: + +```markdown +| Pattern form | Example image | Matches? | Why | +|---|---|---|---| +| Glob | `docker.io/*` | `docker.io/python:3.12.3` | ✓ | +| Glob | `docker.io/*` | `ghcr.io/python:3.12` | ✗ | +| Semver | `python:>=3.10.0,<4.0.0` | `python:3.12.3` | ✓ | +| Semver | `python:>=3.10.0,<4.0.0` | `python:3.12-slim` | ✗ non-PEP-440 | +| Exact | `python:3.12.3` | `python:3.12.3` | ✓ | +| Exact | `python:3.12.3` | `python:3.12` | ✗ | +| Digest | `python@sha256:abc` | `python@sha256:abc` | ✓ | +| Digest | `python@sha256:abc` | `python@sha256:def` | ✗ | +``` + +**Why:** This table is the test matrix. A test generator reads it directly into `@pytest.mark.parametrize`. Without it, edge cases (non-PEP-440 semver, digest-with-tag) are typically missed. + +--- + +## 7. Scenarios Are the Highest-Value Input — Write Them Last + +Scenarios (GIVEN/WHEN/THEN) are the test generator's most direct input — they map 1:1 to Arrange/Act/Assert. But they're only as useful as the requirements they exercise. Write scenarios after finalising requirements, picking the cases that cross subsystem boundaries or combine multiple requirements: + +- Valid end-to-end path (happy path) +- Atomic failure (no partial state created) +- Security rejection (path traversal, sandbox violation, policy denial) +- State machine edge (cancellation mid-run, timeout) +- Idempotency (ensure_pvc called twice, delete_pvc on 404) + +Avoid scenarios that duplicate a single-requirement unit test. Reserve them for interactions. + +--- + +## 8. Assess Testability Before Dispatching a Test Writer + +Before dispatching a `test-writer` agent (or running a mechanical test-generation pipeline), do a module-by-module testability review: + +| Module characteristic | Testability | Approach | +|---|---|---| +| Pure function (no I/O, no state) | High | Parametric unit tests direct from spec | +| Pydantic model with validators | High | One test per validator branch | +| External library with defined behavior | High (if spec states semantics) | Parametric from pattern/enum tables | +| CP/store interaction (mocked) | Medium | Inject mock store; assert task creation | +| Dispatcher with concrete interface (mocked) | Medium | Mock backend; assert call args | +| Real Docker container (clone, run) | Low — integration boundary | `@pytest.mark.integration` only | +| K8s API (PVC, Job, Secret) | Low — integration boundary | Mock k8s client for unit; real cluster for integration | +| Multi-step state machine | Low | End-to-end test; needs full dispatcher mock | + +The review catches spec gaps (untestable "implementation detail") before a test writer spends tokens on them.