From 52cd0a3cb6c4acf5adf3c309f235efa429fb7467 Mon Sep 17 00:00:00 2001 From: Paul O'Reilly Date: Fri, 8 May 2026 19:50:45 +1200 Subject: [PATCH] docs(airouter): 2026-05-08 dogfood batch + test design rules MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three additions to agent-repos.md based on the post-A1-incident dogfood batch (8 successes + 1 operator-induced "failure"): 1. Airouter Qwen3.6 section: pattern reconfirmed across M16 Wave A1/A2/B1 and M25 Waves A1-A5 + B1-B2. Time-to-success bands recorded for cost calibration (1m30s for git rm; ~5 min for Pydantic regex; ~12 min for class addition). Default --max-test-iterations 1 for cheap probes. 2. New section: Test Design for AI Agent Dogfood Pipelines. Triggered by the M16 Wave B2 (MN-4 prompt cap) failure — a 14-minute airouter run blamed on the agent that was actually an over-strict test asserting on sanitised 422 body content. CP's RequestValidationError handler strips Pydantic detail for security; tests asserting body content for that path are structurally impossible. Rules: verify test passes against a reference impl before pushing; status-code-only ceiling for validator- driven 422s; model on previous successes not stricter variants; F70 retries don't recover structurally impossible tests. 3. New section: Dogfood Failure Path: Branch + Logs Lost. When all F70 retries exhaust, the agent's last attempt is not pushed to the agents fork, the CP task record's logs field is empty, and the pod is gone. Operator must reproduce locally — until F70 finalize-on-failure pushes the failed branch. BESTPRACTICES.md index updated to reflect the new sub-topics. Co-Authored-By: Claude Sonnet 4.6 --- BESTPRACTICES.md | 2 +- agent-repos.md | 33 +++++++++++++++++++++++++++++++++ 2 files changed, 34 insertions(+), 1 deletion(-) diff --git a/BESTPRACTICES.md b/BESTPRACTICES.md index cea75ef..1c43631 100644 --- a/BESTPRACTICES.md +++ b/BESTPRACTICES.md @@ -28,7 +28,7 @@ Generalised best practices extracted from real project work via the `/distill-be - [Octopus Process Templates](octopus-process-templates.md) — OCL syntax, step template references, channel scoping, parameters, versioning, Platform Hub patterns - [LLM Code Security](llm-code-security.md) — Security vulnerabilities in AI-generated code: injection flaws, hardcoded secrets, hallucinated packages, over-permissive defaults, IaC risks, crypto mistakes, operational vulnerabilities (idempotency, CI/CD integrity, supply chain provenance, concurrent access), review checklists - [CI Container Builds](ci-container-builds.md) — Registry cache with inline metadata, buildx in DinD, layer ordering, pip caching, path filter gotchas, SHA tagging strategy, runtime-mounted directory triggers -- [Agent Repos & Container Agents](agent-repos.md) — Task submission, harnesses, monitoring, multi-model workflows, agent repo forks, workspace layout, artifact passing via git branches, read-only test protection, infrastructure failure modes, cost-effective model scope boundaries +- [Agent Repos & Container Agents](agent-repos.md) — Task submission, harnesses, monitoring, multi-model workflows, agent repo forks, workspace layout, artifact passing via git branches, read-only test protection, infrastructure failure modes, cost-effective model scope boundaries, airouter dogfood patterns + time-to-success bands, dogfood test design (status-code-only ceiling for sanitised 422s), failure-path branch+log loss - [AI Parallel Agents](ai-parallel-agents.md) — Parallel agent orchestration: multi-facet research dispatch, file contention, WebFetch limits, narrow reads, dataset-wide audits - [Python Patterns](python-patterns.md) — Non-reentrant Lock deadlocks, Pydantic v2 extra='ignore' silent drops, subprocess routing callables for mocking, model_validator for cross-field validation - [Mechanical Test Generation](mechanical-test-generation.md) — Spec properties that enable automated test writing: module layout tables, integration boundary marking, explicit library semantics, concrete interfaces over "implementation detail", error messages as test data, pattern tables as parametric matrices, pre-dispatch testability review diff --git a/agent-repos.md b/agent-repos.md index ff60037..bc4e808 100644 --- a/agent-repos.md +++ b/agent-repos.md @@ -497,3 +497,36 @@ Single-file scope is the reliable unit for airouter/Qwen3.6 dispatches. If a multi-file task is unavoidable, use a **sequential chain**: dispatch task-1 to create file-A, wait for success, then dispatch task-2 to create file-B referencing task-1's output. This bounds the agent's scope to one artifact at a time. **BUG-21** (2026-05-04): D4 task (`017f63cb`) exited 0, showed correct understanding in logs, but produced no harness files. Fix confirmed: decompose airouter tasks to single-file units (D4a + D4b both succeeded). + +**Pattern reconfirmed (2026-05-08, batch of 9):** After the Wave A1 destructive-`Write` incident (task `4a2f2988`) and 12 pipeline fixes, the airouter dogfood pipeline produced 8 successful dispatches in 24 hours plus 1 operator-induced failure: M16 Wave A1 + A2 (validator + class additions to `controlplane/api/identity_deps.py`), M16 Wave B1 (Pydantic `model_validator` regex check), M25 Waves A1-A3, A5, A4 + B1, B2 (seven single-file `git rm` deletions). The single agent "failure" (Wave B2, MN-4 prompt cap) was an operator-side test bug, not an agent regression — see "Test Design for AI Agent Dogfood Pipelines" below. + +**Time-to-success bands** (useful for cost calibration): +- 1-line `git rm` deletion: ~1m30s, ~6 turns (fastest: M25-A4 at 1m37s) +- Add a 5-line Pydantic regex validator: ~5 min, ~30 turns (M16-B1 / MN-5 grammar) +- Add a 13-line class definition: ~12 min, ~60 turns (M16-A2 / RFC9457Problem) + +Tasks running beyond 12 min / 60 turns suggest the agent is over-cautious or stuck. **Default `--max-test-iterations 1` for cheap probes**; reserve 3+ only for tasks where the agent might genuinely benefit from the failure context (i.e., the test is correct and the agent may need a second look at its own output). For tasks that fail on a structurally impossible test, all retries fail identically — see the next section. + +## Test Design for AI Agent Dogfood Pipelines + +When a CLI tool or web framework returns sanitised error responses for security reasons, dogfood tests must respect that contract. A test that asserts on response body content for an error path the framework deliberately strips will block the agent indefinitely — the validator can be perfectly correct and the test still fails. + +**Concrete incident (2026-05-08, task `26717643`):** A 16 KB prompt-cap dogfood test asserted both `response.status_code == 422` AND that the body mentioned "prompt", "exceeds", or "16384". The agent (airouter / Qwen3.6) wrote three correct implementations across F70 retries; each was rejected because the CP's `RequestValidationError` handler sanitises 422 bodies to `{"detail":"Request validation failed","correlation_id":...}` — Pydantic's verbose detail is logged server-side but never leaked to the client. 14 minutes of compute spent on a structurally impossible test. + +**Rules:** +1. **Verify the dogfood test passes against a hand-coded reference implementation BEFORE pushing it.** A red-on-broken-test loop is invisible to the agent; only operator-side validation catches it. The 30-second cost of running pytest locally beats the 14-minute cost of a doomed dispatch. +2. **For validator-driven 422s in security-conscious frameworks, status-code-only assertions are the ceiling.** Add structural body assertions only for 200/201 paths or for handlers that write their own response body. +3. **Model new dogfood tests on previously-successful tests, not stricter variants.** If an existing test for a similar requirement only checks status code, the new one should too — unless you've actively confirmed the framework leaks the structure you want to assert on. +4. **F70 retries don't help when the test is fundamentally wrong.** The agent has no signal to learn from a structural impossibility. The retry mechanism assumes the test is correct and the agent's output is iteratively improvable; that assumption breaks for operator-side test bugs. + +## Dogfood Failure Path: Branch + Logs Lost + +When an agent task fails after exhausting `--max-test-iterations` retries: +- The agent's last attempt is **not pushed** to the agent-repo fork (`finalize.sh` apparently gates on agent_exit_code or F70 pass) +- The CP task record's `logs` field is empty (`include_logs=true` returns nothing) +- The agent pod is gone (K8s Job cleanup deletes it) +- Only the dispatcher's high-level signal remains: `state=failed exit_code=1` + +This means a 14-minute compute spend produces zero post-hoc evidence of what the agent did or why it failed. The only way to debug is to reproduce the test locally against a reference implementation or re-dispatch with extra instrumentation. + +**Until F70 finalize-on-failure pushes the branch (with a `failed-attempt-N` suffix or via a separate ref):** treat every dogfood failure as opaque and verify the test independently before re-dispatching. Don't burn another 14 minutes on the same broken test.