docs(airouter): 2026-05-08 dogfood batch + test design rules

Three additions to agent-repos.md based on the post-A1-incident dogfood
batch (8 successes + 1 operator-induced "failure"):

1. Airouter Qwen3.6 section: pattern reconfirmed across M16 Wave A1/A2/B1
   and M25 Waves A1-A5 + B1-B2. Time-to-success bands recorded for cost
   calibration (1m30s for git rm; ~5 min for Pydantic regex; ~12 min for
   class addition). Default --max-test-iterations 1 for cheap probes.

2. New section: Test Design for AI Agent Dogfood Pipelines. Triggered by
   the M16 Wave B2 (MN-4 prompt cap) failure — a 14-minute airouter run
   blamed on the agent that was actually an over-strict test asserting on
   sanitised 422 body content. CP's RequestValidationError handler strips
   Pydantic detail for security; tests asserting body content for that
   path are structurally impossible. Rules: verify test passes against a
   reference impl before pushing; status-code-only ceiling for validator-
   driven 422s; model on previous successes not stricter variants; F70
   retries don't recover structurally impossible tests.

3. New section: Dogfood Failure Path: Branch + Logs Lost. When all F70
   retries exhaust, the agent's last attempt is not pushed to the agents
   fork, the CP task record's logs field is empty, and the pod is gone.
   Operator must reproduce locally — until F70 finalize-on-failure pushes
   the failed branch.

BESTPRACTICES.md index updated to reflect the new sub-topics.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Paul O'Reilly
2026-05-08 19:50:45 +12:00
parent 7bfabceaac
commit 52cd0a3cb6
2 changed files with 34 additions and 1 deletions

View File

@@ -28,7 +28,7 @@ Generalised best practices extracted from real project work via the `/distill-be
- [Octopus Process Templates](octopus-process-templates.md) — OCL syntax, step template references, channel scoping, parameters, versioning, Platform Hub patterns
- [LLM Code Security](llm-code-security.md) — Security vulnerabilities in AI-generated code: injection flaws, hardcoded secrets, hallucinated packages, over-permissive defaults, IaC risks, crypto mistakes, operational vulnerabilities (idempotency, CI/CD integrity, supply chain provenance, concurrent access), review checklists
- [CI Container Builds](ci-container-builds.md) — Registry cache with inline metadata, buildx in DinD, layer ordering, pip caching, path filter gotchas, SHA tagging strategy, runtime-mounted directory triggers
- [Agent Repos & Container Agents](agent-repos.md) — Task submission, harnesses, monitoring, multi-model workflows, agent repo forks, workspace layout, artifact passing via git branches, read-only test protection, infrastructure failure modes, cost-effective model scope boundaries
- [Agent Repos & Container Agents](agent-repos.md) — Task submission, harnesses, monitoring, multi-model workflows, agent repo forks, workspace layout, artifact passing via git branches, read-only test protection, infrastructure failure modes, cost-effective model scope boundaries, airouter dogfood patterns + time-to-success bands, dogfood test design (status-code-only ceiling for sanitised 422s), failure-path branch+log loss
- [AI Parallel Agents](ai-parallel-agents.md) — Parallel agent orchestration: multi-facet research dispatch, file contention, WebFetch limits, narrow reads, dataset-wide audits
- [Python Patterns](python-patterns.md) — Non-reentrant Lock deadlocks, Pydantic v2 extra='ignore' silent drops, subprocess routing callables for mocking, model_validator for cross-field validation
- [Mechanical Test Generation](mechanical-test-generation.md) — Spec properties that enable automated test writing: module layout tables, integration boundary marking, explicit library semantics, concrete interfaces over "implementation detail", error messages as test data, pattern tables as parametric matrices, pre-dispatch testability review