docs(airouter): 2026-05-08 dogfood batch + test design rules
Three additions to agent-repos.md based on the post-A1-incident dogfood batch (8 successes + 1 operator-induced "failure"): 1. Airouter Qwen3.6 section: pattern reconfirmed across M16 Wave A1/A2/B1 and M25 Waves A1-A5 + B1-B2. Time-to-success bands recorded for cost calibration (1m30s for git rm; ~5 min for Pydantic regex; ~12 min for class addition). Default --max-test-iterations 1 for cheap probes. 2. New section: Test Design for AI Agent Dogfood Pipelines. Triggered by the M16 Wave B2 (MN-4 prompt cap) failure — a 14-minute airouter run blamed on the agent that was actually an over-strict test asserting on sanitised 422 body content. CP's RequestValidationError handler strips Pydantic detail for security; tests asserting body content for that path are structurally impossible. Rules: verify test passes against a reference impl before pushing; status-code-only ceiling for validator- driven 422s; model on previous successes not stricter variants; F70 retries don't recover structurally impossible tests. 3. New section: Dogfood Failure Path: Branch + Logs Lost. When all F70 retries exhaust, the agent's last attempt is not pushed to the agents fork, the CP task record's logs field is empty, and the pod is gone. Operator must reproduce locally — until F70 finalize-on-failure pushes the failed branch. BESTPRACTICES.md index updated to reflect the new sub-topics. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
@@ -28,7 +28,7 @@ Generalised best practices extracted from real project work via the `/distill-be
|
||||
- [Octopus Process Templates](octopus-process-templates.md) — OCL syntax, step template references, channel scoping, parameters, versioning, Platform Hub patterns
|
||||
- [LLM Code Security](llm-code-security.md) — Security vulnerabilities in AI-generated code: injection flaws, hardcoded secrets, hallucinated packages, over-permissive defaults, IaC risks, crypto mistakes, operational vulnerabilities (idempotency, CI/CD integrity, supply chain provenance, concurrent access), review checklists
|
||||
- [CI Container Builds](ci-container-builds.md) — Registry cache with inline metadata, buildx in DinD, layer ordering, pip caching, path filter gotchas, SHA tagging strategy, runtime-mounted directory triggers
|
||||
- [Agent Repos & Container Agents](agent-repos.md) — Task submission, harnesses, monitoring, multi-model workflows, agent repo forks, workspace layout, artifact passing via git branches, read-only test protection, infrastructure failure modes, cost-effective model scope boundaries
|
||||
- [Agent Repos & Container Agents](agent-repos.md) — Task submission, harnesses, monitoring, multi-model workflows, agent repo forks, workspace layout, artifact passing via git branches, read-only test protection, infrastructure failure modes, cost-effective model scope boundaries, airouter dogfood patterns + time-to-success bands, dogfood test design (status-code-only ceiling for sanitised 422s), failure-path branch+log loss
|
||||
- [AI Parallel Agents](ai-parallel-agents.md) — Parallel agent orchestration: multi-facet research dispatch, file contention, WebFetch limits, narrow reads, dataset-wide audits
|
||||
- [Python Patterns](python-patterns.md) — Non-reentrant Lock deadlocks, Pydantic v2 extra='ignore' silent drops, subprocess routing callables for mocking, model_validator for cross-field validation
|
||||
- [Mechanical Test Generation](mechanical-test-generation.md) — Spec properties that enable automated test writing: module layout tables, integration boundary marking, explicit library semantics, concrete interfaces over "implementation detail", error messages as test data, pattern tables as parametric matrices, pre-dispatch testability review
|
||||
|
||||
Reference in New Issue
Block a user