diff --git a/BESTPRACTICES.md b/BESTPRACTICES.md index cea75ef..1c43631 100644 --- a/BESTPRACTICES.md +++ b/BESTPRACTICES.md @@ -28,7 +28,7 @@ Generalised best practices extracted from real project work via the `/distill-be - [Octopus Process Templates](octopus-process-templates.md) — OCL syntax, step template references, channel scoping, parameters, versioning, Platform Hub patterns - [LLM Code Security](llm-code-security.md) — Security vulnerabilities in AI-generated code: injection flaws, hardcoded secrets, hallucinated packages, over-permissive defaults, IaC risks, crypto mistakes, operational vulnerabilities (idempotency, CI/CD integrity, supply chain provenance, concurrent access), review checklists - [CI Container Builds](ci-container-builds.md) — Registry cache with inline metadata, buildx in DinD, layer ordering, pip caching, path filter gotchas, SHA tagging strategy, runtime-mounted directory triggers -- [Agent Repos & Container Agents](agent-repos.md) — Task submission, harnesses, monitoring, multi-model workflows, agent repo forks, workspace layout, artifact passing via git branches, read-only test protection, infrastructure failure modes, cost-effective model scope boundaries +- [Agent Repos & Container Agents](agent-repos.md) — Task submission, harnesses, monitoring, multi-model workflows, agent repo forks, workspace layout, artifact passing via git branches, read-only test protection, infrastructure failure modes, cost-effective model scope boundaries, airouter dogfood patterns + time-to-success bands, dogfood test design (status-code-only ceiling for sanitised 422s), failure-path branch+log loss - [AI Parallel Agents](ai-parallel-agents.md) — Parallel agent orchestration: multi-facet research dispatch, file contention, WebFetch limits, narrow reads, dataset-wide audits - [Python Patterns](python-patterns.md) — Non-reentrant Lock deadlocks, Pydantic v2 extra='ignore' silent drops, subprocess routing callables for mocking, model_validator for cross-field validation - [Mechanical Test Generation](mechanical-test-generation.md) — Spec properties that enable automated test writing: module layout tables, integration boundary marking, explicit library semantics, concrete interfaces over "implementation detail", error messages as test data, pattern tables as parametric matrices, pre-dispatch testability review diff --git a/agent-repos.md b/agent-repos.md index ff60037..bc4e808 100644 --- a/agent-repos.md +++ b/agent-repos.md @@ -497,3 +497,36 @@ Single-file scope is the reliable unit for airouter/Qwen3.6 dispatches. If a multi-file task is unavoidable, use a **sequential chain**: dispatch task-1 to create file-A, wait for success, then dispatch task-2 to create file-B referencing task-1's output. This bounds the agent's scope to one artifact at a time. **BUG-21** (2026-05-04): D4 task (`017f63cb`) exited 0, showed correct understanding in logs, but produced no harness files. Fix confirmed: decompose airouter tasks to single-file units (D4a + D4b both succeeded). + +**Pattern reconfirmed (2026-05-08, batch of 9):** After the Wave A1 destructive-`Write` incident (task `4a2f2988`) and 12 pipeline fixes, the airouter dogfood pipeline produced 8 successful dispatches in 24 hours plus 1 operator-induced failure: M16 Wave A1 + A2 (validator + class additions to `controlplane/api/identity_deps.py`), M16 Wave B1 (Pydantic `model_validator` regex check), M25 Waves A1-A3, A5, A4 + B1, B2 (seven single-file `git rm` deletions). The single agent "failure" (Wave B2, MN-4 prompt cap) was an operator-side test bug, not an agent regression — see "Test Design for AI Agent Dogfood Pipelines" below. + +**Time-to-success bands** (useful for cost calibration): +- 1-line `git rm` deletion: ~1m30s, ~6 turns (fastest: M25-A4 at 1m37s) +- Add a 5-line Pydantic regex validator: ~5 min, ~30 turns (M16-B1 / MN-5 grammar) +- Add a 13-line class definition: ~12 min, ~60 turns (M16-A2 / RFC9457Problem) + +Tasks running beyond 12 min / 60 turns suggest the agent is over-cautious or stuck. **Default `--max-test-iterations 1` for cheap probes**; reserve 3+ only for tasks where the agent might genuinely benefit from the failure context (i.e., the test is correct and the agent may need a second look at its own output). For tasks that fail on a structurally impossible test, all retries fail identically — see the next section. + +## Test Design for AI Agent Dogfood Pipelines + +When a CLI tool or web framework returns sanitised error responses for security reasons, dogfood tests must respect that contract. A test that asserts on response body content for an error path the framework deliberately strips will block the agent indefinitely — the validator can be perfectly correct and the test still fails. + +**Concrete incident (2026-05-08, task `26717643`):** A 16 KB prompt-cap dogfood test asserted both `response.status_code == 422` AND that the body mentioned "prompt", "exceeds", or "16384". The agent (airouter / Qwen3.6) wrote three correct implementations across F70 retries; each was rejected because the CP's `RequestValidationError` handler sanitises 422 bodies to `{"detail":"Request validation failed","correlation_id":...}` — Pydantic's verbose detail is logged server-side but never leaked to the client. 14 minutes of compute spent on a structurally impossible test. + +**Rules:** +1. **Verify the dogfood test passes against a hand-coded reference implementation BEFORE pushing it.** A red-on-broken-test loop is invisible to the agent; only operator-side validation catches it. The 30-second cost of running pytest locally beats the 14-minute cost of a doomed dispatch. +2. **For validator-driven 422s in security-conscious frameworks, status-code-only assertions are the ceiling.** Add structural body assertions only for 200/201 paths or for handlers that write their own response body. +3. **Model new dogfood tests on previously-successful tests, not stricter variants.** If an existing test for a similar requirement only checks status code, the new one should too — unless you've actively confirmed the framework leaks the structure you want to assert on. +4. **F70 retries don't help when the test is fundamentally wrong.** The agent has no signal to learn from a structural impossibility. The retry mechanism assumes the test is correct and the agent's output is iteratively improvable; that assumption breaks for operator-side test bugs. + +## Dogfood Failure Path: Branch + Logs Lost + +When an agent task fails after exhausting `--max-test-iterations` retries: +- The agent's last attempt is **not pushed** to the agent-repo fork (`finalize.sh` apparently gates on agent_exit_code or F70 pass) +- The CP task record's `logs` field is empty (`include_logs=true` returns nothing) +- The agent pod is gone (K8s Job cleanup deletes it) +- Only the dispatcher's high-level signal remains: `state=failed exit_code=1` + +This means a 14-minute compute spend produces zero post-hoc evidence of what the agent did or why it failed. The only way to debug is to reproduce the test locally against a reference implementation or re-dispatch with extra instrumentation. + +**Until F70 finalize-on-failure pushes the branch (with a `failed-attempt-N` suffix or via a separate ref):** treat every dogfood failure as opaque and verify the test independently before re-dispatching. Don't burn another 14 minutes on the same broken test.