diff --git a/agent-repos.md b/agent-repos.md index bc4e808..6d5892c 100644 --- a/agent-repos.md +++ b/agent-repos.md @@ -519,14 +519,38 @@ When a CLI tool or web framework returns sanitised error responses for security 3. **Model new dogfood tests on previously-successful tests, not stricter variants.** If an existing test for a similar requirement only checks status code, the new one should too — unless you've actively confirmed the framework leaks the structure you want to assert on. 4. **F70 retries don't help when the test is fundamentally wrong.** The agent has no signal to learn from a structural impossibility. The retry mechanism assumes the test is correct and the agent's output is iteratively improvable; that assumption breaks for operator-side test bugs. -## Dogfood Failure Path: Branch + Logs Lost +## Dogfood Failure Path: What's Visible, What Isn't -When an agent task fails after exhausting `--max-test-iterations` retries: -- The agent's last attempt is **not pushed** to the agent-repo fork (`finalize.sh` apparently gates on agent_exit_code or F70 pass) -- The CP task record's `logs` field is empty (`include_logs=true` returns nothing) -- The agent pod is gone (K8s Job cleanup deletes it) -- Only the dispatcher's high-level signal remains: `state=failed exit_code=1` +When an agent task fails after exhausting `--max-test-iterations` retries (verified against M16 Wave B2 + B3, 2026-05-08): -This means a 14-minute compute spend produces zero post-hoc evidence of what the agent did or why it failed. The only way to debug is to reproduce the test locally against a reference implementation or re-dispatch with extra instrumentation. +**Visible:** +- The agent's last attempt **IS pushed** to the agent-repo fork as `task-`. Dispatcher logs `branch will still push (partial work preserved)`. Diff with `git show origin/task-` to see exactly what the agent wrote. +- The CP task record's `logs` field **IS populated** (~50 KB) when queried with `?include_logs=true`. Contains dispatcher state transitions, the agent's text streams, and tool-call summaries. +- Dispatcher pod logs persist while the pod is alive (current ReplicaSet). Cross-reference task ID for high-level state. -**Until F70 finalize-on-failure pushes the branch (with a `failed-attempt-N` suffix or via a separate ref):** treat every dogfood failure as opaque and verify the test independently before re-dispatching. Don't burn another 14 minutes on the same broken test. +**NOT visible:** +- The agent pod itself (K8s Job cleanup deletes it within seconds of completion). +- The pytest stdout/stderr from F70's invocation. Only `Tests failed on attempt N/N` is logged. The agent's `run_command` calls show `result_bytes` counts but not contents — so we know F70 said "fail" but not which assertion or why. + +**Operator pattern after a failed dispatch:** +1. `git fetch origin '+refs/heads/task-:refs/remotes/origin/task-'` against the agent-repo fork +2. `git show origin/task- -- ` to see the agent's edit +3. Apply the diff locally with `git apply`, run the dogfood test against it +4. If it passes locally but failed in F70, the failure is **environmental** — see next section +5. Cherry-pick the agent's work to main; the agent earned the credit even if F70 mis-reported + +**Until F70 captures pytest stdout/stderr** (action item: write to `/workspace/.agent-output/f70-attempt-.log`), failed dispatches whose code passes locally are post-mortem black boxes. Spend the 30 seconds to apply-and-run locally before assuming the agent was wrong. + +## F70 Pytest Can Disagree with Local Pytest + +**Symptom (M16 Wave B3, task `a496efd2`, 2026-05-08):** airouter agent writes a 15-line model_validator that passes 5/5 dogfood tests locally; F70 reports `Tests failed on attempt 1/2` and `2/2`. Branch + agent's diff applied to a fresh local checkout of the same SHA → 5/5 pass. + +The agent's code was correct. The pipeline reported failure. Wasted: 9 minutes of compute and 2 retries that all hit the same environmental disagreement. + +**Possible causes (not yet narrowed):** +- `run-ci-tests.sh` runs from a different working dir than the agent edited +- Stale `__pycache__` from a prior attempt's Pydantic model +- Different Python interpreter / virtualenv state inside the container +- F70 pytest discovery uses different conftest paths + +**Operator workaround:** `--max-test-iterations 1` for cheap probes. After failure, inspect the agent's branch and apply locally. If local passes, cherry-pick the agent's work to main and capture the local-vs-container divergence as a separate follow-up — don't punish a correct agent for an environmental glitch.