docs(airouter): correct failure-path picture + F70 vs local divergence
The previous "Dogfood Failure Path: Branch + Logs Lost" section was wrong on two counts (verified by M16 Wave B3, 2026-05-08): 1. The agent's branch IS pushed on F70-failure (dispatcher logs "branch will still push (partial work preserved)"). The earlier "no branch" claim was a fetch-refspec mistake — both B2 and B3 have task-<id> branches on the agents fork. 2. The CP task record's logs ARE populated (~50 KB on B3). What's actually missing is the pytest stdout/stderr from the F70 invocation — only the high-level "Tests failed" line is logged. Section retitled "What's Visible, What Isn't" with the corrected picture. Operator pattern updated: fetch the task branch, apply the diff locally, run the test — if local passes, cherry-pick to main. New section "F70 Pytest Can Disagree with Local Pytest" captures the B3 finding: airouter's 15-line MN-14 validator passed 5/5 locally but F70 reported failure twice. Possible causes listed; workaround is --max-test-iterations 1 + local apply-and-run after failure. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
@@ -519,14 +519,38 @@ When a CLI tool or web framework returns sanitised error responses for security
|
|||||||
3. **Model new dogfood tests on previously-successful tests, not stricter variants.** If an existing test for a similar requirement only checks status code, the new one should too — unless you've actively confirmed the framework leaks the structure you want to assert on.
|
3. **Model new dogfood tests on previously-successful tests, not stricter variants.** If an existing test for a similar requirement only checks status code, the new one should too — unless you've actively confirmed the framework leaks the structure you want to assert on.
|
||||||
4. **F70 retries don't help when the test is fundamentally wrong.** The agent has no signal to learn from a structural impossibility. The retry mechanism assumes the test is correct and the agent's output is iteratively improvable; that assumption breaks for operator-side test bugs.
|
4. **F70 retries don't help when the test is fundamentally wrong.** The agent has no signal to learn from a structural impossibility. The retry mechanism assumes the test is correct and the agent's output is iteratively improvable; that assumption breaks for operator-side test bugs.
|
||||||
|
|
||||||
## Dogfood Failure Path: Branch + Logs Lost
|
## Dogfood Failure Path: What's Visible, What Isn't
|
||||||
|
|
||||||
When an agent task fails after exhausting `--max-test-iterations` retries:
|
When an agent task fails after exhausting `--max-test-iterations` retries (verified against M16 Wave B2 + B3, 2026-05-08):
|
||||||
- The agent's last attempt is **not pushed** to the agent-repo fork (`finalize.sh` apparently gates on agent_exit_code or F70 pass)
|
|
||||||
- The CP task record's `logs` field is empty (`include_logs=true` returns nothing)
|
|
||||||
- The agent pod is gone (K8s Job cleanup deletes it)
|
|
||||||
- Only the dispatcher's high-level signal remains: `state=failed exit_code=1`
|
|
||||||
|
|
||||||
This means a 14-minute compute spend produces zero post-hoc evidence of what the agent did or why it failed. The only way to debug is to reproduce the test locally against a reference implementation or re-dispatch with extra instrumentation.
|
**Visible:**
|
||||||
|
- The agent's last attempt **IS pushed** to the agent-repo fork as `task-<id>`. Dispatcher logs `branch will still push (partial work preserved)`. Diff with `git show origin/task-<id>` to see exactly what the agent wrote.
|
||||||
|
- The CP task record's `logs` field **IS populated** (~50 KB) when queried with `?include_logs=true`. Contains dispatcher state transitions, the agent's text streams, and tool-call summaries.
|
||||||
|
- Dispatcher pod logs persist while the pod is alive (current ReplicaSet). Cross-reference task ID for high-level state.
|
||||||
|
|
||||||
**Until F70 finalize-on-failure pushes the branch (with a `failed-attempt-N` suffix or via a separate ref):** treat every dogfood failure as opaque and verify the test independently before re-dispatching. Don't burn another 14 minutes on the same broken test.
|
**NOT visible:**
|
||||||
|
- The agent pod itself (K8s Job cleanup deletes it within seconds of completion).
|
||||||
|
- The pytest stdout/stderr from F70's invocation. Only `Tests failed on attempt N/N` is logged. The agent's `run_command` calls show `result_bytes` counts but not contents — so we know F70 said "fail" but not which assertion or why.
|
||||||
|
|
||||||
|
**Operator pattern after a failed dispatch:**
|
||||||
|
1. `git fetch origin '+refs/heads/task-<id>:refs/remotes/origin/task-<id>'` against the agent-repo fork
|
||||||
|
2. `git show origin/task-<id> -- <target-file>` to see the agent's edit
|
||||||
|
3. Apply the diff locally with `git apply`, run the dogfood test against it
|
||||||
|
4. If it passes locally but failed in F70, the failure is **environmental** — see next section
|
||||||
|
5. Cherry-pick the agent's work to main; the agent earned the credit even if F70 mis-reported
|
||||||
|
|
||||||
|
**Until F70 captures pytest stdout/stderr** (action item: write to `/workspace/.agent-output/f70-attempt-<N>.log`), failed dispatches whose code passes locally are post-mortem black boxes. Spend the 30 seconds to apply-and-run locally before assuming the agent was wrong.
|
||||||
|
|
||||||
|
## F70 Pytest Can Disagree with Local Pytest
|
||||||
|
|
||||||
|
**Symptom (M16 Wave B3, task `a496efd2`, 2026-05-08):** airouter agent writes a 15-line model_validator that passes 5/5 dogfood tests locally; F70 reports `Tests failed on attempt 1/2` and `2/2`. Branch + agent's diff applied to a fresh local checkout of the same SHA → 5/5 pass.
|
||||||
|
|
||||||
|
The agent's code was correct. The pipeline reported failure. Wasted: 9 minutes of compute and 2 retries that all hit the same environmental disagreement.
|
||||||
|
|
||||||
|
**Possible causes (not yet narrowed):**
|
||||||
|
- `run-ci-tests.sh` runs from a different working dir than the agent edited
|
||||||
|
- Stale `__pycache__` from a prior attempt's Pydantic model
|
||||||
|
- Different Python interpreter / virtualenv state inside the container
|
||||||
|
- F70 pytest discovery uses different conftest paths
|
||||||
|
|
||||||
|
**Operator workaround:** `--max-test-iterations 1` for cheap probes. After failure, inspect the agent's branch and apply locally. If local passes, cherry-pick the agent's work to main and capture the local-vs-container divergence as a separate follow-up — don't punish a correct agent for an environmental glitch.
|
||||||
|
|||||||
Reference in New Issue
Block a user