docs(airouter): correct failure-path picture + F70 vs local divergence

The previous "Dogfood Failure Path: Branch + Logs Lost" section was wrong
on two counts (verified by M16 Wave B3, 2026-05-08):

1. The agent's branch IS pushed on F70-failure (dispatcher logs "branch
   will still push (partial work preserved)"). The earlier "no branch"
   claim was a fetch-refspec mistake — both B2 and B3 have task-<id>
   branches on the agents fork.

2. The CP task record's logs ARE populated (~50 KB on B3). What's actually
   missing is the pytest stdout/stderr from the F70 invocation — only the
   high-level "Tests failed" line is logged.

Section retitled "What's Visible, What Isn't" with the corrected picture.
Operator pattern updated: fetch the task branch, apply the diff locally,
run the test — if local passes, cherry-pick to main.

New section "F70 Pytest Can Disagree with Local Pytest" captures the B3
finding: airouter's 15-line MN-14 validator passed 5/5 locally but F70
reported failure twice. Possible causes listed; workaround is
--max-test-iterations 1 + local apply-and-run after failure.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Paul O'Reilly
2026-05-08 20:27:41 +12:00
parent 52cd0a3cb6
commit bd3a99e2ad

View File

@@ -519,14 +519,38 @@ When a CLI tool or web framework returns sanitised error responses for security
3. **Model new dogfood tests on previously-successful tests, not stricter variants.** If an existing test for a similar requirement only checks status code, the new one should too — unless you've actively confirmed the framework leaks the structure you want to assert on. 3. **Model new dogfood tests on previously-successful tests, not stricter variants.** If an existing test for a similar requirement only checks status code, the new one should too — unless you've actively confirmed the framework leaks the structure you want to assert on.
4. **F70 retries don't help when the test is fundamentally wrong.** The agent has no signal to learn from a structural impossibility. The retry mechanism assumes the test is correct and the agent's output is iteratively improvable; that assumption breaks for operator-side test bugs. 4. **F70 retries don't help when the test is fundamentally wrong.** The agent has no signal to learn from a structural impossibility. The retry mechanism assumes the test is correct and the agent's output is iteratively improvable; that assumption breaks for operator-side test bugs.
## Dogfood Failure Path: Branch + Logs Lost ## Dogfood Failure Path: What's Visible, What Isn't
When an agent task fails after exhausting `--max-test-iterations` retries: When an agent task fails after exhausting `--max-test-iterations` retries (verified against M16 Wave B2 + B3, 2026-05-08):
- The agent's last attempt is **not pushed** to the agent-repo fork (`finalize.sh` apparently gates on agent_exit_code or F70 pass)
- The CP task record's `logs` field is empty (`include_logs=true` returns nothing)
- The agent pod is gone (K8s Job cleanup deletes it)
- Only the dispatcher's high-level signal remains: `state=failed exit_code=1`
This means a 14-minute compute spend produces zero post-hoc evidence of what the agent did or why it failed. The only way to debug is to reproduce the test locally against a reference implementation or re-dispatch with extra instrumentation. **Visible:**
- The agent's last attempt **IS pushed** to the agent-repo fork as `task-<id>`. Dispatcher logs `branch will still push (partial work preserved)`. Diff with `git show origin/task-<id>` to see exactly what the agent wrote.
- The CP task record's `logs` field **IS populated** (~50 KB) when queried with `?include_logs=true`. Contains dispatcher state transitions, the agent's text streams, and tool-call summaries.
- Dispatcher pod logs persist while the pod is alive (current ReplicaSet). Cross-reference task ID for high-level state.
**Until F70 finalize-on-failure pushes the branch (with a `failed-attempt-N` suffix or via a separate ref):** treat every dogfood failure as opaque and verify the test independently before re-dispatching. Don't burn another 14 minutes on the same broken test. **NOT visible:**
- The agent pod itself (K8s Job cleanup deletes it within seconds of completion).
- The pytest stdout/stderr from F70's invocation. Only `Tests failed on attempt N/N` is logged. The agent's `run_command` calls show `result_bytes` counts but not contents — so we know F70 said "fail" but not which assertion or why.
**Operator pattern after a failed dispatch:**
1. `git fetch origin '+refs/heads/task-<id>:refs/remotes/origin/task-<id>'` against the agent-repo fork
2. `git show origin/task-<id> -- <target-file>` to see the agent's edit
3. Apply the diff locally with `git apply`, run the dogfood test against it
4. If it passes locally but failed in F70, the failure is **environmental** — see next section
5. Cherry-pick the agent's work to main; the agent earned the credit even if F70 mis-reported
**Until F70 captures pytest stdout/stderr** (action item: write to `/workspace/.agent-output/f70-attempt-<N>.log`), failed dispatches whose code passes locally are post-mortem black boxes. Spend the 30 seconds to apply-and-run locally before assuming the agent was wrong.
## F70 Pytest Can Disagree with Local Pytest
**Symptom (M16 Wave B3, task `a496efd2`, 2026-05-08):** airouter agent writes a 15-line model_validator that passes 5/5 dogfood tests locally; F70 reports `Tests failed on attempt 1/2` and `2/2`. Branch + agent's diff applied to a fresh local checkout of the same SHA → 5/5 pass.
The agent's code was correct. The pipeline reported failure. Wasted: 9 minutes of compute and 2 retries that all hit the same environmental disagreement.
**Possible causes (not yet narrowed):**
- `run-ci-tests.sh` runs from a different working dir than the agent edited
- Stale `__pycache__` from a prior attempt's Pydantic model
- Different Python interpreter / virtualenv state inside the container
- F70 pytest discovery uses different conftest paths
**Operator workaround:** `--max-test-iterations 1` for cheap probes. After failure, inspect the agent's branch and apply locally. If local passes, cherry-pick the agent's work to main and capture the local-vs-container divergence as a separate follow-up — don't punish a correct agent for an environmental glitch.