best-practices/agent-repos.md

# Agent Repos & Container Agent Operations

Best practices for running AI agents via the agent-runtimes control plane — submitting tasks, using agent repos for persistence, running multi-model workflows, and monitoring progress.

## Architecture Overview

The agent-runtimes system runs AI coding agents inside ephemeral Docker/K8s containers, orchestrated by a control plane (CP). The three-tier model:

1. **Control Plane** — central task queue, dispatcher registry, model routing. External systems (including Claude Code sessions) submit tasks here.
2. **Dispatchers** — poll the CP for tasks, resolve harness+model, create containers. All connections outbound (works behind firewalls).
3. **Agent Containers** — ephemeral, isolated. Receive a JSON payload, execute work, exit.

```
You (Claude Code session) → POST /tasks → Control Plane → Dispatcher → Agent Container
                           ← GET /tasks/{id} ← (poll for results)
```

### Access

| Environment | CP URL |
|---|---|
| K8s (port-forward) | `kubectl port-forward -n agent-runtimes svc/controlplane 8100:8100` then `http://localhost:8100` |
| K8s (MFA) | `https://agents.oreillyit.nz` (behind Authelia) |
| Local dev (Docker Compose) | `http://localhost:8100` after `docker compose up -d` |

---

## Submitting Tasks

### Minimal Task

```bash
curl -s -X POST http://localhost:8100/tasks \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Fix test_auth.py",
    "project_id": "my-project",
    "prompt": "Fix the failing test in test_auth.py",
    "harness": "planning-opus-repo/v1",
    "runtime": {"cli": "claude"},
    "pre_actions": [
      {"type": "clone", "repo": "git@gitea.oreillyit.nz-ai-enablement:skynet/my-project.git", "depth": 1}
    ]
  }'
```

### Full Task Payload

```json
{
  "task_id": "optional-uuid",
  "name": "Short name (shown in monitor)",
  "project_id": "groups tasks in monitor",
  "prompt": "The instruction for the agent",
  "harness": "composite-harness-name/v1",
  "runtime": {
    "cli": "claude",
    "model": "sonnet",
    "timeout": 1800
  },
  "priority": 0,
  "metadata": {
    "workflow": "my-workflow",
    "phase": "plan",
    "model": "opus"
  },
  "pre_actions": [
    {"type": "clone", "repo": "git@...", "branch": "main", "depth": 1}
  ],
  "on_success": [
    {"type": "commit_pr", "branch": "feature-branch", "title": "PR title", "base": "main"}
  ],
  "on_error": [
    {"type": "report", "webhook_url": "https://..."}
  ],
  "max_retries": 1
}
```

### Key Fields

| Field | Required | Description |
|---|---|---|
| `name` | Recommended | Short task name shown in the monitor |
| `project_id` | Recommended | Groups tasks in the monitor; enables filtering |
| `prompt` | Yes | The instruction sent to the agent |
| `harness` | Recommended | Composite harness (defines credentials, context, model provider) |
| `runtime.cli` | No | CLI runner: `claude` (default), `openai_compat`, `agentic` |
| `pre_actions` | No | Setup actions before agent runs (e.g., `clone`) |
| `on_success` | No | Post-agent actions on success (e.g., `commit_pr`, `report`) |
| `on_error` | No | Post-agent actions on failure |
| `metadata` | No | Arbitrary JSONB — used for workflow tracking, filtering |
| `priority` | No | -100 to 100 (higher = picked first, default 0) |

### Python Task Submission

For programmatic use from a Claude Code session:

```python
import requests, json, uuid

CP = "http://localhost:8100"
task_id = str(uuid.uuid4())

payload = {
    "task_id": task_id,
    "name": "My agent task",
    "project_id": "my-project",
    "prompt": "...",
    "harness": "planning-opus-repo/v1",
    "runtime": {"cli": "claude"},
    "pre_actions": [{"type": "clone", "repo": "git@gitea.oreillyit.nz-ai-enablement:skynet/my-project.git"}],
    "metadata": {"workflow": "my-workflow", "phase": "plan"}
}

resp = requests.post(f"{CP}/tasks", json=payload, timeout=10)
data = resp.json()
print(f"Submitted: {data['task_id']}")
```

---

## Available Harnesses

Harnesses define what credentials, context, and model provider an agent gets. Composites combine multiple layers.

### Composite Harnesses (ready to use)

| Harness | Model Provider | Capabilities |
|---|---|---|
| `planning-opus-repo/v1` | Anthropic Claude (subscription) | Planning context + SSH clone |
| `planning-minimax-repo/v1` | MiniMax | Planning context + SSH clone |
| `python-code-review/v1` | Default (Anthropic) | Python dev tools + Gitea admin |

### Context Layers (building blocks)

| Layer | What it provides |
|---|---|
| `planning/v1` | Best-practices files for spec/plan writing |
| `gitea-ssh/v1` | SSH key for clone/push to Gitea |
| `gitea-admin/v1` | Gitea admin context (SSH + API token) |
| `anthropic-cloud/v1` | Anthropic API (subscription pricing) |
| `minimax/v1` | MiniMax API (SOPS-encrypted credentials) |
| `code-methodology/v1` | Coding methodology CLAUDE.md |

### Capability Layers

| Layer | What it provides |
|---|---|
| `python-dev/v1` | pytest, ruff, mypy, hypothesis, uv |

---

## Monitoring Tasks

### agent-monitor (terminal UI)

```bash
# Live view, filtered to your project
~/dev/claude/projects/agent-runtimes/scripts/agent-monitor --filter "project=my-project" --filter "age<20m"

# Single snapshot
~/dev/claude/projects/agent-runtimes/scripts/agent-monitor --once

# All running tasks
~/dev/claude/projects/agent-runtimes/scripts/agent-monitor --filter "state=running"
```

### API Queries

```bash
CP=http://localhost:8100

# Check task status
curl -s "$CP/tasks/{task_id}" | python3 -m json.tool

# List tasks (most recent)
curl -s "$CP/tasks?limit=10" | python3 -m json.tool

# Cancel a task
curl -s -X DELETE "$CP/tasks/{task_id}"
```

### Extracting Agent Output from Logs

Agents write to `/workspace/.agent-output/output.md`. To extract this from stream-json logs:

```python
import requests, json

def extract_output(cp_url, task_id):
    """Extract output.md content from completed task logs."""
    r = requests.get(f"{cp_url}/tasks/{task_id}", timeout=10).json()
    logs = r.get("logs", "") or ""
    output = None
    for line in logs.split("\n"):
        if not line.startswith("{"):
            continue
        try:
            event = json.loads(line)
        except:
            continue
        if event.get("type") == "assistant":
            for block in event.get("message", {}).get("content", []):
                if isinstance(block, dict) and block.get("type") == "tool_use":
                    if block.get("name") == "Write" and "output.md" in str(block.get("input", {}).get("file_path", "")):
                        output = block["input"]["content"]
    return output
```

---

## Workflows (Multi-Model Spec Planning)

Workflow templates define multi-step DAGs where different models collaborate, cross-review, and a human resolves disputes.

### spec-planning v3

The flagship workflow for spec development. 14-node DAG:

```
Phase 0:   interview_a + interview_b (parallel, different models)
Phase 0.5: consolidate_questions
           ↓ HUMAN GATE — answer questions ↓
Phase 1:   plan_a + plan_b (parallel, different models)
Phase 2:   6 cross-reviews (spec/security/TDD × 2 models, each reviews OTHER's plan)
Phase 3:   escalate (surfaces disagreements, recommends best model)
           ↓ HUMAN GATE — resolve disputes ↓
Phase 4:   synthesize (best model writes final spec)
```

### Running a Workflow Manually

Since the CP doesn't yet expand workflows natively, run each phase from a Claude Code session:

1. **Render prompts** — substitute `{{ task_description }}`, `{{ scope_notes }}`, etc. from the template YAML
2. **Submit tasks** — POST to CP with rendered prompts, correct harness per model
3. **Wait** — poll with agent-monitor or curl
4. **Extract artifacts** — read output.md from completed task logs
5. **Inject artifacts** — replace `<<ARTIFACT:node_id:key>>` sentinels in next phase's prompts
6. **Human gates** — present escalation output to user, collect answers, append to artifacts
7. **Repeat** for each phase

### Artifact Passing Between Stages

When a downstream task needs an upstream task's output:

```python
# Extract upstream output
upstream_output = extract_output(CP, upstream_task_id)

# Build downstream prompt with artifact injected
downstream_prompt = f"""
## Plan A (from upstream)
{upstream_output}

## Your Task
Review the above plan for security issues...
"""
```

### Workflow Templates Location

Templates live in `~/dev/claude/projects/agent-runtimes/workflows/`:

| Template | Description |
|---|---|
| `spec-planning.yaml` | Full 14-node spec planning with cross-model review |
| `comparative-plan.yaml` | Simpler 5-node comparative planning |

---

## Agent Repos — Git-Based Persistence

### What Is an Agent Repo

An agent repo is a **Gitea fork** of a project's main repo, named `{repo}-agents` (e.g., `agent-runtimes-agents`). Agents work on task-specific branches in the fork. All agent output is auto-committed to git before the container exits, so nothing is lost when ephemeral containers are removed.

### Creating an Agent Repo

One-time setup per project. The fork must exist before agents can use it.

```bash
# Authenticate with the ai_admin token (see ~/dev/claude/secrets/gitea/ai_admin)
TOKEN="<token from secrets>"

# Check if fork exists (HTTP 200 = yes, 404 = no)
curl -sk -H "Authorization: token $TOKEN" \
  "https://gitea.oreillyit.nz/api/v1/repos/{org}/{repo}-agents"

# Create the fork (HTTP 202 = accepted, HTTP 409 = already exists)
curl -sk -X POST \
  -H "Authorization: token $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"organization":"{org}","name":"{repo}-agents"}' \
  "https://gitea.oreillyit.nz/api/v1/repos/{org}/{repo}/forks"
```

### Naming Convention

| Main repo | Agent repo |
|---|---|
| `skynet/agent-runtimes` | `skynet/agent-runtimes-agents` |
| `skynet/brainiac-app` | `skynet/brainiac-app-agents` |

Always use the `-agents` suffix. The fork relationship enables cross-fork PRs.

### Workspace Layout

```
/workspace/
├── reference/              # Owned by root — read-only (agent gets error if they try to write)
│   ├── main/               # Main repo default branch
│   ├── plan-opus-4f3a/     # Another agent's output branch (if needed)
│   └── best-practices/     # Best practices repo (if cloned)
└── working/                # Agent's branch of the agent repo (read-write)
    ├── CLAUDE.md           # Project code from the fork
    ├── spec/
    └── results/            # Starts empty — guaranteed output location
        ├── output.md
        ├── session-log.md
        └── changelog.md
```

- **`/workspace/reference/`** — each subdirectory is a separate clone. Owned by root so agents get immediate permission errors if they try to modify.
- **`/workspace/working/`** — the agent's branch. All changes auto-committed on exit.
- **`/workspace/working/results/`** — always starts empty. Write outputs here.

### Branch Naming

- **With workflow context**: `{workflow_id}-{stage}-{short_task_id}` (e.g., `f58-plan-4f3a1b2c`)
- **Without workflow context**: `task-{short_task_id}` (e.g., `task-4f3a1b2c`)

### Auto-Commit (Finalize Phase)

After the agent exits (success or failure), the entrypoint runs finalize scripts:

1. Check for changes in `/workspace/working/`
2. `git add -A && git commit` with message: `"Agent task {id} ({status}): {prompt_summary}"`
3. `git push origin {branch}`
4. Write metadata to `/workspace/.agent-output/ci_metadata.json`

Finalize runs unconditionally — partial work from failed agents is preserved. Finalize failure does not override the agent's exit code.

### Metadata Propagation

After container exit, the dispatcher reads `ci_metadata.json` and reports to CP:

```json
{
  "agent_branch": "task-4f3a1b2c",
  "agent_sha": "a1b2c3d4...",
  "agent_repo_url": "git@gitea.oreillyit.nz:skynet/agent-runtimes-agents.git",
  "agent_branch_pushed": true
}
```

Stored in `CPTask.metadata` (JSONB dict). Query via `GET /tasks/{id}`.

### Credential Separation

| Actor | Credential | Purpose |
|---|---|---|
| Dispatcher | `GITEA_API_TOKEN` (read-only) | Verify fork and branches exist |
| Agent container | SSH key (via gitea-ssh harness) | Clone repos, push branches |

The dispatcher verifies prerequisites but doesn't create forks or branches.

### Retry Behaviour

On retry, the agent gets a fresh workspace (clean checkout from base branch). The previous attempt's branch is cloned into `/workspace/reference/previous-attempt/` with a note in CLAUDE.md that the last attempt was incomplete and any work it created can be found there.

### Cross-Repo PRs

When an agent working in the agent repo needs to create a PR against the main repo, Gitea supports cross-fork PRs natively. This is not automated initially — escalate to a human or handle on demand.

### Branch Cleanup

Out of scope for initial implementation. The `task-` prefix and deterministic naming make automated pruning straightforward when needed.

---

## Known Agent Repos

| Project | Agent Repo | Created |
|---|---|---|
| `skynet/agent-runtimes` | `skynet/agent-runtimes-agents` | 2026-04-05 |

---

## Quick Reference

```bash
# Port-forward to CP
kubectl port-forward -n agent-runtimes svc/controlplane 8100:8100 &

# Submit a task
curl -s -X POST http://localhost:8100/tasks -H "Content-Type: application/json" \
  -d '{"name":"my-task","project_id":"my-project","prompt":"...","harness":"planning-opus-repo/v1","runtime":{"cli":"claude"},"pre_actions":[{"type":"clone","repo":"git@gitea.oreillyit.nz-ai-enablement:skynet/my-project.git"}]}'

# Monitor
~/dev/claude/projects/agent-runtimes/scripts/agent-monitor --filter "project=my-project" --filter "age<20m"

# Check result
curl -s http://localhost:8100/tasks/{id} | python3 -m json.tool

# Cancel
curl -s -X DELETE http://localhost:8100/tasks/{id}
```

---

## Artifact Passing via Git Branches Instead of Env Vars

For multi-stage workflows where downstream tasks need upstream outputs, push artifacts to branches in an agent repo rather than embedding in prompts or env vars. This avoids K8s env var size limits (~228KB), survives pod restarts, provides a git audit trail, and scales to any artifact size. The downstream task clones the branch as a reference directory.

## Protect Test Files from Agent Modification via Root-Owned Read-Only Clone

When agents run tests, they may "fix" failing tests by weakening assertions rather than fixing the underlying code. Prevent this by cloning the test suite into `/workspace/reference/tests/` as a root-owned directory (the agent gets a permission error if it tries to write). The agent's working directory gets a symlink or copy of the tests at startup, but the authoritative copy is immutable. This is the same pattern as `/workspace/reference/main/` for source code — root ownership makes modification a hard error, not a policy.

## Infrastructure Failures Dominate Agent Failure Modes

In measured agent runs, the majority of task failures are infrastructure failures, not agent reasoning failures: network timeouts, SSH key not loaded, missing package in the base image, environment variable not propagated. Before debugging agent behaviour, check whether the failure is environmental — a task that consistently fails at "git clone" is an infrastructure problem, not an agent problem.

**Concrete ratio:** in one measured 12-task batch, 6/12 tasks failed and all 6 were infrastructure-class (wrong harness, missing payload fields, stale images) — zero model failures. Task templates that validate payload structure and harness compatibility before dispatch eliminate the entire dominant failure mode.

**Pre-dispatch validation checklist:**
- SSH key reachable from the agent harness (test clone before dispatching)
- Required env vars present (model API keys, registry credentials)
- Harness image has all required tools (`uv`, `ruff`, `pytest`, etc.)
- Target repo and branch exist
- Network egress allows required domains
- **Task templates** — long-term remediation. Validate payload structure and harness compatibility at template-render time, not via ad-hoc per-dispatch checks.

Invest in pre-dispatch validation scripts that catch the top-N infrastructure failures before the first agent container starts.

## Cross-Model Reviews Catch ~38% More Issues Than a Single Model

Running the same security or spec review with two different models (e.g., Opus + MiniMax) and comparing outputs catches ~38% more issues than either alone — in measured reviews, only 62% of findings overlap. Models converge on obvious issues but diverge on edge cases and design concerns. Worth the extra cost for security-critical specs and architecture reviews; overkill for routine code review.

**Pattern:** dispatch parallel review tasks to different models with identical prompts, union the findings, deduplicate against a shared issue key (file + line + category). Present the merged list to the human reviewer along with per-model attribution so reviewers can see where models agreed vs. diverged.

## Agent Worktree Branches Contain Files, Not Commits — Copy, Don't Merge

**Symptom:** Orchestrator merges an agent's branch and sees "Already up to date" because the agent wrote files to its worktree but never ran `git add` / `git commit`. Downstream tasks that depend on the upstream artifact then fail or silently use stale data.

**Fix:** Orchestration must explicitly copy files from dependency worktrees (driven by a `writes` field in the task manifest) into the consuming worktree. Git merge is insufficient when agent output is untracked.

**Alternative:** Require agents to commit before exit (finalize phase auto-commits everything under `/workspace/working/`), which unlocks git-branch artifact passing. The finalize-phase auto-commit described above is the canonical implementation — enforce it for any agent whose output other tasks depend on.

## Include Exact Dataclass/Context Schemas in Template-Writing Agent Prompts

**Symptom:** Agents writing templates invent their own mock context objects (e.g., dict-style `model["fields"]`) while real code provides a different shape (e.g., dataclass `model.fields`). Templates render against the mocks but produce attribute errors against real objects during integration.

**Fix:** Always include exact dataclass/type definitions of the render context in the agent prompt. During review, compare each agent's mock objects against the real normalized types before merging. Treat "wrote its own mock shape" as a review-blocking issue — the mock shape is a contract the agent must follow, not invent.

## Never Assume Web Search in Container Agents; Validate Version-Specific Claims Separately

**Symptom:** Agents in Docker/K8s containers have no `WebSearch` or `WebFetch` capability even with `--dangerously-skip-permissions`. Version numbers, release dates, and "actively maintained" claims come from training data and are often wrong (~30% inaccuracy on package versions in measured runs).

**Fix:**
- Never tell a container agent to "use web search" — it has none and will silently fabricate from training data.
- Run a separate web-validation pass (Opus or similar with network access) after research agents complete.
- Treat all version claims as hypotheses until validated.
- Budget web validation as a distinct pipeline stage, not an afterthought.

## Cost-Effective Models Need Explicit Scope Boundaries

Smaller/cheaper models (e.g., MiniMax, Haiku) need tighter scope constraints than capable flagship models. Without explicit boundaries, they drift into scope creep, run the full test suite when asked to write tests, or attempt broad refactors. For cost-effective model tasks:
- **No full test suite runs** — specify which test file or test ID to run
- **Concrete patterns, not open-ended** — "Write a test matching `test_cp_*.py` naming" not "Write tests for the control plane"
- **Longer timeouts** — cheaper models are often slower per token; set `runtime.timeout` to 2-3× what flagship models need
- **Explicit output location** — "Write to `results/output.md`" not "Write your findings"

Treat scope boundaries as a harness concern, not an agent concern — encode them in the prompt template or harness context, not in ad-hoc task prompts.

## Airouter Qwen3.6 — Scope Decomposition for Reliable Output

Qwen3.6 via airouter harness (`runtime_cli: agentic`, model `Qwen3.6`) produces reliable output for single, concrete file-creation tasks but has shown silent failures on multi-part instructions requiring sustained context. In measured dispatches:
- **Reliable**: Create one new file, one hard rule, clear output path
- **Unreliable**: Create multiple files with distinct content, multi-step instructions, tasks requiring internal state management across turns

**Pattern confirmed (2026-05-04):**
- D7 (1 file, 1 rule) → fully successful airouter output ✅
- D4 original (2 files, 2 rule sets) → silent no-output ❌
- D4a (harness.yaml only, 1 file) → fully successful ✅
- D4b (CLAUDE.md only, 1 file) → fully successful, committed in 5 turns ✅

Single-file scope is the reliable unit for airouter/Qwen3.6 dispatches.

**Dispatch strategy for airouter agents:**
1. **One file per task** — split multi-file tasks into separate dispatches. Each task creates exactly one file.
2. **Reference the template, don't copy it** — give the agent the path to an existing similar file to read, rather than including all structure in the prompt.
3. **State the file path explicitly** — "Create `harnesses/contexts/cp-harness/v1/harness.yaml`" at the start, not implied by context.
4. **Verify output before treating as done** — check the agent-repo branch for the expected file immediately after task succeeds.

If a multi-file task is unavoidable, use a **sequential chain**: dispatch task-1 to create file-A, wait for success, then dispatch task-2 to create file-B referencing task-1's output. This bounds the agent's scope to one artifact at a time.

**BUG-21** (2026-05-04): D4 task (`017f63cb`) exited 0, showed correct understanding in logs, but produced no harness files. Fix confirmed: decompose airouter tasks to single-file units (D4a + D4b both succeeded).

**Pattern reconfirmed (2026-05-08, batch of 9):** After the Wave A1 destructive-`Write` incident (task `4a2f2988`) and 12 pipeline fixes, the airouter dogfood pipeline produced 8 successful dispatches in 24 hours plus 1 operator-induced failure: M16 Wave A1 + A2 (validator + class additions to `controlplane/api/identity_deps.py`), M16 Wave B1 (Pydantic `model_validator` regex check), M25 Waves A1-A3, A5, A4 + B1, B2 (seven single-file `git rm` deletions). The single agent "failure" (Wave B2, MN-4 prompt cap) was an operator-side test bug, not an agent regression — see "Test Design for AI Agent Dogfood Pipelines" below.

**Time-to-success bands** (useful for cost calibration):
- 1-line `git rm` deletion: ~1m30s, ~6 turns (fastest: M25-A4 at 1m37s)
- Add a 5-line Pydantic regex validator: ~5 min, ~30 turns (M16-B1 / MN-5 grammar)
- Add a 13-line class definition: ~12 min, ~60 turns (M16-A2 / RFC9457Problem)

Tasks running beyond 12 min / 60 turns suggest the agent is over-cautious or stuck. **Default `--max-test-iterations 1` for cheap probes**; reserve 3+ only for tasks where the agent might genuinely benefit from the failure context (i.e., the test is correct and the agent may need a second look at its own output). For tasks that fail on a structurally impossible test, all retries fail identically — see the next section.

## Test Design for AI Agent Dogfood Pipelines

When a CLI tool or web framework returns sanitised error responses for security reasons, dogfood tests must respect that contract. A test that asserts on response body content for an error path the framework deliberately strips will block the agent indefinitely — the validator can be perfectly correct and the test still fails.

**Concrete incident (2026-05-08, task `26717643`):** A 16 KB prompt-cap dogfood test asserted both `response.status_code == 422` AND that the body mentioned "prompt", "exceeds", or "16384". The agent (airouter / Qwen3.6) wrote three correct implementations across F70 retries; each was rejected because the CP's `RequestValidationError` handler sanitises 422 bodies to `{"detail":"Request validation failed","correlation_id":...}` — Pydantic's verbose detail is logged server-side but never leaked to the client. 14 minutes of compute spent on a structurally impossible test.

**Rules:**
1. **Verify the dogfood test passes against a hand-coded reference implementation BEFORE pushing it.** A red-on-broken-test loop is invisible to the agent; only operator-side validation catches it. The 30-second cost of running pytest locally beats the 14-minute cost of a doomed dispatch.
2. **For validator-driven 422s in security-conscious frameworks, status-code-only assertions are the ceiling.** Add structural body assertions only for 200/201 paths or for handlers that write their own response body.
3. **Model new dogfood tests on previously-successful tests, not stricter variants.** If an existing test for a similar requirement only checks status code, the new one should too — unless you've actively confirmed the framework leaks the structure you want to assert on.
4. **F70 retries don't help when the test is fundamentally wrong.** The agent has no signal to learn from a structural impossibility. The retry mechanism assumes the test is correct and the agent's output is iteratively improvable; that assumption breaks for operator-side test bugs.

## Dogfood Failure Path: What's Visible, What Isn't

When an agent task fails after exhausting `--max-test-iterations` retries (verified against M16 Wave B2 + B3, 2026-05-08):

**Visible:**
- The agent's last attempt **IS pushed** to the agent-repo fork as `task-<id>`. Dispatcher logs `branch will still push (partial work preserved)`. Diff with `git show origin/task-<id>` to see exactly what the agent wrote.
- The CP task record's `logs` field **IS populated** (~50 KB) when queried with `?include_logs=true`. Contains dispatcher state transitions, the agent's text streams, and tool-call summaries.
- Dispatcher pod logs persist while the pod is alive (current ReplicaSet). Cross-reference task ID for high-level state.

**NOT visible:**
- The agent pod itself (K8s Job cleanup deletes it within seconds of completion).
- The pytest stdout/stderr from F70's invocation. Only `Tests failed on attempt N/N` is logged. The agent's `run_command` calls show `result_bytes` counts but not contents — so we know F70 said "fail" but not which assertion or why.

**Operator pattern after a failed dispatch:**
1. `git fetch origin '+refs/heads/task-<id>:refs/remotes/origin/task-<id>'` against the agent-repo fork
2. `git show origin/task-<id> -- <target-file>` to see the agent's edit
3. Apply the diff locally with `git apply`, run the dogfood test against it
4. If it passes locally but failed in F70, the failure is **environmental** — see next section
5. Cherry-pick the agent's work to main; the agent earned the credit even if F70 mis-reported

**Until F70 captures pytest stdout/stderr** (action item: write to `/workspace/.agent-output/f70-attempt-<N>.log`), failed dispatches whose code passes locally are post-mortem black boxes. Spend the 30 seconds to apply-and-run locally before assuming the agent was wrong.

## F70 Pytest Can Disagree with Local Pytest

**Symptom (M16 Wave B3, task `a496efd2`, 2026-05-08):** airouter agent writes a 15-line model_validator that passes 5/5 dogfood tests locally; F70 reports `Tests failed on attempt 1/2` and `2/2`. Branch + agent's diff applied to a fresh local checkout of the same SHA → 5/5 pass.

The agent's code was correct. The pipeline reported failure. Wasted: 9 minutes of compute and 2 retries that all hit the same environmental disagreement.

**Possible causes (not yet narrowed):**
- `run-ci-tests.sh` runs from a different working dir than the agent edited
- Stale `__pycache__` from a prior attempt's Pydantic model
- Different Python interpreter / virtualenv state inside the container
- F70 pytest discovery uses different conftest paths

**Operator workaround:** `--max-test-iterations 1` for cheap probes. After failure, inspect the agent's branch and apply locally. If local passes, cherry-pick the agent's work to main and capture the local-vs-container divergence as a separate follow-up — don't punish a correct agent for an environmental glitch.