best-practices/agent-repos.md

# Agent Repos & Container Agent Operations

Best practices for running AI agents via the agent-runtimes control plane — submitting tasks, using agent repos for persistence, running multi-model workflows, and monitoring progress.

## Architecture Overview

The agent-runtimes system runs AI coding agents inside ephemeral Docker/K8s containers, orchestrated by a control plane (CP). The three-tier model:

1. **Control Plane** — central task queue, dispatcher registry, model routing. External systems (including Claude Code sessions) submit tasks here.
2. **Dispatchers** — poll the CP for tasks, resolve harness+model, create containers. All connections outbound (works behind firewalls).
3. **Agent Containers** — ephemeral, isolated. Receive a JSON payload, execute work, exit.

```
You (Claude Code session) → POST /tasks → Control Plane → Dispatcher → Agent Container
                           ← GET /tasks/{id} ← (poll for results)
```

### Access

| Environment | CP URL |
|---|---|
| K8s (port-forward) | `kubectl port-forward -n agent-runtimes svc/controlplane 8100:8100` then `http://localhost:8100` |
| K8s (MFA) | `https://agents.oreillyit.nz` (behind Authelia) |
| Local dev (Docker Compose) | `http://localhost:8100` after `docker compose up -d` |

---

## Submitting Tasks

### Minimal Task

```bash
curl -s -X POST http://localhost:8100/tasks \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Fix test_auth.py",
    "project_id": "my-project",
    "prompt": "Fix the failing test in test_auth.py",
    "harness": "planning-opus-repo/v1",
    "runtime": {"cli": "claude"},
    "pre_actions": [
      {"type": "clone", "repo": "git@gitea.oreillyit.nz-ai-enablement:skynet/my-project.git", "depth": 1}
    ]
  }'
```

### Full Task Payload

```json
{
  "task_id": "optional-uuid",
  "name": "Short name (shown in monitor)",
  "project_id": "groups tasks in monitor",
  "prompt": "The instruction for the agent",
  "harness": "composite-harness-name/v1",
  "runtime": {
    "cli": "claude",
    "model": "sonnet",
    "timeout": 1800
  },
  "priority": 0,
  "metadata": {
    "workflow": "my-workflow",
    "phase": "plan",
    "model": "opus"
  },
  "pre_actions": [
    {"type": "clone", "repo": "git@...", "branch": "main", "depth": 1}
  ],
  "on_success": [
    {"type": "commit_pr", "branch": "feature-branch", "title": "PR title", "base": "main"}
  ],
  "on_error": [
    {"type": "report", "webhook_url": "https://..."}
  ],
  "max_retries": 1
}
```

### Key Fields

| Field | Required | Description |
|---|---|---|
| `name` | Recommended | Short task name shown in the monitor |
| `project_id` | Recommended | Groups tasks in the monitor; enables filtering |
| `prompt` | Yes | The instruction sent to the agent |
| `harness` | Recommended | Composite harness (defines credentials, context, model provider) |
| `runtime.cli` | No | CLI runner: `claude` (default), `openai_compat`, `agentic` |
| `pre_actions` | No | Setup actions before agent runs (e.g., `clone`) |
| `on_success` | No | Post-agent actions on success (e.g., `commit_pr`, `report`) |
| `on_error` | No | Post-agent actions on failure |
| `metadata` | No | Arbitrary JSONB — used for workflow tracking, filtering |
| `priority` | No | -100 to 100 (higher = picked first, default 0) |

### Python Task Submission

For programmatic use from a Claude Code session:

```python
import requests, json, uuid

CP = "http://localhost:8100"
task_id = str(uuid.uuid4())

payload = {
    "task_id": task_id,
    "name": "My agent task",
    "project_id": "my-project",
    "prompt": "...",
    "harness": "planning-opus-repo/v1",
    "runtime": {"cli": "claude"},
    "pre_actions": [{"type": "clone", "repo": "git@gitea.oreillyit.nz-ai-enablement:skynet/my-project.git"}],
    "metadata": {"workflow": "my-workflow", "phase": "plan"}
}

resp = requests.post(f"{CP}/tasks", json=payload, timeout=10)
data = resp.json()
print(f"Submitted: {data['task_id']}")
```

---

## Available Harnesses

Harnesses define what credentials, context, and model provider an agent gets. Composites combine multiple layers.

### Composite Harnesses (ready to use)

| Harness | Model Provider | Capabilities |
|---|---|---|
| `planning-opus-repo/v1` | Anthropic Claude (subscription) | Planning context + SSH clone |
| `planning-minimax-repo/v1` | MiniMax | Planning context + SSH clone |
| `python-code-review/v1` | Default (Anthropic) | Python dev tools + Gitea admin |

### Context Layers (building blocks)

| Layer | What it provides |
|---|---|
| `planning/v1` | Best-practices files for spec/plan writing |
| `gitea-ssh/v1` | SSH key for clone/push to Gitea |
| `gitea-admin/v1` | Gitea admin context (SSH + API token) |
| `anthropic-cloud/v1` | Anthropic API (subscription pricing) |
| `minimax/v1` | MiniMax API (SOPS-encrypted credentials) |
| `code-methodology/v1` | Coding methodology CLAUDE.md |

### Capability Layers

| Layer | What it provides |
|---|---|
| `python-dev/v1` | pytest, ruff, mypy, hypothesis, uv |

---

## Monitoring Tasks

### agent-monitor (terminal UI)

```bash
# Live view, filtered to your project
~/dev/claude/projects/agent-runtimes/scripts/agent-monitor --filter "project=my-project" --filter "age<20m"

# Single snapshot
~/dev/claude/projects/agent-runtimes/scripts/agent-monitor --once

# All running tasks
~/dev/claude/projects/agent-runtimes/scripts/agent-monitor --filter "state=running"
```

### API Queries

```bash
CP=http://localhost:8100

# Check task status
curl -s "$CP/tasks/{task_id}" | python3 -m json.tool

# List tasks (most recent)
curl -s "$CP/tasks?limit=10" | python3 -m json.tool

# Cancel a task
curl -s -X DELETE "$CP/tasks/{task_id}"
```

### Extracting Agent Output from Logs

Agents write to `/workspace/.agent-output/output.md`. To extract this from stream-json logs:

```python
import requests, json

def extract_output(cp_url, task_id):
    """Extract output.md content from completed task logs."""
    r = requests.get(f"{cp_url}/tasks/{task_id}", timeout=10).json()
    logs = r.get("logs", "") or ""
    output = None
    for line in logs.split("\n"):
        if not line.startswith("{"):
            continue
        try:
            event = json.loads(line)
        except:
            continue
        if event.get("type") == "assistant":
            for block in event.get("message", {}).get("content", []):
                if isinstance(block, dict) and block.get("type") == "tool_use":
                    if block.get("name") == "Write" and "output.md" in str(block.get("input", {}).get("file_path", "")):
                        output = block["input"]["content"]
    return output
```

---

## Workflows (Multi-Model Spec Planning)

Workflow templates define multi-step DAGs where different models collaborate, cross-review, and a human resolves disputes.

### spec-planning v3

The flagship workflow for spec development. 14-node DAG:

```
Phase 0:   interview_a + interview_b (parallel, different models)
Phase 0.5: consolidate_questions
           ↓ HUMAN GATE — answer questions ↓
Phase 1:   plan_a + plan_b (parallel, different models)
Phase 2:   6 cross-reviews (spec/security/TDD × 2 models, each reviews OTHER's plan)
Phase 3:   escalate (surfaces disagreements, recommends best model)
           ↓ HUMAN GATE — resolve disputes ↓
Phase 4:   synthesize (best model writes final spec)
```

### Running a Workflow Manually

Since the CP doesn't yet expand workflows natively, run each phase from a Claude Code session:

1. **Render prompts** — substitute `{{ task_description }}`, `{{ scope_notes }}`, etc. from the template YAML
2. **Submit tasks** — POST to CP with rendered prompts, correct harness per model
3. **Wait** — poll with agent-monitor or curl
4. **Extract artifacts** — read output.md from completed task logs
5. **Inject artifacts** — replace `<<ARTIFACT:node_id:key>>` sentinels in next phase's prompts
6. **Human gates** — present escalation output to user, collect answers, append to artifacts
7. **Repeat** for each phase

### Artifact Passing Between Stages

When a downstream task needs an upstream task's output:

```python
# Extract upstream output
upstream_output = extract_output(CP, upstream_task_id)

# Build downstream prompt with artifact injected
downstream_prompt = f"""
## Plan A (from upstream)
{upstream_output}

## Your Task
Review the above plan for security issues...
"""
```

### Workflow Templates Location

Templates live in `~/dev/claude/projects/agent-runtimes/workflows/`:

| Template | Description |
|---|---|
| `spec-planning.yaml` | Full 14-node spec planning with cross-model review |
| `comparative-plan.yaml` | Simpler 5-node comparative planning |

---

## Agent Repos — Git-Based Persistence

### What Is an Agent Repo

An agent repo is a **Gitea fork** of a project's main repo, named `{repo}-agents` (e.g., `agent-runtimes-agents`). Agents work on task-specific branches in the fork. All agent output is auto-committed to git before the container exits, so nothing is lost when ephemeral containers are removed.

### Creating an Agent Repo

One-time setup per project. The fork must exist before agents can use it.

```bash
# Authenticate with the ai_admin token (see ~/dev/claude/secrets/gitea/ai_admin)
TOKEN="<token from secrets>"

# Check if fork exists (HTTP 200 = yes, 404 = no)
curl -sk -H "Authorization: token $TOKEN" \
  "https://gitea.oreillyit.nz/api/v1/repos/{org}/{repo}-agents"

# Create the fork (HTTP 202 = accepted, HTTP 409 = already exists)
curl -sk -X POST \
  -H "Authorization: token $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"organization":"{org}","name":"{repo}-agents"}' \
  "https://gitea.oreillyit.nz/api/v1/repos/{org}/{repo}/forks"
```

### Naming Convention

| Main repo | Agent repo |
|---|---|
| `skynet/agent-runtimes` | `skynet/agent-runtimes-agents` |
| `skynet/brainiac-app` | `skynet/brainiac-app-agents` |

Always use the `-agents` suffix. The fork relationship enables cross-fork PRs.

### Workspace Layout

```
/workspace/
├── reference/              # Owned by root — read-only (agent gets error if they try to write)
│   ├── main/               # Main repo default branch
│   ├── plan-opus-4f3a/     # Another agent's output branch (if needed)
│   └── best-practices/     # Best practices repo (if cloned)
└── working/                # Agent's branch of the agent repo (read-write)
    ├── CLAUDE.md           # Project code from the fork
    ├── spec/
    └── results/            # Starts empty — guaranteed output location
        ├── output.md
        ├── session-log.md
        └── changelog.md
```

- **`/workspace/reference/`** — each subdirectory is a separate clone. Owned by root so agents get immediate permission errors if they try to modify.
- **`/workspace/working/`** — the agent's branch. All changes auto-committed on exit.
- **`/workspace/working/results/`** — always starts empty. Write outputs here.

### Branch Naming

- **With workflow context**: `{workflow_id}-{stage}-{short_task_id}` (e.g., `f58-plan-4f3a1b2c`)
- **Without workflow context**: `task-{short_task_id}` (e.g., `task-4f3a1b2c`)

### Auto-Commit (Finalize Phase)

After the agent exits (success or failure), the entrypoint runs finalize scripts:

1. Check for changes in `/workspace/working/`
2. `git add -A && git commit` with message: `"Agent task {id} ({status}): {prompt_summary}"`
3. `git push origin {branch}`
4. Write metadata to `/workspace/.agent-output/ci_metadata.json`

Finalize runs unconditionally — partial work from failed agents is preserved. Finalize failure does not override the agent's exit code.

### Metadata Propagation

After container exit, the dispatcher reads `ci_metadata.json` and reports to CP:

```json
{
  "agent_branch": "task-4f3a1b2c",
  "agent_sha": "a1b2c3d4...",
  "agent_repo_url": "git@gitea.oreillyit.nz:skynet/agent-runtimes-agents.git",
  "agent_branch_pushed": true
}
```

Stored in `CPTask.metadata` (JSONB dict). Query via `GET /tasks/{id}`.

### Credential Separation

| Actor | Credential | Purpose |
|---|---|---|
| Dispatcher | `GITEA_API_TOKEN` (read-only) | Verify fork and branches exist |
| Agent container | SSH key (via gitea-ssh harness) | Clone repos, push branches |

The dispatcher verifies prerequisites but doesn't create forks or branches.

### Retry Behaviour

On retry, the agent gets a fresh workspace (clean checkout from base branch). The previous attempt's branch is cloned into `/workspace/reference/previous-attempt/` with a note in CLAUDE.md that the last attempt was incomplete and any work it created can be found there.

### Cross-Repo PRs

When an agent working in the agent repo needs to create a PR against the main repo, Gitea supports cross-fork PRs natively. This is not automated initially — escalate to a human or handle on demand.

### Branch Cleanup

Out of scope for initial implementation. The `task-` prefix and deterministic naming make automated pruning straightforward when needed.

---

## Known Agent Repos

| Project | Agent Repo | Created |
|---|---|---|
| `skynet/agent-runtimes` | `skynet/agent-runtimes-agents` | 2026-04-05 |

---

## Quick Reference

```bash
# Port-forward to CP
kubectl port-forward -n agent-runtimes svc/controlplane 8100:8100 &

# Submit a task
curl -s -X POST http://localhost:8100/tasks -H "Content-Type: application/json" \
  -d '{"name":"my-task","project_id":"my-project","prompt":"...","harness":"planning-opus-repo/v1","runtime":{"cli":"claude"},"pre_actions":[{"type":"clone","repo":"git@gitea.oreillyit.nz-ai-enablement:skynet/my-project.git"}]}'

# Monitor
~/dev/claude/projects/agent-runtimes/scripts/agent-monitor --filter "project=my-project" --filter "age<20m"

# Check result
curl -s http://localhost:8100/tasks/{id} | python3 -m json.tool

# Cancel
curl -s -X DELETE http://localhost:8100/tasks/{id}
```

---

## Artifact Passing via Git Branches Instead of Env Vars

For multi-stage workflows where downstream tasks need upstream outputs, push artifacts to branches in an agent repo rather than embedding in prompts or env vars. This avoids K8s env var size limits (~228KB), survives pod restarts, provides a git audit trail, and scales to any artifact size. The downstream task clones the branch as a reference directory.