best-practices/agent-repos.md

# Agent Repos & Container Agent Operations

Best practices for running AI agents via the agent-runtimes control plane — submitting tasks, using agent repos for persistence, running multi-model workflows, and monitoring progress.

## Architecture Overview

The agent-runtimes system runs AI coding agents inside ephemeral Docker/K8s containers, orchestrated by a control plane (CP). The three-tier model:

1. **Control Plane** — central task queue, dispatcher registry, model routing. External systems (including Claude Code sessions) submit tasks here.
2. **Dispatchers** — poll the CP for tasks, resolve harness+model, create containers. All connections outbound (works behind firewalls).
3. **Agent Containers** — ephemeral, isolated. Receive a JSON payload, execute work, exit.

```
You (Claude Code session) → POST /tasks → Control Plane → Dispatcher → Agent Container
                           ← GET /tasks/{id} ← (poll for results)
```

### Access

| Environment | CP URL |
|---|---|
| K8s (port-forward) | `kubectl port-forward -n agent-runtimes svc/controlplane 8100:8100` then `http://localhost:8100` |
| K8s (MFA) | `https://agents.oreillyit.nz` (behind Authelia) |
| Local dev (Docker Compose) | `http://localhost:8100` after `docker compose up -d` |

---

## Submitting Tasks

### Minimal Task

```bash
curl -s -X POST http://localhost:8100/tasks \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Fix test_auth.py",
    "project_id": "my-project",
    "prompt": "Fix the failing test in test_auth.py",
    "harness": "planning-opus-repo/v1",
    "runtime": {"cli": "claude"},
    "pre_actions": [
      {"type": "clone", "repo": "git@gitea.oreillyit.nz-ai-enablement:skynet/my-project.git", "depth": 1}
    ]
  }'
```

### Full Task Payload

```json
{
  "task_id": "optional-uuid",
  "name": "Short name (shown in monitor)",
  "project_id": "groups tasks in monitor",
  "prompt": "The instruction for the agent",
  "harness": "composite-harness-name/v1",
  "runtime": {
    "cli": "claude",
    "model": "sonnet",
    "timeout": 1800
  },
  "priority": 0,
  "metadata": {
    "workflow": "my-workflow",
    "phase": "plan",
    "model": "opus"
  },
  "pre_actions": [
    {"type": "clone", "repo": "git@...", "branch": "main", "depth": 1}
  ],
  "on_success": [
    {"type": "commit_pr", "branch": "feature-branch", "title": "PR title", "base": "main"}
  ],
  "on_error": [
    {"type": "report", "webhook_url": "https://..."}
  ],
  "max_retries": 1
}
```

### Key Fields

| Field | Required | Description |
|---|---|---|
| `name` | Recommended | Short task name shown in the monitor |
| `project_id` | Recommended | Groups tasks in the monitor; enables filtering |
| `prompt` | Yes | The instruction sent to the agent |
| `harness` | Recommended | Composite harness (defines credentials, context, model provider) |
| `runtime.cli` | No | CLI runner: `claude` (default), `openai_compat`, `agentic` |
| `pre_actions` | No | Setup actions before agent runs (e.g., `clone`) |
| `on_success` | No | Post-agent actions on success (e.g., `commit_pr`, `report`) |
| `on_error` | No | Post-agent actions on failure |
| `metadata` | No | Arbitrary JSONB — used for workflow tracking, filtering |
| `priority` | No | -100 to 100 (higher = picked first, default 0) |

### Python Task Submission

For programmatic use from a Claude Code session:

```python
import requests, json, uuid

CP = "http://localhost:8100"
task_id = str(uuid.uuid4())

payload = {
    "task_id": task_id,
    "name": "My agent task",
    "project_id": "my-project",
    "prompt": "...",
    "harness": "planning-opus-repo/v1",
    "runtime": {"cli": "claude"},
    "pre_actions": [{"type": "clone", "repo": "git@gitea.oreillyit.nz-ai-enablement:skynet/my-project.git"}],
    "metadata": {"workflow": "my-workflow", "phase": "plan"}
}

resp = requests.post(f"{CP}/tasks", json=payload, timeout=10)
data = resp.json()
print(f"Submitted: {data['task_id']}")
```

---

## Available Harnesses

Harnesses define what credentials, context, and model provider an agent gets. Composites combine multiple layers.

### Composite Harnesses (ready to use)

| Harness | Model Provider | Capabilities |
|---|---|---|
| `planning-opus-repo/v1` | Anthropic Claude (subscription) | Planning context + SSH clone |
| `planning-minimax-repo/v1` | MiniMax | Planning context + SSH clone |
| `python-code-review/v1` | Default (Anthropic) | Python dev tools + Gitea admin |

### Context Layers (building blocks)

| Layer | What it provides |
|---|---|
| `planning/v1` | Best-practices files for spec/plan writing |
| `gitea-ssh/v1` | SSH key for clone/push to Gitea |
| `gitea-admin/v1` | Gitea admin context (SSH + API token) |
| `anthropic-cloud/v1` | Anthropic API (subscription pricing) |
| `minimax/v1` | MiniMax API (SOPS-encrypted credentials) |
| `code-methodology/v1` | Coding methodology CLAUDE.md |

### Capability Layers

| Layer | What it provides |
|---|---|
| `python-dev/v1` | pytest, ruff, mypy, hypothesis, uv |

---

## Monitoring Tasks

### agent-monitor (terminal UI)

```bash
# Live view, filtered to your project
~/dev/claude/projects/agent-runtimes/scripts/agent-monitor --filter "project=my-project" --filter "age<20m"

# Single snapshot
~/dev/claude/projects/agent-runtimes/scripts/agent-monitor --once

# All running tasks
~/dev/claude/projects/agent-runtimes/scripts/agent-monitor --filter "state=running"
```

### API Queries

```bash
CP=http://localhost:8100

# Check task status
curl -s "$CP/tasks/{task_id}" | python3 -m json.tool

# List tasks (most recent)
curl -s "$CP/tasks?limit=10" | python3 -m json.tool

# Cancel a task
curl -s -X DELETE "$CP/tasks/{task_id}"
```

### Extracting Agent Output from Logs

Agents write to `/workspace/.agent-output/output.md`. To extract this from stream-json logs:

```python
import requests, json

def extract_output(cp_url, task_id):
    """Extract output.md content from completed task logs."""
    r = requests.get(f"{cp_url}/tasks/{task_id}", timeout=10).json()
    logs = r.get("logs", "") or ""
    output = None
    for line in logs.split("\n"):
        if not line.startswith("{"):
            continue
        try:
            event = json.loads(line)
        except:
            continue
        if event.get("type") == "assistant":
            for block in event.get("message", {}).get("content", []):
                if isinstance(block, dict) and block.get("type") == "tool_use":
                    if block.get("name") == "Write" and "output.md" in str(block.get("input", {}).get("file_path", "")):
                        output = block["input"]["content"]
    return output
```

---

## Workflows (Multi-Model Spec Planning)

Workflow templates define multi-step DAGs where different models collaborate, cross-review, and a human resolves disputes.

### spec-planning v3

The flagship workflow for spec development. 14-node DAG:

```
Phase 0:   interview_a + interview_b (parallel, different models)
Phase 0.5: consolidate_questions
           ↓ HUMAN GATE — answer questions ↓
Phase 1:   plan_a + plan_b (parallel, different models)
Phase 2:   6 cross-reviews (spec/security/TDD × 2 models, each reviews OTHER's plan)
Phase 3:   escalate (surfaces disagreements, recommends best model)
           ↓ HUMAN GATE — resolve disputes ↓
Phase 4:   synthesize (best model writes final spec)
```

### Running a Workflow Manually

Since the CP doesn't yet expand workflows natively, run each phase from a Claude Code session:

1. **Render prompts** — substitute `{{ task_description }}`, `{{ scope_notes }}`, etc. from the template YAML
2. **Submit tasks** — POST to CP with rendered prompts, correct harness per model
3. **Wait** — poll with agent-monitor or curl
4. **Extract artifacts** — read output.md from completed task logs
5. **Inject artifacts** — replace `<<ARTIFACT:node_id:key>>` sentinels in next phase's prompts
6. **Human gates** — present escalation output to user, collect answers, append to artifacts
7. **Repeat** for each phase

### Artifact Passing Between Stages

When a downstream task needs an upstream task's output:

```python
# Extract upstream output
upstream_output = extract_output(CP, upstream_task_id)

# Build downstream prompt with artifact injected
downstream_prompt = f"""
## Plan A (from upstream)
{upstream_output}

## Your Task
Review the above plan for security issues...
"""
```

### Workflow Templates Location

Templates live in `~/dev/claude/projects/agent-runtimes/workflows/`:

| Template | Description |
|---|---|
| `spec-planning.yaml` | Full 14-node spec planning with cross-model review |
| `comparative-plan.yaml` | Simpler 5-node comparative planning |

---

## Agent Repos — Git-Based Persistence

### What Is an Agent Repo

An agent repo is a **Gitea fork** of a project's main repo, named `{repo}-agents` (e.g., `agent-runtimes-agents`). Agents work on task-specific branches in the fork. All agent output is auto-committed to git before the container exits, so nothing is lost when ephemeral containers are removed.

### Creating an Agent Repo

One-time setup per project. The fork must exist before agents can use it.

```bash
# Authenticate with the ai_admin token (see ~/dev/claude/secrets/gitea/ai_admin)
TOKEN="<token from secrets>"

# Check if fork exists (HTTP 200 = yes, 404 = no)
curl -sk -H "Authorization: token $TOKEN" \
  "https://gitea.oreillyit.nz/api/v1/repos/{org}/{repo}-agents"

# Create the fork (HTTP 202 = accepted, HTTP 409 = already exists)
curl -sk -X POST \
  -H "Authorization: token $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"organization":"{org}","name":"{repo}-agents"}' \
  "https://gitea.oreillyit.nz/api/v1/repos/{org}/{repo}/forks"
```

### Naming Convention

| Main repo | Agent repo |
|---|---|
| `skynet/agent-runtimes` | `skynet/agent-runtimes-agents` |
| `skynet/brainiac-app` | `skynet/brainiac-app-agents` |

Always use the `-agents` suffix. The fork relationship enables cross-fork PRs.

### Workspace Layout

```
/workspace/
├── reference/              # Owned by root — read-only (agent gets error if they try to write)
│   ├── main/               # Main repo default branch
│   ├── plan-opus-4f3a/     # Another agent's output branch (if needed)
│   └── best-practices/     # Best practices repo (if cloned)
└── working/                # Agent's branch of the agent repo (read-write)
    ├── CLAUDE.md           # Project code from the fork
    ├── spec/
    └── results/            # Starts empty — guaranteed output location
        ├── output.md
        ├── session-log.md
        └── changelog.md
```

- **`/workspace/reference/`** — each subdirectory is a separate clone. Owned by root so agents get immediate permission errors if they try to modify.
- **`/workspace/working/`** — the agent's branch. All changes auto-committed on exit.
- **`/workspace/working/results/`** — always starts empty. Write outputs here.

### Branch Naming

- **With workflow context**: `{workflow_id}-{stage}-{short_task_id}` (e.g., `f58-plan-4f3a1b2c`)
- **Without workflow context**: `task-{short_task_id}` (e.g., `task-4f3a1b2c`)

### Auto-Commit (Finalize Phase)

After the agent exits (success or failure), the entrypoint runs finalize scripts:

1. Check for changes in `/workspace/working/`
2. `git add -A && git commit` with message: `"Agent task {id} ({status}): {prompt_summary}"`
3. `git push origin {branch}`
4. Write metadata to `/workspace/.agent-output/ci_metadata.json`

Finalize runs unconditionally — partial work from failed agents is preserved. Finalize failure does not override the agent's exit code.

### Metadata Propagation

After container exit, the dispatcher reads `ci_metadata.json` and reports to CP:

```json
{
  "agent_branch": "task-4f3a1b2c",
  "agent_sha": "a1b2c3d4...",
  "agent_repo_url": "git@gitea.oreillyit.nz:skynet/agent-runtimes-agents.git",
  "agent_branch_pushed": true
}
```

Stored in `CPTask.metadata` (JSONB dict). Query via `GET /tasks/{id}`.

### Credential Separation

| Actor | Credential | Purpose |
|---|---|---|
| Dispatcher | `GITEA_API_TOKEN` (read-only) | Verify fork and branches exist |
| Agent container | SSH key (via gitea-ssh harness) | Clone repos, push branches |

The dispatcher verifies prerequisites but doesn't create forks or branches.

### Retry Behaviour

On retry, the agent gets a fresh workspace (clean checkout from base branch). The previous attempt's branch is cloned into `/workspace/reference/previous-attempt/` with a note in CLAUDE.md that the last attempt was incomplete and any work it created can be found there.

### Cross-Repo PRs

When an agent working in the agent repo needs to create a PR against the main repo, Gitea supports cross-fork PRs natively. This is not automated initially — escalate to a human or handle on demand.

### Branch Cleanup

Out of scope for initial implementation. The `task-` prefix and deterministic naming make automated pruning straightforward when needed.

---

## Known Agent Repos

| Project | Agent Repo | Created |
|---|---|---|
| `skynet/agent-runtimes` | `skynet/agent-runtimes-agents` | 2026-04-05 |

---

## Quick Reference

```bash
# Port-forward to CP
kubectl port-forward -n agent-runtimes svc/controlplane 8100:8100 &

# Submit a task
curl -s -X POST http://localhost:8100/tasks -H "Content-Type: application/json" \
  -d '{"name":"my-task","project_id":"my-project","prompt":"...","harness":"planning-opus-repo/v1","runtime":{"cli":"claude"},"pre_actions":[{"type":"clone","repo":"git@gitea.oreillyit.nz-ai-enablement:skynet/my-project.git"}]}'

# Monitor
~/dev/claude/projects/agent-runtimes/scripts/agent-monitor --filter "project=my-project" --filter "age<20m"

# Check result
curl -s http://localhost:8100/tasks/{id} | python3 -m json.tool

# Cancel
curl -s -X DELETE http://localhost:8100/tasks/{id}
```