Add agent repos & container agent operations best practice
Comprehensive guide covering task submission to the agent-runtimes control plane, available harnesses, monitoring, multi-model workflows, agent repo forks with workspace layout, and artifact extraction patterns. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
Paul O'Reilly
@@ -26,3 +26,5 @@ Generalised best practices extracted from real project work via the `/distill-be
|
||||
- [API Design](api-design.md) — Transport security, auth (OAuth2/JWT/mTLS), versioning, pagination, error handling, idempotency, rate limiting, input validation, zero-trust patterns
|
||||
- [Octopus Process Templates](octopus-process-templates.md) — OCL syntax, step template references, channel scoping, parameters, versioning, Platform Hub patterns
|
||||
- [LLM Code Security](llm-code-security.md) — Security vulnerabilities in AI-generated code: injection flaws, hardcoded secrets, hallucinated packages, over-permissive defaults, IaC risks, crypto mistakes, review checklist
|
||||
- [CI Container Builds](ci-container-builds.md) — Registry cache with inline metadata, buildx in DinD, layer ordering, pip caching, scheduled base images
|
||||
- [Agent Repos & Container Agents](agent-repos.md) — Task submission, harnesses, monitoring, multi-model workflows, agent repo forks, workspace layout, artifact extraction
|
||||
|
||||
409
agent-repos.md
Normal file
409
agent-repos.md
Normal file
@@ -0,0 +1,409 @@
|
||||
# Agent Repos & Container Agent Operations
|
||||
|
||||
Best practices for running AI agents via the agent-runtimes control plane — submitting tasks, using agent repos for persistence, running multi-model workflows, and monitoring progress.
|
||||
|
||||
## Architecture Overview
|
||||
|
||||
The agent-runtimes system runs AI coding agents inside ephemeral Docker/K8s containers, orchestrated by a control plane (CP). The three-tier model:
|
||||
|
||||
1. **Control Plane** — central task queue, dispatcher registry, model routing. External systems (including Claude Code sessions) submit tasks here.
|
||||
2. **Dispatchers** — poll the CP for tasks, resolve harness+model, create containers. All connections outbound (works behind firewalls).
|
||||
3. **Agent Containers** — ephemeral, isolated. Receive a JSON payload, execute work, exit.
|
||||
|
||||
```
|
||||
You (Claude Code session) → POST /tasks → Control Plane → Dispatcher → Agent Container
|
||||
← GET /tasks/{id} ← (poll for results)
|
||||
```
|
||||
|
||||
### Access
|
||||
|
||||
| Environment | CP URL |
|
||||
|---|---|
|
||||
| K8s (port-forward) | `kubectl port-forward -n agent-runtimes svc/controlplane 8100:8100` then `http://localhost:8100` |
|
||||
| K8s (MFA) | `https://agents.oreillyit.nz` (behind Authelia) |
|
||||
| Local dev (Docker Compose) | `http://localhost:8100` after `docker compose up -d` |
|
||||
|
||||
---
|
||||
|
||||
## Submitting Tasks
|
||||
|
||||
### Minimal Task
|
||||
|
||||
```bash
|
||||
curl -s -X POST http://localhost:8100/tasks \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"name": "Fix test_auth.py",
|
||||
"project_id": "my-project",
|
||||
"prompt": "Fix the failing test in test_auth.py",
|
||||
"harness": "planning-opus-repo/v1",
|
||||
"runtime": {"cli": "claude"},
|
||||
"pre_actions": [
|
||||
{"type": "clone", "repo": "git@gitea.oreillyit.nz-ai-enablement:skynet/my-project.git", "depth": 1}
|
||||
]
|
||||
}'
|
||||
```
|
||||
|
||||
### Full Task Payload
|
||||
|
||||
```json
|
||||
{
|
||||
"task_id": "optional-uuid",
|
||||
"name": "Short name (shown in monitor)",
|
||||
"project_id": "groups tasks in monitor",
|
||||
"prompt": "The instruction for the agent",
|
||||
"harness": "composite-harness-name/v1",
|
||||
"runtime": {
|
||||
"cli": "claude",
|
||||
"model": "sonnet",
|
||||
"timeout": 1800
|
||||
},
|
||||
"priority": 0,
|
||||
"metadata": {
|
||||
"workflow": "my-workflow",
|
||||
"phase": "plan",
|
||||
"model": "opus"
|
||||
},
|
||||
"pre_actions": [
|
||||
{"type": "clone", "repo": "git@...", "branch": "main", "depth": 1}
|
||||
],
|
||||
"on_success": [
|
||||
{"type": "commit_pr", "branch": "feature-branch", "title": "PR title", "base": "main"}
|
||||
],
|
||||
"on_error": [
|
||||
{"type": "report", "webhook_url": "https://..."}
|
||||
],
|
||||
"max_retries": 1
|
||||
}
|
||||
```
|
||||
|
||||
### Key Fields
|
||||
|
||||
| Field | Required | Description |
|
||||
|---|---|---|
|
||||
| `name` | Recommended | Short task name shown in the monitor |
|
||||
| `project_id` | Recommended | Groups tasks in the monitor; enables filtering |
|
||||
| `prompt` | Yes | The instruction sent to the agent |
|
||||
| `harness` | Recommended | Composite harness (defines credentials, context, model provider) |
|
||||
| `runtime.cli` | No | CLI runner: `claude` (default), `openai_compat`, `agentic` |
|
||||
| `pre_actions` | No | Setup actions before agent runs (e.g., `clone`) |
|
||||
| `on_success` | No | Post-agent actions on success (e.g., `commit_pr`, `report`) |
|
||||
| `on_error` | No | Post-agent actions on failure |
|
||||
| `metadata` | No | Arbitrary JSONB — used for workflow tracking, filtering |
|
||||
| `priority` | No | -100 to 100 (higher = picked first, default 0) |
|
||||
|
||||
### Python Task Submission
|
||||
|
||||
For programmatic use from a Claude Code session:
|
||||
|
||||
```python
|
||||
import requests, json, uuid
|
||||
|
||||
CP = "http://localhost:8100"
|
||||
task_id = str(uuid.uuid4())
|
||||
|
||||
payload = {
|
||||
"task_id": task_id,
|
||||
"name": "My agent task",
|
||||
"project_id": "my-project",
|
||||
"prompt": "...",
|
||||
"harness": "planning-opus-repo/v1",
|
||||
"runtime": {"cli": "claude"},
|
||||
"pre_actions": [{"type": "clone", "repo": "git@gitea.oreillyit.nz-ai-enablement:skynet/my-project.git"}],
|
||||
"metadata": {"workflow": "my-workflow", "phase": "plan"}
|
||||
}
|
||||
|
||||
resp = requests.post(f"{CP}/tasks", json=payload, timeout=10)
|
||||
data = resp.json()
|
||||
print(f"Submitted: {data['task_id']}")
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Available Harnesses
|
||||
|
||||
Harnesses define what credentials, context, and model provider an agent gets. Composites combine multiple layers.
|
||||
|
||||
### Composite Harnesses (ready to use)
|
||||
|
||||
| Harness | Model Provider | Capabilities |
|
||||
|---|---|---|
|
||||
| `planning-opus-repo/v1` | Anthropic Claude (subscription) | Planning context + SSH clone |
|
||||
| `planning-minimax-repo/v1` | MiniMax | Planning context + SSH clone |
|
||||
| `python-code-review/v1` | Default (Anthropic) | Python dev tools + Gitea admin |
|
||||
|
||||
### Context Layers (building blocks)
|
||||
|
||||
| Layer | What it provides |
|
||||
|---|---|
|
||||
| `planning/v1` | Best-practices files for spec/plan writing |
|
||||
| `gitea-ssh/v1` | SSH key for clone/push to Gitea |
|
||||
| `gitea-admin/v1` | Gitea admin context (SSH + API token) |
|
||||
| `anthropic-cloud/v1` | Anthropic API (subscription pricing) |
|
||||
| `minimax/v1` | MiniMax API (SOPS-encrypted credentials) |
|
||||
| `code-methodology/v1` | Coding methodology CLAUDE.md |
|
||||
|
||||
### Capability Layers
|
||||
|
||||
| Layer | What it provides |
|
||||
|---|---|
|
||||
| `python-dev/v1` | pytest, ruff, mypy, hypothesis, uv |
|
||||
|
||||
---
|
||||
|
||||
## Monitoring Tasks
|
||||
|
||||
### agent-monitor (terminal UI)
|
||||
|
||||
```bash
|
||||
# Live view, filtered to your project
|
||||
~/dev/claude/projects/agent-runtimes/scripts/agent-monitor --filter "project=my-project" --filter "age<20m"
|
||||
|
||||
# Single snapshot
|
||||
~/dev/claude/projects/agent-runtimes/scripts/agent-monitor --once
|
||||
|
||||
# All running tasks
|
||||
~/dev/claude/projects/agent-runtimes/scripts/agent-monitor --filter "state=running"
|
||||
```
|
||||
|
||||
### API Queries
|
||||
|
||||
```bash
|
||||
CP=http://localhost:8100
|
||||
|
||||
# Check task status
|
||||
curl -s "$CP/tasks/{task_id}" | python3 -m json.tool
|
||||
|
||||
# List tasks (most recent)
|
||||
curl -s "$CP/tasks?limit=10" | python3 -m json.tool
|
||||
|
||||
# Cancel a task
|
||||
curl -s -X DELETE "$CP/tasks/{task_id}"
|
||||
```
|
||||
|
||||
### Extracting Agent Output from Logs
|
||||
|
||||
Agents write to `/workspace/.agent-output/output.md`. To extract this from stream-json logs:
|
||||
|
||||
```python
|
||||
import requests, json
|
||||
|
||||
def extract_output(cp_url, task_id):
|
||||
"""Extract output.md content from completed task logs."""
|
||||
r = requests.get(f"{cp_url}/tasks/{task_id}", timeout=10).json()
|
||||
logs = r.get("logs", "") or ""
|
||||
output = None
|
||||
for line in logs.split("\n"):
|
||||
if not line.startswith("{"):
|
||||
continue
|
||||
try:
|
||||
event = json.loads(line)
|
||||
except:
|
||||
continue
|
||||
if event.get("type") == "assistant":
|
||||
for block in event.get("message", {}).get("content", []):
|
||||
if isinstance(block, dict) and block.get("type") == "tool_use":
|
||||
if block.get("name") == "Write" and "output.md" in str(block.get("input", {}).get("file_path", "")):
|
||||
output = block["input"]["content"]
|
||||
return output
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Workflows (Multi-Model Spec Planning)
|
||||
|
||||
Workflow templates define multi-step DAGs where different models collaborate, cross-review, and a human resolves disputes.
|
||||
|
||||
### spec-planning v3
|
||||
|
||||
The flagship workflow for spec development. 14-node DAG:
|
||||
|
||||
```
|
||||
Phase 0: interview_a + interview_b (parallel, different models)
|
||||
Phase 0.5: consolidate_questions
|
||||
↓ HUMAN GATE — answer questions ↓
|
||||
Phase 1: plan_a + plan_b (parallel, different models)
|
||||
Phase 2: 6 cross-reviews (spec/security/TDD × 2 models, each reviews OTHER's plan)
|
||||
Phase 3: escalate (surfaces disagreements, recommends best model)
|
||||
↓ HUMAN GATE — resolve disputes ↓
|
||||
Phase 4: synthesize (best model writes final spec)
|
||||
```
|
||||
|
||||
### Running a Workflow Manually
|
||||
|
||||
Since the CP doesn't yet expand workflows natively, run each phase from a Claude Code session:
|
||||
|
||||
1. **Render prompts** — substitute `{{ task_description }}`, `{{ scope_notes }}`, etc. from the template YAML
|
||||
2. **Submit tasks** — POST to CP with rendered prompts, correct harness per model
|
||||
3. **Wait** — poll with agent-monitor or curl
|
||||
4. **Extract artifacts** — read output.md from completed task logs
|
||||
5. **Inject artifacts** — replace `<<ARTIFACT:node_id:key>>` sentinels in next phase's prompts
|
||||
6. **Human gates** — present escalation output to user, collect answers, append to artifacts
|
||||
7. **Repeat** for each phase
|
||||
|
||||
### Artifact Passing Between Stages
|
||||
|
||||
When a downstream task needs an upstream task's output:
|
||||
|
||||
```python
|
||||
# Extract upstream output
|
||||
upstream_output = extract_output(CP, upstream_task_id)
|
||||
|
||||
# Build downstream prompt with artifact injected
|
||||
downstream_prompt = f"""
|
||||
## Plan A (from upstream)
|
||||
{upstream_output}
|
||||
|
||||
## Your Task
|
||||
Review the above plan for security issues...
|
||||
"""
|
||||
```
|
||||
|
||||
### Workflow Templates Location
|
||||
|
||||
Templates live in `~/dev/claude/projects/agent-runtimes/workflows/`:
|
||||
|
||||
| Template | Description |
|
||||
|---|---|
|
||||
| `spec-planning.yaml` | Full 14-node spec planning with cross-model review |
|
||||
| `comparative-plan.yaml` | Simpler 5-node comparative planning |
|
||||
|
||||
---
|
||||
|
||||
## Agent Repos — Git-Based Persistence
|
||||
|
||||
### What Is an Agent Repo
|
||||
|
||||
An agent repo is a **Gitea fork** of a project's main repo, named `{repo}-agents` (e.g., `agent-runtimes-agents`). Agents work on task-specific branches in the fork. All agent output is auto-committed to git before the container exits, so nothing is lost when ephemeral containers are removed.
|
||||
|
||||
### Creating an Agent Repo
|
||||
|
||||
One-time setup per project. The fork must exist before agents can use it.
|
||||
|
||||
```bash
|
||||
# Authenticate with the ai_admin token (see ~/dev/claude/secrets/gitea/ai_admin)
|
||||
TOKEN="<token from secrets>"
|
||||
|
||||
# Check if fork exists (HTTP 200 = yes, 404 = no)
|
||||
curl -sk -H "Authorization: token $TOKEN" \
|
||||
"https://gitea.oreillyit.nz/api/v1/repos/{org}/{repo}-agents"
|
||||
|
||||
# Create the fork (HTTP 202 = accepted, HTTP 409 = already exists)
|
||||
curl -sk -X POST \
|
||||
-H "Authorization: token $TOKEN" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"organization":"{org}","name":"{repo}-agents"}' \
|
||||
"https://gitea.oreillyit.nz/api/v1/repos/{org}/{repo}/forks"
|
||||
```
|
||||
|
||||
### Naming Convention
|
||||
|
||||
| Main repo | Agent repo |
|
||||
|---|---|
|
||||
| `skynet/agent-runtimes` | `skynet/agent-runtimes-agents` |
|
||||
| `skynet/brainiac-app` | `skynet/brainiac-app-agents` |
|
||||
|
||||
Always use the `-agents` suffix. The fork relationship enables cross-fork PRs.
|
||||
|
||||
### Workspace Layout
|
||||
|
||||
```
|
||||
/workspace/
|
||||
├── reference/ # Owned by root — read-only (agent gets error if they try to write)
|
||||
│ ├── main/ # Main repo default branch
|
||||
│ ├── plan-opus-4f3a/ # Another agent's output branch (if needed)
|
||||
│ └── best-practices/ # Best practices repo (if cloned)
|
||||
└── working/ # Agent's branch of the agent repo (read-write)
|
||||
├── CLAUDE.md # Project code from the fork
|
||||
├── spec/
|
||||
└── results/ # Starts empty — guaranteed output location
|
||||
├── output.md
|
||||
├── session-log.md
|
||||
└── changelog.md
|
||||
```
|
||||
|
||||
- **`/workspace/reference/`** — each subdirectory is a separate clone. Owned by root so agents get immediate permission errors if they try to modify.
|
||||
- **`/workspace/working/`** — the agent's branch. All changes auto-committed on exit.
|
||||
- **`/workspace/working/results/`** — always starts empty. Write outputs here.
|
||||
|
||||
### Branch Naming
|
||||
|
||||
- **With workflow context**: `{workflow_id}-{stage}-{short_task_id}` (e.g., `f58-plan-4f3a1b2c`)
|
||||
- **Without workflow context**: `task-{short_task_id}` (e.g., `task-4f3a1b2c`)
|
||||
|
||||
### Auto-Commit (Finalize Phase)
|
||||
|
||||
After the agent exits (success or failure), the entrypoint runs finalize scripts:
|
||||
|
||||
1. Check for changes in `/workspace/working/`
|
||||
2. `git add -A && git commit` with message: `"Agent task {id} ({status}): {prompt_summary}"`
|
||||
3. `git push origin {branch}`
|
||||
4. Write metadata to `/workspace/.agent-output/ci_metadata.json`
|
||||
|
||||
Finalize runs unconditionally — partial work from failed agents is preserved. Finalize failure does not override the agent's exit code.
|
||||
|
||||
### Metadata Propagation
|
||||
|
||||
After container exit, the dispatcher reads `ci_metadata.json` and reports to CP:
|
||||
|
||||
```json
|
||||
{
|
||||
"agent_branch": "task-4f3a1b2c",
|
||||
"agent_sha": "a1b2c3d4...",
|
||||
"agent_repo_url": "git@gitea.oreillyit.nz:skynet/agent-runtimes-agents.git",
|
||||
"agent_branch_pushed": true
|
||||
}
|
||||
```
|
||||
|
||||
Stored in `CPTask.metadata` (JSONB dict). Query via `GET /tasks/{id}`.
|
||||
|
||||
### Credential Separation
|
||||
|
||||
| Actor | Credential | Purpose |
|
||||
|---|---|---|
|
||||
| Dispatcher | `GITEA_API_TOKEN` (read-only) | Verify fork and branches exist |
|
||||
| Agent container | SSH key (via gitea-ssh harness) | Clone repos, push branches |
|
||||
|
||||
The dispatcher verifies prerequisites but doesn't create forks or branches.
|
||||
|
||||
### Retry Behaviour
|
||||
|
||||
On retry, the agent gets a fresh workspace (clean checkout from base branch). The previous attempt's branch is cloned into `/workspace/reference/previous-attempt/` with a note in CLAUDE.md that the last attempt was incomplete and any work it created can be found there.
|
||||
|
||||
### Cross-Repo PRs
|
||||
|
||||
When an agent working in the agent repo needs to create a PR against the main repo, Gitea supports cross-fork PRs natively. This is not automated initially — escalate to a human or handle on demand.
|
||||
|
||||
### Branch Cleanup
|
||||
|
||||
Out of scope for initial implementation. The `task-` prefix and deterministic naming make automated pruning straightforward when needed.
|
||||
|
||||
---
|
||||
|
||||
## Known Agent Repos
|
||||
|
||||
| Project | Agent Repo | Created |
|
||||
|---|---|---|
|
||||
| `skynet/agent-runtimes` | `skynet/agent-runtimes-agents` | 2026-04-05 |
|
||||
|
||||
---
|
||||
|
||||
## Quick Reference
|
||||
|
||||
```bash
|
||||
# Port-forward to CP
|
||||
kubectl port-forward -n agent-runtimes svc/controlplane 8100:8100 &
|
||||
|
||||
# Submit a task
|
||||
curl -s -X POST http://localhost:8100/tasks -H "Content-Type: application/json" \
|
||||
-d '{"name":"my-task","project_id":"my-project","prompt":"...","harness":"planning-opus-repo/v1","runtime":{"cli":"claude"},"pre_actions":[{"type":"clone","repo":"git@gitea.oreillyit.nz-ai-enablement:skynet/my-project.git"}]}'
|
||||
|
||||
# Monitor
|
||||
~/dev/claude/projects/agent-runtimes/scripts/agent-monitor --filter "project=my-project" --filter "age<20m"
|
||||
|
||||
# Check result
|
||||
curl -s http://localhost:8100/tasks/{id} | python3 -m json.tool
|
||||
|
||||
# Cancel
|
||||
curl -s -X DELETE http://localhost:8100/tasks/{id}
|
||||
```
|
||||
Reference in New Issue
Block a user