From bff46b91823651ebdb8b8074d6fc61b6d795b30f Mon Sep 17 00:00:00 2001 From: Paul O'Reilly Date: Sun, 5 Apr 2026 11:32:48 +1200 Subject: [PATCH] Add agent repos & container agent operations best practice Comprehensive guide covering task submission to the agent-runtimes control plane, available harnesses, monitoring, multi-model workflows, agent repo forks with workspace layout, and artifact extraction patterns. Co-Authored-By: Claude Opus 4.6 (1M context) --- BESTPRACTICES.md | 2 + agent-repos.md | 409 +++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 411 insertions(+) create mode 100644 agent-repos.md diff --git a/BESTPRACTICES.md b/BESTPRACTICES.md index cf160d6..c13a0c2 100644 --- a/BESTPRACTICES.md +++ b/BESTPRACTICES.md @@ -26,3 +26,5 @@ Generalised best practices extracted from real project work via the `/distill-be - [API Design](api-design.md) — Transport security, auth (OAuth2/JWT/mTLS), versioning, pagination, error handling, idempotency, rate limiting, input validation, zero-trust patterns - [Octopus Process Templates](octopus-process-templates.md) — OCL syntax, step template references, channel scoping, parameters, versioning, Platform Hub patterns - [LLM Code Security](llm-code-security.md) — Security vulnerabilities in AI-generated code: injection flaws, hardcoded secrets, hallucinated packages, over-permissive defaults, IaC risks, crypto mistakes, review checklist +- [CI Container Builds](ci-container-builds.md) — Registry cache with inline metadata, buildx in DinD, layer ordering, pip caching, scheduled base images +- [Agent Repos & Container Agents](agent-repos.md) — Task submission, harnesses, monitoring, multi-model workflows, agent repo forks, workspace layout, artifact extraction diff --git a/agent-repos.md b/agent-repos.md new file mode 100644 index 0000000..018e44c --- /dev/null +++ b/agent-repos.md @@ -0,0 +1,409 @@ +# Agent Repos & Container Agent Operations + +Best practices for running AI agents via the agent-runtimes control plane — submitting tasks, using agent repos for persistence, running multi-model workflows, and monitoring progress. + +## Architecture Overview + +The agent-runtimes system runs AI coding agents inside ephemeral Docker/K8s containers, orchestrated by a control plane (CP). The three-tier model: + +1. **Control Plane** — central task queue, dispatcher registry, model routing. External systems (including Claude Code sessions) submit tasks here. +2. **Dispatchers** — poll the CP for tasks, resolve harness+model, create containers. All connections outbound (works behind firewalls). +3. **Agent Containers** — ephemeral, isolated. Receive a JSON payload, execute work, exit. + +``` +You (Claude Code session) → POST /tasks → Control Plane → Dispatcher → Agent Container + ← GET /tasks/{id} ← (poll for results) +``` + +### Access + +| Environment | CP URL | +|---|---| +| K8s (port-forward) | `kubectl port-forward -n agent-runtimes svc/controlplane 8100:8100` then `http://localhost:8100` | +| K8s (MFA) | `https://agents.oreillyit.nz` (behind Authelia) | +| Local dev (Docker Compose) | `http://localhost:8100` after `docker compose up -d` | + +--- + +## Submitting Tasks + +### Minimal Task + +```bash +curl -s -X POST http://localhost:8100/tasks \ + -H "Content-Type: application/json" \ + -d '{ + "name": "Fix test_auth.py", + "project_id": "my-project", + "prompt": "Fix the failing test in test_auth.py", + "harness": "planning-opus-repo/v1", + "runtime": {"cli": "claude"}, + "pre_actions": [ + {"type": "clone", "repo": "git@gitea.oreillyit.nz-ai-enablement:skynet/my-project.git", "depth": 1} + ] + }' +``` + +### Full Task Payload + +```json +{ + "task_id": "optional-uuid", + "name": "Short name (shown in monitor)", + "project_id": "groups tasks in monitor", + "prompt": "The instruction for the agent", + "harness": "composite-harness-name/v1", + "runtime": { + "cli": "claude", + "model": "sonnet", + "timeout": 1800 + }, + "priority": 0, + "metadata": { + "workflow": "my-workflow", + "phase": "plan", + "model": "opus" + }, + "pre_actions": [ + {"type": "clone", "repo": "git@...", "branch": "main", "depth": 1} + ], + "on_success": [ + {"type": "commit_pr", "branch": "feature-branch", "title": "PR title", "base": "main"} + ], + "on_error": [ + {"type": "report", "webhook_url": "https://..."} + ], + "max_retries": 1 +} +``` + +### Key Fields + +| Field | Required | Description | +|---|---|---| +| `name` | Recommended | Short task name shown in the monitor | +| `project_id` | Recommended | Groups tasks in the monitor; enables filtering | +| `prompt` | Yes | The instruction sent to the agent | +| `harness` | Recommended | Composite harness (defines credentials, context, model provider) | +| `runtime.cli` | No | CLI runner: `claude` (default), `openai_compat`, `agentic` | +| `pre_actions` | No | Setup actions before agent runs (e.g., `clone`) | +| `on_success` | No | Post-agent actions on success (e.g., `commit_pr`, `report`) | +| `on_error` | No | Post-agent actions on failure | +| `metadata` | No | Arbitrary JSONB — used for workflow tracking, filtering | +| `priority` | No | -100 to 100 (higher = picked first, default 0) | + +### Python Task Submission + +For programmatic use from a Claude Code session: + +```python +import requests, json, uuid + +CP = "http://localhost:8100" +task_id = str(uuid.uuid4()) + +payload = { + "task_id": task_id, + "name": "My agent task", + "project_id": "my-project", + "prompt": "...", + "harness": "planning-opus-repo/v1", + "runtime": {"cli": "claude"}, + "pre_actions": [{"type": "clone", "repo": "git@gitea.oreillyit.nz-ai-enablement:skynet/my-project.git"}], + "metadata": {"workflow": "my-workflow", "phase": "plan"} +} + +resp = requests.post(f"{CP}/tasks", json=payload, timeout=10) +data = resp.json() +print(f"Submitted: {data['task_id']}") +``` + +--- + +## Available Harnesses + +Harnesses define what credentials, context, and model provider an agent gets. Composites combine multiple layers. + +### Composite Harnesses (ready to use) + +| Harness | Model Provider | Capabilities | +|---|---|---| +| `planning-opus-repo/v1` | Anthropic Claude (subscription) | Planning context + SSH clone | +| `planning-minimax-repo/v1` | MiniMax | Planning context + SSH clone | +| `python-code-review/v1` | Default (Anthropic) | Python dev tools + Gitea admin | + +### Context Layers (building blocks) + +| Layer | What it provides | +|---|---| +| `planning/v1` | Best-practices files for spec/plan writing | +| `gitea-ssh/v1` | SSH key for clone/push to Gitea | +| `gitea-admin/v1` | Gitea admin context (SSH + API token) | +| `anthropic-cloud/v1` | Anthropic API (subscription pricing) | +| `minimax/v1` | MiniMax API (SOPS-encrypted credentials) | +| `code-methodology/v1` | Coding methodology CLAUDE.md | + +### Capability Layers + +| Layer | What it provides | +|---|---| +| `python-dev/v1` | pytest, ruff, mypy, hypothesis, uv | + +--- + +## Monitoring Tasks + +### agent-monitor (terminal UI) + +```bash +# Live view, filtered to your project +~/dev/claude/projects/agent-runtimes/scripts/agent-monitor --filter "project=my-project" --filter "age<20m" + +# Single snapshot +~/dev/claude/projects/agent-runtimes/scripts/agent-monitor --once + +# All running tasks +~/dev/claude/projects/agent-runtimes/scripts/agent-monitor --filter "state=running" +``` + +### API Queries + +```bash +CP=http://localhost:8100 + +# Check task status +curl -s "$CP/tasks/{task_id}" | python3 -m json.tool + +# List tasks (most recent) +curl -s "$CP/tasks?limit=10" | python3 -m json.tool + +# Cancel a task +curl -s -X DELETE "$CP/tasks/{task_id}" +``` + +### Extracting Agent Output from Logs + +Agents write to `/workspace/.agent-output/output.md`. To extract this from stream-json logs: + +```python +import requests, json + +def extract_output(cp_url, task_id): + """Extract output.md content from completed task logs.""" + r = requests.get(f"{cp_url}/tasks/{task_id}", timeout=10).json() + logs = r.get("logs", "") or "" + output = None + for line in logs.split("\n"): + if not line.startswith("{"): + continue + try: + event = json.loads(line) + except: + continue + if event.get("type") == "assistant": + for block in event.get("message", {}).get("content", []): + if isinstance(block, dict) and block.get("type") == "tool_use": + if block.get("name") == "Write" and "output.md" in str(block.get("input", {}).get("file_path", "")): + output = block["input"]["content"] + return output +``` + +--- + +## Workflows (Multi-Model Spec Planning) + +Workflow templates define multi-step DAGs where different models collaborate, cross-review, and a human resolves disputes. + +### spec-planning v3 + +The flagship workflow for spec development. 14-node DAG: + +``` +Phase 0: interview_a + interview_b (parallel, different models) +Phase 0.5: consolidate_questions + ↓ HUMAN GATE — answer questions ↓ +Phase 1: plan_a + plan_b (parallel, different models) +Phase 2: 6 cross-reviews (spec/security/TDD × 2 models, each reviews OTHER's plan) +Phase 3: escalate (surfaces disagreements, recommends best model) + ↓ HUMAN GATE — resolve disputes ↓ +Phase 4: synthesize (best model writes final spec) +``` + +### Running a Workflow Manually + +Since the CP doesn't yet expand workflows natively, run each phase from a Claude Code session: + +1. **Render prompts** — substitute `{{ task_description }}`, `{{ scope_notes }}`, etc. from the template YAML +2. **Submit tasks** — POST to CP with rendered prompts, correct harness per model +3. **Wait** — poll with agent-monitor or curl +4. **Extract artifacts** — read output.md from completed task logs +5. **Inject artifacts** — replace `<>` sentinels in next phase's prompts +6. **Human gates** — present escalation output to user, collect answers, append to artifacts +7. **Repeat** for each phase + +### Artifact Passing Between Stages + +When a downstream task needs an upstream task's output: + +```python +# Extract upstream output +upstream_output = extract_output(CP, upstream_task_id) + +# Build downstream prompt with artifact injected +downstream_prompt = f""" +## Plan A (from upstream) +{upstream_output} + +## Your Task +Review the above plan for security issues... +""" +``` + +### Workflow Templates Location + +Templates live in `~/dev/claude/projects/agent-runtimes/workflows/`: + +| Template | Description | +|---|---| +| `spec-planning.yaml` | Full 14-node spec planning with cross-model review | +| `comparative-plan.yaml` | Simpler 5-node comparative planning | + +--- + +## Agent Repos — Git-Based Persistence + +### What Is an Agent Repo + +An agent repo is a **Gitea fork** of a project's main repo, named `{repo}-agents` (e.g., `agent-runtimes-agents`). Agents work on task-specific branches in the fork. All agent output is auto-committed to git before the container exits, so nothing is lost when ephemeral containers are removed. + +### Creating an Agent Repo + +One-time setup per project. The fork must exist before agents can use it. + +```bash +# Authenticate with the ai_admin token (see ~/dev/claude/secrets/gitea/ai_admin) +TOKEN="" + +# Check if fork exists (HTTP 200 = yes, 404 = no) +curl -sk -H "Authorization: token $TOKEN" \ + "https://gitea.oreillyit.nz/api/v1/repos/{org}/{repo}-agents" + +# Create the fork (HTTP 202 = accepted, HTTP 409 = already exists) +curl -sk -X POST \ + -H "Authorization: token $TOKEN" \ + -H "Content-Type: application/json" \ + -d '{"organization":"{org}","name":"{repo}-agents"}' \ + "https://gitea.oreillyit.nz/api/v1/repos/{org}/{repo}/forks" +``` + +### Naming Convention + +| Main repo | Agent repo | +|---|---| +| `skynet/agent-runtimes` | `skynet/agent-runtimes-agents` | +| `skynet/brainiac-app` | `skynet/brainiac-app-agents` | + +Always use the `-agents` suffix. The fork relationship enables cross-fork PRs. + +### Workspace Layout + +``` +/workspace/ +├── reference/ # Owned by root — read-only (agent gets error if they try to write) +│ ├── main/ # Main repo default branch +│ ├── plan-opus-4f3a/ # Another agent's output branch (if needed) +│ └── best-practices/ # Best practices repo (if cloned) +└── working/ # Agent's branch of the agent repo (read-write) + ├── CLAUDE.md # Project code from the fork + ├── spec/ + └── results/ # Starts empty — guaranteed output location + ├── output.md + ├── session-log.md + └── changelog.md +``` + +- **`/workspace/reference/`** — each subdirectory is a separate clone. Owned by root so agents get immediate permission errors if they try to modify. +- **`/workspace/working/`** — the agent's branch. All changes auto-committed on exit. +- **`/workspace/working/results/`** — always starts empty. Write outputs here. + +### Branch Naming + +- **With workflow context**: `{workflow_id}-{stage}-{short_task_id}` (e.g., `f58-plan-4f3a1b2c`) +- **Without workflow context**: `task-{short_task_id}` (e.g., `task-4f3a1b2c`) + +### Auto-Commit (Finalize Phase) + +After the agent exits (success or failure), the entrypoint runs finalize scripts: + +1. Check for changes in `/workspace/working/` +2. `git add -A && git commit` with message: `"Agent task {id} ({status}): {prompt_summary}"` +3. `git push origin {branch}` +4. Write metadata to `/workspace/.agent-output/ci_metadata.json` + +Finalize runs unconditionally — partial work from failed agents is preserved. Finalize failure does not override the agent's exit code. + +### Metadata Propagation + +After container exit, the dispatcher reads `ci_metadata.json` and reports to CP: + +```json +{ + "agent_branch": "task-4f3a1b2c", + "agent_sha": "a1b2c3d4...", + "agent_repo_url": "git@gitea.oreillyit.nz:skynet/agent-runtimes-agents.git", + "agent_branch_pushed": true +} +``` + +Stored in `CPTask.metadata` (JSONB dict). Query via `GET /tasks/{id}`. + +### Credential Separation + +| Actor | Credential | Purpose | +|---|---|---| +| Dispatcher | `GITEA_API_TOKEN` (read-only) | Verify fork and branches exist | +| Agent container | SSH key (via gitea-ssh harness) | Clone repos, push branches | + +The dispatcher verifies prerequisites but doesn't create forks or branches. + +### Retry Behaviour + +On retry, the agent gets a fresh workspace (clean checkout from base branch). The previous attempt's branch is cloned into `/workspace/reference/previous-attempt/` with a note in CLAUDE.md that the last attempt was incomplete and any work it created can be found there. + +### Cross-Repo PRs + +When an agent working in the agent repo needs to create a PR against the main repo, Gitea supports cross-fork PRs natively. This is not automated initially — escalate to a human or handle on demand. + +### Branch Cleanup + +Out of scope for initial implementation. The `task-` prefix and deterministic naming make automated pruning straightforward when needed. + +--- + +## Known Agent Repos + +| Project | Agent Repo | Created | +|---|---|---| +| `skynet/agent-runtimes` | `skynet/agent-runtimes-agents` | 2026-04-05 | + +--- + +## Quick Reference + +```bash +# Port-forward to CP +kubectl port-forward -n agent-runtimes svc/controlplane 8100:8100 & + +# Submit a task +curl -s -X POST http://localhost:8100/tasks -H "Content-Type: application/json" \ + -d '{"name":"my-task","project_id":"my-project","prompt":"...","harness":"planning-opus-repo/v1","runtime":{"cli":"claude"},"pre_actions":[{"type":"clone","repo":"git@gitea.oreillyit.nz-ai-enablement:skynet/my-project.git"}]}' + +# Monitor +~/dev/claude/projects/agent-runtimes/scripts/agent-monitor --filter "project=my-project" --filter "age<20m" + +# Check result +curl -s http://localhost:8100/tasks/{id} | python3 -m json.tool + +# Cancel +curl -s -X DELETE http://localhost:8100/tasks/{id} +```