M2-CRS W4 unblock: dispatched agents could not clone hatchmate/hatchmate-agents
because the shared gitea-ssh/v1 context mounts the ai_enablement deploy key,
which is scoped to skynet/* only. Same bug would hit every project whose
working repo lives outside the skynet org.
This implements the per-org pattern the framework already uses for accelerators
(gitea-ssh-accelerators/v1) and homelab (gitea-ssh-homelab/v1):
- harnesses/contexts/gitea-ssh-hatchmate/v1/{init.sh,harness.yaml} — NEW
forge-credential context mirroring accelerators, with account_ref
'hatchmate.space:platform' and write_targets covering hatchmate/*. The
init.sh stages the ESO-mounted private key to ~/.ssh/gitea-oreillyit-nz-hatchmate
(matching the ssh alias declared in harness.yaml). URL host pattern
'gitea.oreillyit.nz-hatchmate' (per-org alias) drives SSH config block
selection — already auto-written by harness_init.py from each layer's
ssh_hosts: declarations.
- harnesses/composites/code-*-tdd-repo-hatchmate/v1/ — NEW composites for
fable/minimax/airouter/opus that compose BOTH gitea-ssh/v1 (for reference
clones, skynet org) AND gitea-ssh-hatchmate/v1 (for working-repo clone +
push, hatchmate org). Composites are unchanged for non-hatchmate projects.
- tests/invariants/test_forge_credentials_declare_write_targets.py — the
existing xfail(strict=True) marker was wrong: the 4 existing layers all
declare write_targets today, so the test was XPASSing. Removed the
xfail marker so HC-11 is now a real positive gate (5 layers instead
of 4). This was a pre-existing test bug surfaced by the 5th layer.
W4 next step: dispatcher's account_ref resolution needs the hatchmate CP
account + grant to be created (via the new scripts/provision-hatchmate-fork-key.sh
in the hatchmate app repo).
Helper tests/invariants/forge_credentials.py detects credential carriers via
mounts[]/env/secrets_required[]/ssh_hosts[]; today gitea-admin, gitea-ssh,
gitea-ssh-accelerators, gitea-ssh-homelab are found and none declare
write_targets — the invariant xfails(strict) until the CRS-103 rollout stamps them.
Claude-Session: https://claude.ai/code/session_018LRh1S2bHJYhEgrMu3UJx8
Bug 57d8a047-d26d-47ce-ae6f-bc5cb98fef0e: Opus-tier tasks 404'd because
models/claude-opus-4.yaml pinned default_flags.model to "claude-opus-4-5",
a retired ID. models/claude-sonnet-4.yaml was pinned to a dated snapshot
("claude-sonnet-4-20250514") with the same problem.
Slug strategy: the registry slug (filename / RegistryEntry.name) doubles
as the literal --model argument on the workflow-template pass-through path
(task-templates/workflow/*.yaml `model:` field -> compiler WF-COMP-18 ->
payload.runtime.model -> entrypoint/runners/claude.py `--model`), which has
no translation step through this file's default_flags. So rather than
patch default_flags.model in place (leaving the slug stale and the
pass-through bug latent), rename the slug to match the current API ID:
- models/claude-opus-4.yaml, model-registry/claude-opus-4.yaml
-> claude-opus-5.yaml (name/endpoint/default_flags.model: claude-opus-5)
- models/claude-sonnet-4.yaml, model-registry/claude-sonnet-4.yaml
-> claude-sonnet-5.yaml (claude-sonnet-5)
- model-registry/claude-haiku-4.yaml -> claude-haiku-4-5.yaml
(claude-haiku-4-5); added the previously-missing models/claude-haiku-4-5.yaml
so the claude runner can actually resolve this endpoint
- model-registry/harness-roles.yaml, model-registry/tiers.yaml,
workflows/smart-agent.yaml, task-templates/workflow/{spec,plan}-review-loop@1.yaml:
updated every claude-opus-4 / claude-sonnet-4 / claude-haiku-4 reference
to the new slugs
- models/openrouter-sonnet.yaml: dated snapshot "claude-sonnet-4-20250514"
-> "claude-sonnet-5" (third-party router proxying the same retired ID)
- refreshed cost_per_1k_tokens on the three renamed registry entries to
current published pricing
Claude-Session: https://claude.ai/code/session_015nXvGNNNixJPFwZKZHNyha
uv is now pre-baked into the base image
(agent-runtimes/images/base/Dockerfile) and is faster than pip with
better retry on transient network failures. Closes the last runtime
hang vector for project-deps install — apt install was removed 2026-09-06
in 0c57972f-pre-bake work but the pip install -e line in init.sh
remained. uv pip install is the equivalent operation with the new
pre-baked tooling.
Refs F-WE-PDIV/RCA-2026-09-04-zombies.
verdict-tx (kind: script) never worked: script nodes are harness-exempt
(WF-COMP-6/MN-38) and run in their own empty container, so it could
never credentially read /workspace/verdict from the agent's node.
Option 3: give the agent node's own composite (planning-sonnet-repo) a
cp-service-token/v1 layer, and have the agent PATCH its own item's
flow_state (verified|rejected) to the CP directly as its last step,
before task_complete. Removes verdict-tx entirely.
See agent-runtimes work item 88ecd23e and spec/typed-workflows.md
WF-COMP-25 (agent-runtimes commit 8972d266) for the finalize.py
precedence fix this depends on (ambiguous pipe-separated output.state
must never be auto-PATCHed over the agent's own value).
Claude-Session: https://claude.ai/code/session_01U3biMxRhFGkwtf4jVTNjcj
Minimal harness for kind: customer_script nodes (agent-runtimes
spec/tenant-namespaces.md SCR-1..SCR-6). No LLM turn - runs via the
"shell" runner, agent-repo/v1 clone access only. Per-script credential
needs are deliberately not baked into this base (SCR-6: narrow,
per-script review) - compose a customer-script-<need>/v1 variant
layering whichever specific credential context a given script actually
needs, rather than widening this shared base.
Without this commit, kind: customer_script was not dispatchable from
any machine other than the one it was authored on - the harness lookup
against a fresh CRS sync would fail.
Claude-Session: https://claude.ai/code/session_012arAwUuBQrpnMzJ36mEVBk
Attempted to wire verdict-tx to clone the agent's branch and read
scratchpad/verdict (the new cross-node artifact convention,
agent-repo/v1 init.sh) instead of the ephemeral /workspace/verdict.
Reverted: script tasks (dispatcher/task_manager.py::_run_script_task)
have no credentialed way to clone a private repo — the per-task SSH key
is provisioned through the full ESO/account harness resolution
pipeline, which script tasks deliberately bypass entirely. Left the
template on its current (still-broken-for-cross-container-reasons)
/workspace/verdict contract with a note, pending a scoped follow-up on
how script nodes get read access to the branch (new secrets plumbing,
vs. folding the state transition into the agent node's own already-
credentialed CP access).
Cross-node artifact passing within a workflow manifest — nodes each run
in their own ephemeral container/pod with no shared filesystem, so an
agent node writing e.g. a verdict to /workspace/verdict is invisible to
a downstream kind: script node in the same manifest (found live
2026-09-06, bug-verify-sonnet's verdict-tx). The git task branch is the
only channel every node in a manifest actually shares: write to
scratchpad/, commit (picked up by finalize.sh's existing git add -A),
next node clones the same branch and reads it back.
scratchpad/ is scratch, never deliverable content, and must never
survive a merge into main — a CI gate in the target repo
(agent-runtimes .gitea/workflows/build-images.yaml,
no-scratchpad-in-main) is the hard backstop.
Adds the CRS-side deliverables for agent-runtimes work item f59f0132
(ai-resolve bug-verification stage, M42 concept 7666d278). Mirrors
planning-opus-repo's layer stack with the sonnet model applied at the
node level (model: sonnet), per the established convention that the
composite name is descriptive only — the workflow node's own model:/
tier: field is what actually selects the model.
The template's terminal node runs the (separately-tracked, sibling
task ad975ac6) state-tx-verdict script — this template will not be
runnable end-to-end until that script lands, but the declarative
config itself is independently valid: schema-validates cleanly and
both eligibility scenarios from the task spec pass (verify-pending +
ai-resolve -> ok=True; missing flow_state -> state_mismatch).
Hand-sync of hatchmate-crs's plan-review-loop@1 under a project-prefixed name
so hatchmate's triggers can reference it via the global WorkflowRegistry.
Interim pattern until agent-runtimes ships project-scope template serving
via crs_manager (M42 gap: DispatchService only consults the concepts_syncer
WorkflowRegistry, not crs_manager's repo store). Template is hatchmate-specific
throughout: Fable-low + MiniMax-M3-max + Airouter-Qwen3.8-max reviewers,
hatchmate/hatchmate-agents clone URL, hatchmate-aware prompts and conventions.
Claude-Session: https://claude.ai/code/session_01PdFrhZypEDBTsbAeTJqHq1
The ESO-mounted secret is root-owned mode 0400 (correct — init.sh runs as
root before the gosu drop to the agent user). The original version of
this context pointed CP_SERVICE_TOKEN_FILE directly at that raw mount,
which the agent process (uid 1000) can never read.
Confirmed live 2026-09-03: a real scope-decompose-sonnet@1 dispatch
against concept 7666d278-ae58-4f12-990d-c4959a3e19a9 hit exactly this —
the agent correctly diagnosed 'cp_cli invocation can't proceed... token
is root-owned mode 0400 so it's not readable by the agent user', wrote
its decompose plan, but could never actually call cp-cli to create the
child tasks. The trigger engine's new side-effect verification (finalize.py,
bug 9dffc5b8) correctly caught this and refused to advance flow_state —
so this was a visible, retriable failure rather than another silent
false-positive.
Fix: init.sh restages the secret to /run/agent/cp-service-token/token,
mode 0600, owned by the agent user — mirrors
anthropic-cloud-paul-oauth/v1/init.sh's existing pattern.
CP_SERVICE_TOKEN_FILE now points at the restaged copy.
New context mounts the cp-decompose-service-token account (system
provider type, account_id 1d963673-6ac9-4f85-875a-2ce5323e76ad, owner
(cp, cp)) via ESO and exports CP_SERVICE_TOKEN_FILE pointing at the
mounted secret file. Wired into spec-writing-sonnet-repo (backs
scope-decompose-sonnet@1, which calls back POST
/v1/work-items/{id}/decompose — bug b3a96acc / fa41caac / M42 stall).
The mounted value must match the CP's CP_INTERNAL_BEARER_TOKEN
(agent-runtimes deploy repo, overlays/{production,staging}/app-secret.sops.yaml).
No init.sh needed — the mount path is static at harness-authoring time
and entrypoint/actions/decompose_work_items.py (AU-51b, agent-runtimes
commit 0fa8b760) reads CP_SERVICE_TOKEN_FILE directly, so no
transformation/staging step is required.
'requires: [claude-code]' failed composition ('Invalid harness reference
format'). Both contexts need no hard dependency — layer ordering in the
composites already guarantees the auth layer precedes them. Validated
locally with entrypoint.harness compose_harness for all 6 affected
composites this time.
Claude-Session: https://claude.ai/code/session_019tJk7P8tZzJ24PvtgoGhLN
The validator pattern-matches *TOKEN* as credential-shaped and rejected
both contexts, failing every task on the new harnesses. The env var name
is fixed by Claude Code, so deliver the thinking budget via the
settings.json env map in init.sh instead (jq-merge preserves apiKeyHelper
from the minimax/oauth layer).
Claude-Session: https://claude.ai/code/session_019tJk7P8tZzJ24PvtgoGhLN
Integration runs were burning the personal Anthropic 5-hour window
(429 storm + window drain, 2026-09-02). New env-only context
minimax-m3-max/v1 pins ANTHROPIC_MODEL=MiniMax-M3[1m], maxes the
thinking budget, and sets the 1M auto-compact window; layered on
minimax/v1 for base URL + apiKeyHelper auth.
Claude-Session: https://claude.ai/code/session_019tJk7P8tZzJ24PvtgoGhLN
All 6 concept/idea-stage templates hardcoded the literal shared branch
spec/auto-draft instead of a per-item branch. Concurrent concept items
(now routine via the M41 auto-advance sweep) all pushed to the same
branch — a git collision risk. Switched to spec/${item.uuid}, matching
the established task/${item.uuid} convention used by every task-stage
template (merge@1, scaffold@1, etc — see spec/typed-workflows.md WF-COMP
stable-per-item-branch rationale). spec-review-loop@1's 4 nodes keep the
literal branch shared ACROSS the node set (now per-item, not global) so
loop iterations still accumulate on one branch per item, per spec's
cross-iteration-artifact caveat.
Files: review-spec-arch-opus@1.yaml, spec-draft-opus@1.yaml,
spec-review-loop@1.yaml, review-spec-arch-airouter@1.yaml,
spec-draft-airouter@2.yaml, scope-decompose-sonnet@1.yaml
Was silently dispatching to Qwen3.6 via airouter despite the name;
live runs were timing out (exit_code=124), wedging four M41 auto-advance
concepts. Mirrors agent-runtimes commit acc5cbe5.
Claude-Session: https://claude.ai/code/session_012arAwUuBQrpnMzJ36mEVBk
The 237KB library was mounted at /workspace/best-practices/ in every
consuming ConfigMap but nothing in the prompt mentioned it (directory
targets never enter the context manifest). New ~90-word CLAUDE.md file
target enters the system prompt: points at INDEX.md, instructs reading
only task-relevant topics.
Claude-Session: https://claude.ac/code/session_012arAwUuBQrpnMzJ36mEVBk
- tdd-gate: pass criterion is the task's TEST_FILES scope (matches
run-ci-tests.sh M19 P-14), not the full suite — contradicted
qwen-code-methodology rule 5 in the same system prompt
- tdd-protect: tests/ path described relative to project working dir
(AGENT_WORKING_DIR is /workspace/project in repo composites)
- qwen-code-methodology: task_complete tool DOES exist — rule 8 said
the opposite; pytest examples use -q
- airouter-code-tdd template: TT-39 runtime_overrides
(max_result_bytes 24KB, max_history_turns 16)
Claude-Session: https://claude.ai/code/session_012arAwUuBQrpnMzJ36mEVBk
The python3 -c block imported internal entrypoint code directly and
pasted module paths into agent prompts — a security boundary violation.
Switch to the thin-client verb (added in agent-runtimes a116aedb).
Claude-Session: https://claude.ai/code/session_0124mK2BVfCdkELLXDCr5Zzs
All tasks dispatched to integration-direct/v1 were failing with
harness_missing_agent_repo_layer because dispatcher DSP-1 guard
requires agent-repo in the composite when AGENT_REPO_URL is set.
direct-push/v1 sets env vars (AGENT_SKIP_BRANCH_PUSH,
AGENT_EMPTY_DELIVERABLE_CHECK) that configure agent-repo's finalize.sh,
so agent-repo/v1 must precede it in the layer list.
metadata.task_description is not in the consumer snapshot — the trigger
consumer only populates metadata.automation from the outbox typed_payload.
artifacts.item_body IS populated via dispatch_service from
item_snapshot["body"] (set from typed_payload.body in the consumer,
which comes from item.get("body") in emit_work_item_event).
For M39 items the body contains the full task description, so
${artifacts.item_body} gives the agent complete implementation instructions.
${metadata.automation.test_files} and ${metadata.automation.spec_ids} are
absent from M39 work items (created from dashboard without these fields),
resulting in an empty, useless prompt:
"Implement the failing test. Test files: . Spec IDs: ."
Switch to ${metadata.task_description} — a sibling metadata key (not under
automation, so write-protected path does not apply) — populated by patching
items with their body text. The _TriggerItemView in dispatch_service.py
correctly exposes item.metadata so this resolves at compile time.
All 16 scaffolded M39 items patched with metadata.task_description via CP API.
Add direct-impl-sonnet@1, plan-review-loop@1, spec-review-loop@1,
test-write-review-loop@1 — present in agent-runtimes but missing here.
These are needed before agent-runtimes drops its baked-in copies and the
CP loads workflow templates exclusively from this CRS repo.
Remove `tags_required: [airouter-eligible]` from airouter-impl-ready so all
non-skip tasks route through the full airouter→sonnet→opus ladder automatically.
Add `airouter-skip` to airouter's tags_forbidden so opt-out tasks go directly
to sonnet from attempt 0.
Relax sonnet-impl-ready tags_forbidden (drop multi-file + algorithmic-large)
so sonnet handles complex tasks at escalation tier without template rejection.
Add opus-impl-ready@1.yaml — last-resort escalation, code-opus-tdd-repo
composite, scope_budget 5 files / 400 lines / 3600s.
MiniMax moved their Anthropic-compatible API from api.minimax.io to
api.minimaxi.chat. Update ANTHROPIC_BASE_URL in the minimax/v1 context
harness and corresponding comments in init.sh and the TODO note.
Model routing is unaffected: the API defaults unrecognised model names
(e.g. "minimax" from the registry slug) to MiniMax-M3 already.
Claude-Session: https://claude.ai/code/session_01XtWRazqRbMnMRa4iV3zbVx
W1 stage-routing work:
- model-registry/tiers.yaml: defines planning/spec-test/coding tier floors
(complexity≥9/creativity≥9/context≥9 | spec_adherence≥9/test_pass_rate≥9 |
spec_adherence≥7/test_pass_rate≥7) with qualifies_today lists
- spec-draft-airouter@2.yaml: honest name for what was spec-draft-opus@1
(always ran Qwen3.6/airouter, not Opus)
- review-spec-arch-airouter@1.yaml: honest name for review-spec-arch-opus@1
Old @1 files kept for in-flight task backward-compatibility.
Claude-Session: https://claude.ai/code/session_01B35bPAKv5uyW1F9gzMcRN7
New code-airouter-deepseekv4flash-tdd-repo/v1 composite (deepseek
methodology + the same tdd-protect/tdd-gate/python-dev/tdd-file-lock
layers as the Qwen TDD composite), plus a task-template that sets
reasoning_effort=max via the new extra_body passthrough (agent-runtimes
TT-38). Model/endpoint/model-registry entries already existed; only
the TDD-capable harness and the template were missing.
fix-test-fixture-sonnet@1, review-spec-opus@1, replan-spec-opus@1 — the
dangling escalation refs from airouter-impl-*/spec-draft/feature-delivery-loop.
Sonnet-authored, Opus-reviewed; review HIGH fixes applied (explicit model: opus
on both opus nodes — node model: sets runtime model, composite name does not).
Known gap filed in work-items epic (E1-escalation-state-gap.md): escalation
edges from non-primary states 422 state_mismatch pending escalation_input_states.
concept-intake@1: optional idea-elaboration pre-stage (output state
idea-elaborated — not yet wired into spec-draft-opus input).
All four validate against controlplane WorkflowTemplate schema.
