Env-only layer exporting CP_WRITE_TOKEN_FILE=/run/agent/cp-write-token/token.
The per-step wfcw_ token is delivered by the dispatcher into the script
container only (AU-55b), so the layer has no secrets_required, init script or
files. Not referenced by any composite yet; safe to merge at any time.
Claude-Session: https://claude.ai/code/session_01N218bERyNeLhbLmDRDvbWF
repo-readonly/v1 now declares write_targets: [], an ssh_hosts alias and its
own secrets_required (account platform-readonly.space:platform, a read-only
Gitea deploy key), and stages the key before cloning. workflow-script/v1 no
longer layers gitea-ssh/v1, so script nodes cannot push. Invariants updated:
credential layers in the composite must declare write_targets: [].
Bug 45a48685-23e9-4605-af1f-d93306108cef
Claude-Session: https://claude.ai/code/session_01N218bERyNeLhbLmDRDvbWF
init.sh invoked 'python3 -m entrypoint.agent_repo_base' from /workspace/project, so a project
repo's own entrypoint/ shadowed the image's and the selector crashed. Invoke the image copy by
path. scope-decompose-sonnet@2 gains AGENT_UPSTREAM_REPO_URL/BRANCH so fresh spec branches seed
from upstream main instead of the stale fork main.
Claude-Session: https://claude.ai/code/session_01N218bERyNeLhbLmDRDvbWF
M2-CRS W4 unblock: dispatched agents could not clone hatchmate/hatchmate-agents
because the shared gitea-ssh/v1 context mounts the ai_enablement deploy key,
which is scoped to skynet/* only. Same bug would hit every project whose
working repo lives outside the skynet org.
This implements the per-org pattern the framework already uses for accelerators
(gitea-ssh-accelerators/v1) and homelab (gitea-ssh-homelab/v1):
- harnesses/contexts/gitea-ssh-hatchmate/v1/{init.sh,harness.yaml} — NEW
forge-credential context mirroring accelerators, with account_ref
'hatchmate.space:platform' and write_targets covering hatchmate/*. The
init.sh stages the ESO-mounted private key to ~/.ssh/gitea-oreillyit-nz-hatchmate
(matching the ssh alias declared in harness.yaml). URL host pattern
'gitea.oreillyit.nz-hatchmate' (per-org alias) drives SSH config block
selection — already auto-written by harness_init.py from each layer's
ssh_hosts: declarations.
- harnesses/composites/code-*-tdd-repo-hatchmate/v1/ — NEW composites for
fable/minimax/airouter/opus that compose BOTH gitea-ssh/v1 (for reference
clones, skynet org) AND gitea-ssh-hatchmate/v1 (for working-repo clone +
push, hatchmate org). Composites are unchanged for non-hatchmate projects.
- tests/invariants/test_forge_credentials_declare_write_targets.py — the
existing xfail(strict=True) marker was wrong: the 4 existing layers all
declare write_targets today, so the test was XPASSing. Removed the
xfail marker so HC-11 is now a real positive gate (5 layers instead
of 4). This was a pre-existing test bug surfaced by the 5th layer.
W4 next step: dispatcher's account_ref resolution needs the hatchmate CP
account + grant to be created (via the new scripts/provision-hatchmate-fork-key.sh
in the hatchmate app repo).
uv is now pre-baked into the base image
(agent-runtimes/images/base/Dockerfile) and is faster than pip with
better retry on transient network failures. Closes the last runtime
hang vector for project-deps install — apt install was removed 2026-09-06
in 0c57972f-pre-bake work but the pip install -e line in init.sh
remained. uv pip install is the equivalent operation with the new
pre-baked tooling.
Refs F-WE-PDIV/RCA-2026-09-04-zombies.
verdict-tx (kind: script) never worked: script nodes are harness-exempt
(WF-COMP-6/MN-38) and run in their own empty container, so it could
never credentially read /workspace/verdict from the agent's node.
Option 3: give the agent node's own composite (planning-sonnet-repo) a
cp-service-token/v1 layer, and have the agent PATCH its own item's
flow_state (verified|rejected) to the CP directly as its last step,
before task_complete. Removes verdict-tx entirely.
See agent-runtimes work item 88ecd23e and spec/typed-workflows.md
WF-COMP-25 (agent-runtimes commit 8972d266) for the finalize.py
precedence fix this depends on (ambiguous pipe-separated output.state
must never be auto-PATCHed over the agent's own value).
Claude-Session: https://claude.ai/code/session_01U3biMxRhFGkwtf4jVTNjcj
Minimal harness for kind: customer_script nodes (agent-runtimes
spec/tenant-namespaces.md SCR-1..SCR-6). No LLM turn - runs via the
"shell" runner, agent-repo/v1 clone access only. Per-script credential
needs are deliberately not baked into this base (SCR-6: narrow,
per-script review) - compose a customer-script-<need>/v1 variant
layering whichever specific credential context a given script actually
needs, rather than widening this shared base.
Without this commit, kind: customer_script was not dispatchable from
any machine other than the one it was authored on - the harness lookup
against a fresh CRS sync would fail.
Claude-Session: https://claude.ai/code/session_012arAwUuBQrpnMzJ36mEVBk
Cross-node artifact passing within a workflow manifest — nodes each run
in their own ephemeral container/pod with no shared filesystem, so an
agent node writing e.g. a verdict to /workspace/verdict is invisible to
a downstream kind: script node in the same manifest (found live
2026-09-06, bug-verify-sonnet's verdict-tx). The git task branch is the
only channel every node in a manifest actually shares: write to
scratchpad/, commit (picked up by finalize.sh's existing git add -A),
next node clones the same branch and reads it back.
scratchpad/ is scratch, never deliverable content, and must never
survive a merge into main — a CI gate in the target repo
(agent-runtimes .gitea/workflows/build-images.yaml,
no-scratchpad-in-main) is the hard backstop.
Adds the CRS-side deliverables for agent-runtimes work item f59f0132
(ai-resolve bug-verification stage, M42 concept 7666d278). Mirrors
planning-opus-repo's layer stack with the sonnet model applied at the
node level (model: sonnet), per the established convention that the
composite name is descriptive only — the workflow node's own model:/
tier: field is what actually selects the model.
The template's terminal node runs the (separately-tracked, sibling
task ad975ac6) state-tx-verdict script — this template will not be
runnable end-to-end until that script lands, but the declarative
config itself is independently valid: schema-validates cleanly and
both eligibility scenarios from the task spec pass (verify-pending +
ai-resolve -> ok=True; missing flow_state -> state_mismatch).
The ESO-mounted secret is root-owned mode 0400 (correct — init.sh runs as
root before the gosu drop to the agent user). The original version of
this context pointed CP_SERVICE_TOKEN_FILE directly at that raw mount,
which the agent process (uid 1000) can never read.
Confirmed live 2026-09-03: a real scope-decompose-sonnet@1 dispatch
against concept 7666d278-ae58-4f12-990d-c4959a3e19a9 hit exactly this —
the agent correctly diagnosed 'cp_cli invocation can't proceed... token
is root-owned mode 0400 so it's not readable by the agent user', wrote
its decompose plan, but could never actually call cp-cli to create the
child tasks. The trigger engine's new side-effect verification (finalize.py,
bug 9dffc5b8) correctly caught this and refused to advance flow_state —
so this was a visible, retriable failure rather than another silent
false-positive.
Fix: init.sh restages the secret to /run/agent/cp-service-token/token,
mode 0600, owned by the agent user — mirrors
anthropic-cloud-paul-oauth/v1/init.sh's existing pattern.
CP_SERVICE_TOKEN_FILE now points at the restaged copy.
New context mounts the cp-decompose-service-token account (system
provider type, account_id 1d963673-6ac9-4f85-875a-2ce5323e76ad, owner
(cp, cp)) via ESO and exports CP_SERVICE_TOKEN_FILE pointing at the
mounted secret file. Wired into spec-writing-sonnet-repo (backs
scope-decompose-sonnet@1, which calls back POST
/v1/work-items/{id}/decompose — bug b3a96acc / fa41caac / M42 stall).
The mounted value must match the CP's CP_INTERNAL_BEARER_TOKEN
(agent-runtimes deploy repo, overlays/{production,staging}/app-secret.sops.yaml).
No init.sh needed — the mount path is static at harness-authoring time
and entrypoint/actions/decompose_work_items.py (AU-51b, agent-runtimes
commit 0fa8b760) reads CP_SERVICE_TOKEN_FILE directly, so no
transformation/staging step is required.
'requires: [claude-code]' failed composition ('Invalid harness reference
format'). Both contexts need no hard dependency — layer ordering in the
composites already guarantees the auth layer precedes them. Validated
locally with entrypoint.harness compose_harness for all 6 affected
composites this time.
Claude-Session: https://claude.ai/code/session_019tJk7P8tZzJ24PvtgoGhLN
The validator pattern-matches *TOKEN* as credential-shaped and rejected
both contexts, failing every task on the new harnesses. The env var name
is fixed by Claude Code, so deliver the thinking budget via the
settings.json env map in init.sh instead (jq-merge preserves apiKeyHelper
from the minimax/oauth layer).
Claude-Session: https://claude.ai/code/session_019tJk7P8tZzJ24PvtgoGhLN
Integration runs were burning the personal Anthropic 5-hour window
(429 storm + window drain, 2026-09-02). New env-only context
minimax-m3-max/v1 pins ANTHROPIC_MODEL=MiniMax-M3[1m], maxes the
thinking budget, and sets the 1M auto-compact window; layered on
minimax/v1 for base URL + apiKeyHelper auth.
Claude-Session: https://claude.ai/code/session_019tJk7P8tZzJ24PvtgoGhLN
The 237KB library was mounted at /workspace/best-practices/ in every
consuming ConfigMap but nothing in the prompt mentioned it (directory
targets never enter the context manifest). New ~90-word CLAUDE.md file
target enters the system prompt: points at INDEX.md, instructs reading
only task-relevant topics.
Claude-Session: https://claude.ac/code/session_012arAwUuBQrpnMzJ36mEVBk
- tdd-gate: pass criterion is the task's TEST_FILES scope (matches
run-ci-tests.sh M19 P-14), not the full suite — contradicted
qwen-code-methodology rule 5 in the same system prompt
- tdd-protect: tests/ path described relative to project working dir
(AGENT_WORKING_DIR is /workspace/project in repo composites)
- qwen-code-methodology: task_complete tool DOES exist — rule 8 said
the opposite; pytest examples use -q
- airouter-code-tdd template: TT-39 runtime_overrides
(max_result_bytes 24KB, max_history_turns 16)
Claude-Session: https://claude.ai/code/session_012arAwUuBQrpnMzJ36mEVBk
All tasks dispatched to integration-direct/v1 were failing with
harness_missing_agent_repo_layer because dispatcher DSP-1 guard
requires agent-repo in the composite when AGENT_REPO_URL is set.
direct-push/v1 sets env vars (AGENT_SKIP_BRANCH_PUSH,
AGENT_EMPTY_DELIVERABLE_CHECK) that configure agent-repo's finalize.sh,
so agent-repo/v1 must precede it in the layer list.
MiniMax moved their Anthropic-compatible API from api.minimax.io to
api.minimaxi.chat. Update ANTHROPIC_BASE_URL in the minimax/v1 context
harness and corresponding comments in init.sh and the TODO note.
Model routing is unaffected: the API defaults unrecognised model names
(e.g. "minimax" from the registry slug) to MiniMax-M3 already.
Claude-Session: https://claude.ai/code/session_01XtWRazqRbMnMRa4iV3zbVx
New code-airouter-deepseekv4flash-tdd-repo/v1 composite (deepseek
methodology + the same tdd-protect/tdd-gate/python-dev/tdd-file-lock
layers as the Qwen TDD composite), plus a task-template that sets
reasoning_effort=max via the new extra_body passthrough (agent-runtimes
TT-38). Model/endpoint/model-registry entries already existed; only
the TDD-capable harness and the template were missing.
New context: hugo-content-workspace/v1
- init.sh: clones hugo-{customer}-content at HUGO_CONTENT_BRANCH and
hugo-{customer}-integration at main via gitea-ssh-accelerators
- finalize.sh: detects changes, commits content repo first then
integration repo; skips silently if no changes
- CLAUDE.md: instructs the agent to work in /workspace/content/
New composite: hugo-content-airouter/v1
- Combines airouter/v1 + gitea-ssh-accelerators/v1 + hugo-content-workspace/v1
- Label-gated to airouter dispatchers (ESO secret required)
Used by cms-proxy /ai/{customer}/draft endpoint.
Brings the framework CRS repo up to date with all content that was
living in agent-runtimes (local-dev fallback) but hadn't been promoted.
New composites: feature-delivery-loop, integration-direct, scaffolding-repo,
sonnet-impl-narrow, sonnet-manager, test-writing-repo
New contexts: integration/v1, scaffolding/v1, sonnet-manager/v1, z-ai/v1,
airouter/v1/bin (anthropic-compat-wrapper.sh), cp-harness/v1/init.sh
New task-templates: sonnet-integrator.yaml, workflow/* (17 typed workflow
task templates for the Epic 1 pipeline)
Updated: agent-repo/v1/finalize.sh — adds AR-38/F97 empty-deliverable audit
(SKIP_BRANCH_PUSH support, boilerplate-path filtering, ci_metadata.json flag)
Also adds MEMORY.md index and memory/ topic files for the framework repo.
Required by the Epic 1 trigger pipeline in agent-runtimes:
- spec-draft-opus@1 uses spec-writing-airouter-repo (concept→spec-draft workflow)
- review-spec-arch-opus@1 uses best-practices-airouter-repo (spec-draft→spec-reviewed-arch)
- scope-decompose-sonnet@1 uses spec-writing-airouter-repo (spec-reviewed-arch→scoped-tagged)
Both composites require the airouter label so only the airouter dispatcher claims them.
agent-repo/v1 init.sh creates /workspace/project/results/ and .gitignore,
making the directory non-empty before the integrator clone action runs.
git clone into a non-empty directory fails with 'already exists'.
Integrators don't need the agent-repo lifecycle — they clone origin/main
directly and push back. Remove agent-repo/v1 from the composite.
Copies code-sonnet-direct/v1 (composite) and direct-push/v1 (context)
from agent-runtimes local harnesses to the CRS-served framework repo.
These are required by the sonnet-integrator task template used in the
ralph coding loop. Without them, all integrator dispatches fail with
"harness not found in any concepts repo: code-sonnet-direct/v1".
Probes 7-9 (2026-05-08) all "succeeded" but with empty diffs because
AR-14a fell back to fork main: "WARNING: /workspace/reference/main/.git
not found; using fork main (may be stale)". The reference clone python
loop printed "Cloning git@..." but never "Cloned main successfully" —
no error visible either.
Most likely cause: full-history clone (post-3087ad7) is hitting some
silent failure mode (auth, fs perms, OOM, timeout). With capture_output=True
the git clone's own output was hidden.
Add explicit logging:
- python3 -u (unbuffered stdout)
- pre-loop "reference_branches loop: N entries"
- post-clone "clone returncode=N"
- always-print stdout (last 1KB) and stderr (last 1KB) from subprocess
- explicit chmod return code check
- post-clone .git existence assertion
Next probe will tell us EXACTLY what's failing.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Probe 7 (2026-05-08) finally surfaced the actual push error:
! [remote rejected] HEAD -> task-9dc266b5 (shallow update not allowed)
Cause: REFERENCE_BRANCHES handler clones with --depth 1, AR-14a then
fetches upstream-ref --depth 1. The agent's task branch is a single
commit on top of a single shallow commit — no ancestry visible. Gitea
rejects shallow pushes server-side.
Fix:
- Reference clone drops --depth 1 (full history)
- AR-14a fetch upstream-ref drops --depth 1 (full fetch from local-path)
- Cost: a few extra MB per task on tmpfs/PVC. Acceptable.
This unblocks AR-14a's upstream-seeding for the dogfood pipeline. The
agent's branch now has the full upstream history visible to gitea.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Probe 6 (2026-05-08) showed git push failing twice with exit 1 and
zero visible output — the previous form `if cmd 2>&1; then` redirected
git's stderr to stdout where the entrypoint's stderr-only log capture
missed it. CP-side log showed only the bash `set -x` trace, not the
actual git error message (e.g., "Permission denied (publickey)" or
"remote: pre-receive hook rejected").
Refactor the push retry loop:
- Capture output to PUSH_OUT via $() with `2>&1`
- Wrap in set +e/set -e to detect non-zero without aborting
- echo PUSH_OUT to stderr (where set -x trace also goes) so the
entrypoint's stderr capture sees it
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The 2026-05-08 attempt-2 dogfood batch had 8/8 tasks "succeed" with
zero branches pushed. Root cause: my AR-21 diff-verification block was
running under set -euo pipefail without explicit error handling. A
single non-zero exit anywhere in the `git diff | tr | sed` pipeline
killed finalize.sh before the metadata write or push ran.
Specific risk: `git diff <REF_HEAD>..HEAD` returns non-zero when the
SHA is unreachable (e.g., shallow clone with init.sh fork-fallback
where upstream-ref wasn't fetched). pipefail then kills the pipeline,
set -e kills the script.
Fix: wrap the entire AR-21 block in `set +eo pipefail` (with explicit
`set -eo pipefail` restore at the end). Also:
- Use `${arr[@]:-}` instead of `${arr[@]}` for set -u safety on empty
arrays
- Add `|| true` to git command substitutions (belt-and-braces)
- Use `printf` instead of `echo` for the comma-wrap (more portable)
Verified locally: when `/workspace/reference/main/.git` is absent the
block correctly skips with the existing fallback; when present and
upstream-ref is reachable, the block runs and reports DIFF_VERIFIED.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Three load-bearing fixes for the airouter dogfood pipeline, derived from
the 2026-05-08 batch-3 dogfood postmortem (gotchas-airouter.md items 27-30):
1. agent-repo/v1/init.sh — seed fresh task branches from
/workspace/reference/main/ (the upstream clone) rather than the agent
repo's stale main. This was THE killer for batch 3: the
agent-runtimes-agents fork has been frozen at 2026-05-04 since the
"Fork cleanup" PR, so every agent started from old state, missing
recent test files and the M16/M22 scripts to delete. The fork remains
the push remote (so finalize.sh works); only the working-tree seed
moves to the upstream reference. Falls back to fork main when the
reference clone isn't available (preserves legacy behavior). Tagged
AR-14a.
2. requires_labels on contexts/composites — airouter context + both
airouter composites declare requires_labels: [airouter] so the
dispatcher's _collect_supported_harnesses (with the matching agent-
runtimes change) advertises them only on dispatchers carrying the
airouter label. Stops the main dispatcher from claiming airouter-
labeled tasks and dying at init time. Composites that wrap label-
restricted contexts MUST redeclare their own requires_labels — no
auto-traversal of layers (kept simple).
3. agent-repo/v1/finalize.sh — AR-21 diff-against-upstream verification.
New env-var protocol:
- AGENT_EXPECTED_CHANGED_FILES (comma-separated paths that MUST
appear in `git diff <ref/main>..HEAD`)
- AGENT_FORBIDDEN_CHANGED_FILES (paths that MUST NOT appear)
finalize.sh fails the task (exit 1) if either invariant is violated;
the branch is still pushed for forensics so the operator can inspect.
Catches BOTH the false-success mode (item 30 — agent reports succeeded
but never changed the target file) AND the destructive-Write mode
(item 21 — task 4a2f2988 stripped 9 unrelated functions). Also writes
diff_verified, diff_mismatch, diff_changed_files into ci_metadata.json.
CRS pulls all three on next CP poll — no agent-runtimes image rebuild
needed for the framework parts. The matching dispatcher poller filter
ships in agent-runtimes (separate commit).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
After the 2026-05-08 M16 Wave A1 dogfood task `4a2f2988` produced a
correct narrow change but accidentally deleted 9 unrelated functions in
the same file (Write rather than Edit), the existing Rule 1 in
qwen-code-methodology/v1/CLAUDE.md is clearly insufficient — the agent
saw it and overrode it anyway. Concrete real-world example added inline
so the next agent reading this CLAUDE.md sees the consequence: "the
narrow test passed but the application broke everywhere else, the
branch was rejected, the agent thought it had succeeded."
CRS picks up automatically on next CP poll.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The agent-runtimes repo is moving to a CRS-only harness model: dispatcher
images stop baking in /app/harnesses/ and discover them via CRS at runtime.
cp-harness was the only harness that lived only in the agent-runtimes
checkout (created in M16 Phase 1 D4, commit c1df4cc) — copied here so
nothing is lost in the cutover.
cp-harness has no current consumers (no composite or template references
it); it is infrastructure for the future M16 self-orchestration api_key
submission flow.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The airouter ESO ExternalSecret materialises a single key `api_key`
(matching the provider schema in agent-runtimes M22 Phase 8e cutover,
acct-59b7fb0b). The harness init script was checking for `auth_token` +
`base_url` (an Anthropic-compat shape that never existed in real Vault
state) and failing on every dispatch:
ERROR: /run/agent/secrets/airouter/auth_token not readable.
Check ESO ExternalSecret for airouter.
Surfaced as the second blocker for the M16 Wave A1 dogfood (the first
was the airouter dispatcher missing CRS sync; that fix went into
agent-runtimes-deploy@0f11cd1).
Same shape of bug as the minimax + gitea-ssh init scripts that landed
during the same M22 phase — those were fixed at the time, airouter was
not.
Changes:
- init.sh: verify the single `api_key` file (root-only, 0400 ESO mount).
Stage to /var/agent-secrets/airouter/api_key with mode 0600 agent-owned
(mirrors minimax pattern). H-SECRET-4 compliant — no exports.
- harness.yaml: add OPENAI_API_KEY_FILE pointing at the staged path.
Agentic runner reads the file at request time per
entrypoint/runners/agentic.py:146 (OPENAI_API_KEY_FILE precedence).
- Delete dead bin/anthropic-compat-wrapper.sh — confirmed unused per
agent-runtimes/memory/log/2026-05-07.214249.md (post-M22-Phase-9 cleanup
found these per-provider wrappers were never invoked; runner only
prepends /opt/agent/claude-wrapper.sh).
CRS picks this up automatically on next CP poll; no agent-runtimes image
rebuild needed.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Removes harnesses/contexts/z-ai/v1/. Mirrors the agent-runtimes
companion PR — the z-ai harness was never cut over to ESO and
there's no active subscription. Revival path documented in
agent-runtimes planning/future/providers F49.
Removed:
- harnesses/contexts/z-ai/v1/{harness.yaml, init.sh}
Updated:
- CLAUDE.md — drops z-ai from the 'no secrets in this repo'
context list. Retirement note added.
- gitea-admin/v1/init.sh — comment ref to z-ai removed.
- gitea-https/v1/init.sh — comment ref to z-ai removed.
No composites layer z-ai (verified via grep across composites/);
no model registry entries reference it.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Phase 8 cutover (M22) renamed the anthropic harness from anthropic-cloud
→ anthropic-cloud-paul-oauth in agent-runtimes, but the equivalent
rename in this framework's composites and the deletion of the stale
SOPS-era anthropic-cloud/v1 dir were never done. Every cloud-Anthropic
composite (haiku/sonnet/opus code, planning, security-review, spec-
writing, test-writing, best-practices) layered the dead context and
silently never resolved a credential.
- Update 14 composites to layer anthropic-cloud-paul-oauth/v1
- Delete the orphaned harnesses/contexts/anthropic-cloud/v1/ directory
(used pre-Phase-9 SOPS file delivery; now obsolete)
Closes BUG-22b. Pairs with the live PATCH /v1/accounts/{id} rename
shipped under BUG-22a.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Same bug as upstream agent-runtimes: scoped form never matched
because the dispatcher does literal scope.id compare and
owner_principal_id stores the user UUID, not 'paul'.
Pairs with PATCH /v1/accounts/{id} renaming the live account_label
to 'anthropic-cloud-paul-oauth' so the bare ref resolves uniquely.
Closes BUG-22a.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
