The validator pattern-matches *TOKEN* as credential-shaped and rejected
both contexts, failing every task on the new harnesses. The env var name
is fixed by Claude Code, so deliver the thinking budget via the
settings.json env map in init.sh instead (jq-merge preserves apiKeyHelper
from the minimax/oauth layer).
Claude-Session: https://claude.ai/code/session_019tJk7P8tZzJ24PvtgoGhLN
Integration runs were burning the personal Anthropic 5-hour window
(429 storm + window drain, 2026-09-02). New env-only context
minimax-m3-max/v1 pins ANTHROPIC_MODEL=MiniMax-M3[1m], maxes the
thinking budget, and sets the 1M auto-compact window; layered on
minimax/v1 for base URL + apiKeyHelper auth.
Claude-Session: https://claude.ai/code/session_019tJk7P8tZzJ24PvtgoGhLN
All 6 concept/idea-stage templates hardcoded the literal shared branch
spec/auto-draft instead of a per-item branch. Concurrent concept items
(now routine via the M41 auto-advance sweep) all pushed to the same
branch — a git collision risk. Switched to spec/${item.uuid}, matching
the established task/${item.uuid} convention used by every task-stage
template (merge@1, scaffold@1, etc — see spec/typed-workflows.md WF-COMP
stable-per-item-branch rationale). spec-review-loop@1's 4 nodes keep the
literal branch shared ACROSS the node set (now per-item, not global) so
loop iterations still accumulate on one branch per item, per spec's
cross-iteration-artifact caveat.
Files: review-spec-arch-opus@1.yaml, spec-draft-opus@1.yaml,
spec-review-loop@1.yaml, review-spec-arch-airouter@1.yaml,
spec-draft-airouter@2.yaml, scope-decompose-sonnet@1.yaml
Was silently dispatching to Qwen3.6 via airouter despite the name;
live runs were timing out (exit_code=124), wedging four M41 auto-advance
concepts. Mirrors agent-runtimes commit acc5cbe5.
Claude-Session: https://claude.ai/code/session_012arAwUuBQrpnMzJ36mEVBk
The 237KB library was mounted at /workspace/best-practices/ in every
consuming ConfigMap but nothing in the prompt mentioned it (directory
targets never enter the context manifest). New ~90-word CLAUDE.md file
target enters the system prompt: points at INDEX.md, instructs reading
only task-relevant topics.
Claude-Session: https://claude.ac/code/session_012arAwUuBQrpnMzJ36mEVBk
- tdd-gate: pass criterion is the task's TEST_FILES scope (matches
run-ci-tests.sh M19 P-14), not the full suite — contradicted
qwen-code-methodology rule 5 in the same system prompt
- tdd-protect: tests/ path described relative to project working dir
(AGENT_WORKING_DIR is /workspace/project in repo composites)
- qwen-code-methodology: task_complete tool DOES exist — rule 8 said
the opposite; pytest examples use -q
- airouter-code-tdd template: TT-39 runtime_overrides
(max_result_bytes 24KB, max_history_turns 16)
Claude-Session: https://claude.ai/code/session_012arAwUuBQrpnMzJ36mEVBk
The python3 -c block imported internal entrypoint code directly and
pasted module paths into agent prompts — a security boundary violation.
Switch to the thin-client verb (added in agent-runtimes a116aedb).
Claude-Session: https://claude.ai/code/session_0124mK2BVfCdkELLXDCr5Zzs
All tasks dispatched to integration-direct/v1 were failing with
harness_missing_agent_repo_layer because dispatcher DSP-1 guard
requires agent-repo in the composite when AGENT_REPO_URL is set.
direct-push/v1 sets env vars (AGENT_SKIP_BRANCH_PUSH,
AGENT_EMPTY_DELIVERABLE_CHECK) that configure agent-repo's finalize.sh,
so agent-repo/v1 must precede it in the layer list.
metadata.task_description is not in the consumer snapshot — the trigger
consumer only populates metadata.automation from the outbox typed_payload.
artifacts.item_body IS populated via dispatch_service from
item_snapshot["body"] (set from typed_payload.body in the consumer,
which comes from item.get("body") in emit_work_item_event).
For M39 items the body contains the full task description, so
${artifacts.item_body} gives the agent complete implementation instructions.
${metadata.automation.test_files} and ${metadata.automation.spec_ids} are
absent from M39 work items (created from dashboard without these fields),
resulting in an empty, useless prompt:
"Implement the failing test. Test files: . Spec IDs: ."
Switch to ${metadata.task_description} — a sibling metadata key (not under
automation, so write-protected path does not apply) — populated by patching
items with their body text. The _TriggerItemView in dispatch_service.py
correctly exposes item.metadata so this resolves at compile time.
All 16 scaffolded M39 items patched with metadata.task_description via CP API.
Add direct-impl-sonnet@1, plan-review-loop@1, spec-review-loop@1,
test-write-review-loop@1 — present in agent-runtimes but missing here.
These are needed before agent-runtimes drops its baked-in copies and the
CP loads workflow templates exclusively from this CRS repo.
Remove `tags_required: [airouter-eligible]` from airouter-impl-ready so all
non-skip tasks route through the full airouter→sonnet→opus ladder automatically.
Add `airouter-skip` to airouter's tags_forbidden so opt-out tasks go directly
to sonnet from attempt 0.
Relax sonnet-impl-ready tags_forbidden (drop multi-file + algorithmic-large)
so sonnet handles complex tasks at escalation tier without template rejection.
Add opus-impl-ready@1.yaml — last-resort escalation, code-opus-tdd-repo
composite, scope_budget 5 files / 400 lines / 3600s.
MiniMax moved their Anthropic-compatible API from api.minimax.io to
api.minimaxi.chat. Update ANTHROPIC_BASE_URL in the minimax/v1 context
harness and corresponding comments in init.sh and the TODO note.
Model routing is unaffected: the API defaults unrecognised model names
(e.g. "minimax" from the registry slug) to MiniMax-M3 already.
Claude-Session: https://claude.ai/code/session_01XtWRazqRbMnMRa4iV3zbVx
W1 stage-routing work:
- model-registry/tiers.yaml: defines planning/spec-test/coding tier floors
(complexity≥9/creativity≥9/context≥9 | spec_adherence≥9/test_pass_rate≥9 |
spec_adherence≥7/test_pass_rate≥7) with qualifies_today lists
- spec-draft-airouter@2.yaml: honest name for what was spec-draft-opus@1
(always ran Qwen3.6/airouter, not Opus)
- review-spec-arch-airouter@1.yaml: honest name for review-spec-arch-opus@1
Old @1 files kept for in-flight task backward-compatibility.
Claude-Session: https://claude.ai/code/session_01B35bPAKv5uyW1F9gzMcRN7
New code-airouter-deepseekv4flash-tdd-repo/v1 composite (deepseek
methodology + the same tdd-protect/tdd-gate/python-dev/tdd-file-lock
layers as the Qwen TDD composite), plus a task-template that sets
reasoning_effort=max via the new extra_body passthrough (agent-runtimes
TT-38). Model/endpoint/model-registry entries already existed; only
the TDD-capable harness and the template were missing.
fix-test-fixture-sonnet@1, review-spec-opus@1, replan-spec-opus@1 — the
dangling escalation refs from airouter-impl-*/spec-draft/feature-delivery-loop.
Sonnet-authored, Opus-reviewed; review HIGH fixes applied (explicit model: opus
on both opus nodes — node model: sets runtime model, composite name does not).
Known gap filed in work-items epic (E1-escalation-state-gap.md): escalation
edges from non-primary states 422 state_mismatch pending escalation_input_states.
concept-intake@1: optional idea-elaboration pre-stage (output state
idea-elaborated — not yet wired into spec-draft-opus input).
All four validate against controlplane WorkflowTemplate schema.
New context: hugo-content-workspace/v1
- init.sh: clones hugo-{customer}-content at HUGO_CONTENT_BRANCH and
hugo-{customer}-integration at main via gitea-ssh-accelerators
- finalize.sh: detects changes, commits content repo first then
integration repo; skips silently if no changes
- CLAUDE.md: instructs the agent to work in /workspace/content/
New composite: hugo-content-airouter/v1
- Combines airouter/v1 + gitea-ssh-accelerators/v1 + hugo-content-workspace/v1
- Label-gated to airouter dispatchers (ESO secret required)
Used by cms-proxy /ai/{customer}/draft endpoint.
Brings the framework CRS repo up to date with all content that was
living in agent-runtimes (local-dev fallback) but hadn't been promoted.
New composites: feature-delivery-loop, integration-direct, scaffolding-repo,
sonnet-impl-narrow, sonnet-manager, test-writing-repo
New contexts: integration/v1, scaffolding/v1, sonnet-manager/v1, z-ai/v1,
airouter/v1/bin (anthropic-compat-wrapper.sh), cp-harness/v1/init.sh
New task-templates: sonnet-integrator.yaml, workflow/* (17 typed workflow
task templates for the Epic 1 pipeline)
Updated: agent-repo/v1/finalize.sh — adds AR-38/F97 empty-deliverable audit
(SKIP_BRANCH_PUSH support, boilerplate-path filtering, ci_metadata.json flag)
Also adds MEMORY.md index and memory/ topic files for the framework repo.
Required by the Epic 1 trigger pipeline in agent-runtimes:
- spec-draft-opus@1 uses spec-writing-airouter-repo (concept→spec-draft workflow)
- review-spec-arch-opus@1 uses best-practices-airouter-repo (spec-draft→spec-reviewed-arch)
- scope-decompose-sonnet@1 uses spec-writing-airouter-repo (spec-reviewed-arch→scoped-tagged)
Both composites require the airouter label so only the airouter dispatcher claims them.
agent-repo/v1 init.sh creates /workspace/project/results/ and .gitignore,
making the directory non-empty before the integrator clone action runs.
git clone into a non-empty directory fails with 'already exists'.
Integrators don't need the agent-repo lifecycle — they clone origin/main
directly and push back. Remove agent-repo/v1 from the composite.
Copies code-sonnet-direct/v1 (composite) and direct-push/v1 (context)
from agent-runtimes local harnesses to the CRS-served framework repo.
These are required by the sonnet-integrator task template used in the
ralph coding loop. Without them, all integrator dispatches fail with
"harness not found in any concepts repo: code-sonnet-direct/v1".
Probes 7-9 (2026-05-08) all "succeeded" but with empty diffs because
AR-14a fell back to fork main: "WARNING: /workspace/reference/main/.git
not found; using fork main (may be stale)". The reference clone python
loop printed "Cloning git@..." but never "Cloned main successfully" —
no error visible either.
Most likely cause: full-history clone (post-3087ad7) is hitting some
silent failure mode (auth, fs perms, OOM, timeout). With capture_output=True
the git clone's own output was hidden.
Add explicit logging:
- python3 -u (unbuffered stdout)
- pre-loop "reference_branches loop: N entries"
- post-clone "clone returncode=N"
- always-print stdout (last 1KB) and stderr (last 1KB) from subprocess
- explicit chmod return code check
- post-clone .git existence assertion
Next probe will tell us EXACTLY what's failing.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Probe 7 (2026-05-08) finally surfaced the actual push error:
! [remote rejected] HEAD -> task-9dc266b5 (shallow update not allowed)
Cause: REFERENCE_BRANCHES handler clones with --depth 1, AR-14a then
fetches upstream-ref --depth 1. The agent's task branch is a single
commit on top of a single shallow commit — no ancestry visible. Gitea
rejects shallow pushes server-side.
Fix:
- Reference clone drops --depth 1 (full history)
- AR-14a fetch upstream-ref drops --depth 1 (full fetch from local-path)
- Cost: a few extra MB per task on tmpfs/PVC. Acceptable.
This unblocks AR-14a's upstream-seeding for the dogfood pipeline. The
agent's branch now has the full upstream history visible to gitea.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Probe 6 (2026-05-08) showed git push failing twice with exit 1 and
zero visible output — the previous form `if cmd 2>&1; then` redirected
git's stderr to stdout where the entrypoint's stderr-only log capture
missed it. CP-side log showed only the bash `set -x` trace, not the
actual git error message (e.g., "Permission denied (publickey)" or
"remote: pre-receive hook rejected").
Refactor the push retry loop:
- Capture output to PUSH_OUT via $() with `2>&1`
- Wrap in set +e/set -e to detect non-zero without aborting
- echo PUSH_OUT to stderr (where set -x trace also goes) so the
entrypoint's stderr capture sees it
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The 2026-05-08 attempt-2 dogfood batch had 8/8 tasks "succeed" with
zero branches pushed. Root cause: my AR-21 diff-verification block was
running under set -euo pipefail without explicit error handling. A
single non-zero exit anywhere in the `git diff | tr | sed` pipeline
killed finalize.sh before the metadata write or push ran.
Specific risk: `git diff <REF_HEAD>..HEAD` returns non-zero when the
SHA is unreachable (e.g., shallow clone with init.sh fork-fallback
where upstream-ref wasn't fetched). pipefail then kills the pipeline,
set -e kills the script.
Fix: wrap the entire AR-21 block in `set +eo pipefail` (with explicit
`set -eo pipefail` restore at the end). Also:
- Use `${arr[@]:-}` instead of `${arr[@]}` for set -u safety on empty
arrays
- Add `|| true` to git command substitutions (belt-and-braces)
- Use `printf` instead of `echo` for the comma-wrap (more portable)
Verified locally: when `/workspace/reference/main/.git` is absent the
block correctly skips with the existing fallback; when present and
upstream-ref is reachable, the block runs and reports DIFF_VERIFIED.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Three load-bearing fixes for the airouter dogfood pipeline, derived from
the 2026-05-08 batch-3 dogfood postmortem (gotchas-airouter.md items 27-30):
1. agent-repo/v1/init.sh — seed fresh task branches from
/workspace/reference/main/ (the upstream clone) rather than the agent
repo's stale main. This was THE killer for batch 3: the
agent-runtimes-agents fork has been frozen at 2026-05-04 since the
"Fork cleanup" PR, so every agent started from old state, missing
recent test files and the M16/M22 scripts to delete. The fork remains
the push remote (so finalize.sh works); only the working-tree seed
moves to the upstream reference. Falls back to fork main when the
reference clone isn't available (preserves legacy behavior). Tagged
AR-14a.
2. requires_labels on contexts/composites — airouter context + both
airouter composites declare requires_labels: [airouter] so the
dispatcher's _collect_supported_harnesses (with the matching agent-
runtimes change) advertises them only on dispatchers carrying the
airouter label. Stops the main dispatcher from claiming airouter-
labeled tasks and dying at init time. Composites that wrap label-
restricted contexts MUST redeclare their own requires_labels — no
auto-traversal of layers (kept simple).
3. agent-repo/v1/finalize.sh — AR-21 diff-against-upstream verification.
New env-var protocol:
- AGENT_EXPECTED_CHANGED_FILES (comma-separated paths that MUST
appear in `git diff <ref/main>..HEAD`)
- AGENT_FORBIDDEN_CHANGED_FILES (paths that MUST NOT appear)
finalize.sh fails the task (exit 1) if either invariant is violated;
the branch is still pushed for forensics so the operator can inspect.
Catches BOTH the false-success mode (item 30 — agent reports succeeded
but never changed the target file) AND the destructive-Write mode
(item 21 — task 4a2f2988 stripped 9 unrelated functions). Also writes
diff_verified, diff_mismatch, diff_changed_files into ci_metadata.json.
CRS pulls all three on next CP poll — no agent-runtimes image rebuild
needed for the framework parts. The matching dispatcher poller filter
ships in agent-runtimes (separate commit).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
After the 2026-05-08 M16 Wave A1 dogfood task `4a2f2988` produced a
correct narrow change but accidentally deleted 9 unrelated functions in
the same file (Write rather than Edit), the existing Rule 1 in
qwen-code-methodology/v1/CLAUDE.md is clearly insufficient — the agent
saw it and overrode it anyway. Concrete real-world example added inline
so the next agent reading this CLAUDE.md sees the consequence: "the
narrow test passed but the application broke everywhere else, the
branch was rejected, the agent thought it had succeeded."
CRS picks up automatically on next CP poll.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The agent-runtimes repo is moving to a CRS-only harness model: dispatcher
images stop baking in /app/harnesses/ and discover them via CRS at runtime.
cp-harness was the only harness that lived only in the agent-runtimes
checkout (created in M16 Phase 1 D4, commit c1df4cc) — copied here so
nothing is lost in the cutover.
cp-harness has no current consumers (no composite or template references
it); it is infrastructure for the future M16 self-orchestration api_key
submission flow.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The airouter ESO ExternalSecret materialises a single key `api_key`
(matching the provider schema in agent-runtimes M22 Phase 8e cutover,
acct-59b7fb0b). The harness init script was checking for `auth_token` +
`base_url` (an Anthropic-compat shape that never existed in real Vault
state) and failing on every dispatch:
ERROR: /run/agent/secrets/airouter/auth_token not readable.
Check ESO ExternalSecret for airouter.
Surfaced as the second blocker for the M16 Wave A1 dogfood (the first
was the airouter dispatcher missing CRS sync; that fix went into
agent-runtimes-deploy@0f11cd1).
Same shape of bug as the minimax + gitea-ssh init scripts that landed
during the same M22 phase — those were fixed at the time, airouter was
not.
Changes:
- init.sh: verify the single `api_key` file (root-only, 0400 ESO mount).
Stage to /var/agent-secrets/airouter/api_key with mode 0600 agent-owned
(mirrors minimax pattern). H-SECRET-4 compliant — no exports.
- harness.yaml: add OPENAI_API_KEY_FILE pointing at the staged path.
Agentic runner reads the file at request time per
entrypoint/runners/agentic.py:146 (OPENAI_API_KEY_FILE precedence).
- Delete dead bin/anthropic-compat-wrapper.sh — confirmed unused per
agent-runtimes/memory/log/2026-05-07.214249.md (post-M22-Phase-9 cleanup
found these per-provider wrappers were never invoked; runner only
prepends /opt/agent/claude-wrapper.sh).
CRS picks this up automatically on next CP poll; no agent-runtimes image
rebuild needed.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Removes harnesses/contexts/z-ai/v1/. Mirrors the agent-runtimes
companion PR — the z-ai harness was never cut over to ESO and
there's no active subscription. Revival path documented in
agent-runtimes planning/future/providers F49.
Removed:
- harnesses/contexts/z-ai/v1/{harness.yaml, init.sh}
Updated:
- CLAUDE.md — drops z-ai from the 'no secrets in this repo'
context list. Retirement note added.
- gitea-admin/v1/init.sh — comment ref to z-ai removed.
- gitea-https/v1/init.sh — comment ref to z-ai removed.
No composites layer z-ai (verified via grep across composites/);
no model registry entries reference it.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Phase 8 cutover (M22) renamed the anthropic harness from anthropic-cloud
→ anthropic-cloud-paul-oauth in agent-runtimes, but the equivalent
rename in this framework's composites and the deletion of the stale
SOPS-era anthropic-cloud/v1 dir were never done. Every cloud-Anthropic
composite (haiku/sonnet/opus code, planning, security-review, spec-
writing, test-writing, best-practices) layered the dead context and
silently never resolved a credential.
- Update 14 composites to layer anthropic-cloud-paul-oauth/v1
- Delete the orphaned harnesses/contexts/anthropic-cloud/v1/ directory
(used pre-Phase-9 SOPS file delivery; now obsolete)
Closes BUG-22b. Pairs with the live PATCH /v1/accounts/{id} rename
shipped under BUG-22a.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Same bug as upstream agent-runtimes: scoped form never matched
because the dispatcher does literal scope.id compare and
owner_principal_id stores the user UUID, not 'paul'.
Pairs with PATCH /v1/accounts/{id} renaming the live account_label
to 'anthropic-cloud-paul-oauth' so the bare ref resolves uniquely.
Closes BUG-22a.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
