Integration runs were burning the personal Anthropic 5-hour window
(429 storm + window drain, 2026-09-02). New env-only context
minimax-m3-max/v1 pins ANTHROPIC_MODEL=MiniMax-M3[1m], maxes the
thinking budget, and sets the 1M auto-compact window; layered on
minimax/v1 for base URL + apiKeyHelper auth.
Claude-Session: https://claude.ai/code/session_019tJk7P8tZzJ24PvtgoGhLN
All 6 concept/idea-stage templates hardcoded the literal shared branch
spec/auto-draft instead of a per-item branch. Concurrent concept items
(now routine via the M41 auto-advance sweep) all pushed to the same
branch — a git collision risk. Switched to spec/${item.uuid}, matching
the established task/${item.uuid} convention used by every task-stage
template (merge@1, scaffold@1, etc — see spec/typed-workflows.md WF-COMP
stable-per-item-branch rationale). spec-review-loop@1's 4 nodes keep the
literal branch shared ACROSS the node set (now per-item, not global) so
loop iterations still accumulate on one branch per item, per spec's
cross-iteration-artifact caveat.
Files: review-spec-arch-opus@1.yaml, spec-draft-opus@1.yaml,
spec-review-loop@1.yaml, review-spec-arch-airouter@1.yaml,
spec-draft-airouter@2.yaml, scope-decompose-sonnet@1.yaml
Was silently dispatching to Qwen3.6 via airouter despite the name;
live runs were timing out (exit_code=124), wedging four M41 auto-advance
concepts. Mirrors agent-runtimes commit acc5cbe5.
Claude-Session: https://claude.ai/code/session_012arAwUuBQrpnMzJ36mEVBk
The 237KB library was mounted at /workspace/best-practices/ in every
consuming ConfigMap but nothing in the prompt mentioned it (directory
targets never enter the context manifest). New ~90-word CLAUDE.md file
target enters the system prompt: points at INDEX.md, instructs reading
only task-relevant topics.
Claude-Session: https://claude.ac/code/session_012arAwUuBQrpnMzJ36mEVBk
- tdd-gate: pass criterion is the task's TEST_FILES scope (matches
run-ci-tests.sh M19 P-14), not the full suite — contradicted
qwen-code-methodology rule 5 in the same system prompt
- tdd-protect: tests/ path described relative to project working dir
(AGENT_WORKING_DIR is /workspace/project in repo composites)
- qwen-code-methodology: task_complete tool DOES exist — rule 8 said
the opposite; pytest examples use -q
- airouter-code-tdd template: TT-39 runtime_overrides
(max_result_bytes 24KB, max_history_turns 16)
Claude-Session: https://claude.ai/code/session_012arAwUuBQrpnMzJ36mEVBk
The python3 -c block imported internal entrypoint code directly and
pasted module paths into agent prompts — a security boundary violation.
Switch to the thin-client verb (added in agent-runtimes a116aedb).
Claude-Session: https://claude.ai/code/session_0124mK2BVfCdkELLXDCr5Zzs
All tasks dispatched to integration-direct/v1 were failing with
harness_missing_agent_repo_layer because dispatcher DSP-1 guard
requires agent-repo in the composite when AGENT_REPO_URL is set.
direct-push/v1 sets env vars (AGENT_SKIP_BRANCH_PUSH,
AGENT_EMPTY_DELIVERABLE_CHECK) that configure agent-repo's finalize.sh,
so agent-repo/v1 must precede it in the layer list.
metadata.task_description is not in the consumer snapshot — the trigger
consumer only populates metadata.automation from the outbox typed_payload.
artifacts.item_body IS populated via dispatch_service from
item_snapshot["body"] (set from typed_payload.body in the consumer,
which comes from item.get("body") in emit_work_item_event).
For M39 items the body contains the full task description, so
${artifacts.item_body} gives the agent complete implementation instructions.
${metadata.automation.test_files} and ${metadata.automation.spec_ids} are
absent from M39 work items (created from dashboard without these fields),
resulting in an empty, useless prompt:
"Implement the failing test. Test files: . Spec IDs: ."
Switch to ${metadata.task_description} — a sibling metadata key (not under
automation, so write-protected path does not apply) — populated by patching
items with their body text. The _TriggerItemView in dispatch_service.py
correctly exposes item.metadata so this resolves at compile time.
All 16 scaffolded M39 items patched with metadata.task_description via CP API.
Add direct-impl-sonnet@1, plan-review-loop@1, spec-review-loop@1,
test-write-review-loop@1 — present in agent-runtimes but missing here.
These are needed before agent-runtimes drops its baked-in copies and the
CP loads workflow templates exclusively from this CRS repo.
Remove `tags_required: [airouter-eligible]` from airouter-impl-ready so all
non-skip tasks route through the full airouter→sonnet→opus ladder automatically.
Add `airouter-skip` to airouter's tags_forbidden so opt-out tasks go directly
to sonnet from attempt 0.
Relax sonnet-impl-ready tags_forbidden (drop multi-file + algorithmic-large)
so sonnet handles complex tasks at escalation tier without template rejection.
Add opus-impl-ready@1.yaml — last-resort escalation, code-opus-tdd-repo
composite, scope_budget 5 files / 400 lines / 3600s.
MiniMax moved their Anthropic-compatible API from api.minimax.io to
api.minimaxi.chat. Update ANTHROPIC_BASE_URL in the minimax/v1 context
harness and corresponding comments in init.sh and the TODO note.
Model routing is unaffected: the API defaults unrecognised model names
(e.g. "minimax" from the registry slug) to MiniMax-M3 already.
Claude-Session: https://claude.ai/code/session_01XtWRazqRbMnMRa4iV3zbVx
W1 stage-routing work:
- model-registry/tiers.yaml: defines planning/spec-test/coding tier floors
(complexity≥9/creativity≥9/context≥9 | spec_adherence≥9/test_pass_rate≥9 |
spec_adherence≥7/test_pass_rate≥7) with qualifies_today lists
- spec-draft-airouter@2.yaml: honest name for what was spec-draft-opus@1
(always ran Qwen3.6/airouter, not Opus)
- review-spec-arch-airouter@1.yaml: honest name for review-spec-arch-opus@1
Old @1 files kept for in-flight task backward-compatibility.
Claude-Session: https://claude.ai/code/session_01B35bPAKv5uyW1F9gzMcRN7
New code-airouter-deepseekv4flash-tdd-repo/v1 composite (deepseek
methodology + the same tdd-protect/tdd-gate/python-dev/tdd-file-lock
layers as the Qwen TDD composite), plus a task-template that sets
reasoning_effort=max via the new extra_body passthrough (agent-runtimes
TT-38). Model/endpoint/model-registry entries already existed; only
the TDD-capable harness and the template were missing.
fix-test-fixture-sonnet@1, review-spec-opus@1, replan-spec-opus@1 — the
dangling escalation refs from airouter-impl-*/spec-draft/feature-delivery-loop.
Sonnet-authored, Opus-reviewed; review HIGH fixes applied (explicit model: opus
on both opus nodes — node model: sets runtime model, composite name does not).
Known gap filed in work-items epic (E1-escalation-state-gap.md): escalation
edges from non-primary states 422 state_mismatch pending escalation_input_states.
concept-intake@1: optional idea-elaboration pre-stage (output state
idea-elaborated — not yet wired into spec-draft-opus input).
All four validate against controlplane WorkflowTemplate schema.
New context: hugo-content-workspace/v1
- init.sh: clones hugo-{customer}-content at HUGO_CONTENT_BRANCH and
hugo-{customer}-integration at main via gitea-ssh-accelerators
- finalize.sh: detects changes, commits content repo first then
integration repo; skips silently if no changes
- CLAUDE.md: instructs the agent to work in /workspace/content/
New composite: hugo-content-airouter/v1
- Combines airouter/v1 + gitea-ssh-accelerators/v1 + hugo-content-workspace/v1
- Label-gated to airouter dispatchers (ESO secret required)
Used by cms-proxy /ai/{customer}/draft endpoint.
Brings the framework CRS repo up to date with all content that was
living in agent-runtimes (local-dev fallback) but hadn't been promoted.
New composites: feature-delivery-loop, integration-direct, scaffolding-repo,
sonnet-impl-narrow, sonnet-manager, test-writing-repo
New contexts: integration/v1, scaffolding/v1, sonnet-manager/v1, z-ai/v1,
airouter/v1/bin (anthropic-compat-wrapper.sh), cp-harness/v1/init.sh
New task-templates: sonnet-integrator.yaml, workflow/* (17 typed workflow
task templates for the Epic 1 pipeline)
Updated: agent-repo/v1/finalize.sh — adds AR-38/F97 empty-deliverable audit
(SKIP_BRANCH_PUSH support, boilerplate-path filtering, ci_metadata.json flag)
Also adds MEMORY.md index and memory/ topic files for the framework repo.
Required by the Epic 1 trigger pipeline in agent-runtimes:
- spec-draft-opus@1 uses spec-writing-airouter-repo (concept→spec-draft workflow)
- review-spec-arch-opus@1 uses best-practices-airouter-repo (spec-draft→spec-reviewed-arch)
- scope-decompose-sonnet@1 uses spec-writing-airouter-repo (spec-reviewed-arch→scoped-tagged)
Both composites require the airouter label so only the airouter dispatcher claims them.
agent-repo/v1 init.sh creates /workspace/project/results/ and .gitignore,
making the directory non-empty before the integrator clone action runs.
git clone into a non-empty directory fails with 'already exists'.
Integrators don't need the agent-repo lifecycle — they clone origin/main
directly and push back. Remove agent-repo/v1 from the composite.
Copies code-sonnet-direct/v1 (composite) and direct-push/v1 (context)
from agent-runtimes local harnesses to the CRS-served framework repo.
These are required by the sonnet-integrator task template used in the
ralph coding loop. Without them, all integrator dispatches fail with
"harness not found in any concepts repo: code-sonnet-direct/v1".
Probes 7-9 (2026-05-08) all "succeeded" but with empty diffs because
AR-14a fell back to fork main: "WARNING: /workspace/reference/main/.git
not found; using fork main (may be stale)". The reference clone python
loop printed "Cloning git@..." but never "Cloned main successfully" —
no error visible either.
Most likely cause: full-history clone (post-3087ad7) is hitting some
silent failure mode (auth, fs perms, OOM, timeout). With capture_output=True
the git clone's own output was hidden.
Add explicit logging:
- python3 -u (unbuffered stdout)
- pre-loop "reference_branches loop: N entries"
- post-clone "clone returncode=N"
- always-print stdout (last 1KB) and stderr (last 1KB) from subprocess
- explicit chmod return code check
- post-clone .git existence assertion
Next probe will tell us EXACTLY what's failing.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Probe 7 (2026-05-08) finally surfaced the actual push error:
! [remote rejected] HEAD -> task-9dc266b5 (shallow update not allowed)
Cause: REFERENCE_BRANCHES handler clones with --depth 1, AR-14a then
fetches upstream-ref --depth 1. The agent's task branch is a single
commit on top of a single shallow commit — no ancestry visible. Gitea
rejects shallow pushes server-side.
Fix:
- Reference clone drops --depth 1 (full history)
- AR-14a fetch upstream-ref drops --depth 1 (full fetch from local-path)
- Cost: a few extra MB per task on tmpfs/PVC. Acceptable.
This unblocks AR-14a's upstream-seeding for the dogfood pipeline. The
agent's branch now has the full upstream history visible to gitea.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Probe 6 (2026-05-08) showed git push failing twice with exit 1 and
zero visible output — the previous form `if cmd 2>&1; then` redirected
git's stderr to stdout where the entrypoint's stderr-only log capture
missed it. CP-side log showed only the bash `set -x` trace, not the
actual git error message (e.g., "Permission denied (publickey)" or
"remote: pre-receive hook rejected").
Refactor the push retry loop:
- Capture output to PUSH_OUT via $() with `2>&1`
- Wrap in set +e/set -e to detect non-zero without aborting
- echo PUSH_OUT to stderr (where set -x trace also goes) so the
entrypoint's stderr capture sees it
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The 2026-05-08 attempt-2 dogfood batch had 8/8 tasks "succeed" with
zero branches pushed. Root cause: my AR-21 diff-verification block was
running under set -euo pipefail without explicit error handling. A
single non-zero exit anywhere in the `git diff | tr | sed` pipeline
killed finalize.sh before the metadata write or push ran.
Specific risk: `git diff <REF_HEAD>..HEAD` returns non-zero when the
SHA is unreachable (e.g., shallow clone with init.sh fork-fallback
where upstream-ref wasn't fetched). pipefail then kills the pipeline,
set -e kills the script.
Fix: wrap the entire AR-21 block in `set +eo pipefail` (with explicit
`set -eo pipefail` restore at the end). Also:
- Use `${arr[@]:-}` instead of `${arr[@]}` for set -u safety on empty
arrays
- Add `|| true` to git command substitutions (belt-and-braces)
- Use `printf` instead of `echo` for the comma-wrap (more portable)
Verified locally: when `/workspace/reference/main/.git` is absent the
block correctly skips with the existing fallback; when present and
upstream-ref is reachable, the block runs and reports DIFF_VERIFIED.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Three load-bearing fixes for the airouter dogfood pipeline, derived from
the 2026-05-08 batch-3 dogfood postmortem (gotchas-airouter.md items 27-30):
1. agent-repo/v1/init.sh — seed fresh task branches from
/workspace/reference/main/ (the upstream clone) rather than the agent
repo's stale main. This was THE killer for batch 3: the
agent-runtimes-agents fork has been frozen at 2026-05-04 since the
"Fork cleanup" PR, so every agent started from old state, missing
recent test files and the M16/M22 scripts to delete. The fork remains
the push remote (so finalize.sh works); only the working-tree seed
moves to the upstream reference. Falls back to fork main when the
reference clone isn't available (preserves legacy behavior). Tagged
AR-14a.
2. requires_labels on contexts/composites — airouter context + both
airouter composites declare requires_labels: [airouter] so the
dispatcher's _collect_supported_harnesses (with the matching agent-
runtimes change) advertises them only on dispatchers carrying the
airouter label. Stops the main dispatcher from claiming airouter-
labeled tasks and dying at init time. Composites that wrap label-
restricted contexts MUST redeclare their own requires_labels — no
auto-traversal of layers (kept simple).
3. agent-repo/v1/finalize.sh — AR-21 diff-against-upstream verification.
New env-var protocol:
- AGENT_EXPECTED_CHANGED_FILES (comma-separated paths that MUST
appear in `git diff <ref/main>..HEAD`)
- AGENT_FORBIDDEN_CHANGED_FILES (paths that MUST NOT appear)
finalize.sh fails the task (exit 1) if either invariant is violated;
the branch is still pushed for forensics so the operator can inspect.
Catches BOTH the false-success mode (item 30 — agent reports succeeded
but never changed the target file) AND the destructive-Write mode
(item 21 — task 4a2f2988 stripped 9 unrelated functions). Also writes
diff_verified, diff_mismatch, diff_changed_files into ci_metadata.json.
CRS pulls all three on next CP poll — no agent-runtimes image rebuild
needed for the framework parts. The matching dispatcher poller filter
ships in agent-runtimes (separate commit).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
After the 2026-05-08 M16 Wave A1 dogfood task `4a2f2988` produced a
correct narrow change but accidentally deleted 9 unrelated functions in
the same file (Write rather than Edit), the existing Rule 1 in
qwen-code-methodology/v1/CLAUDE.md is clearly insufficient — the agent
saw it and overrode it anyway. Concrete real-world example added inline
so the next agent reading this CLAUDE.md sees the consequence: "the
narrow test passed but the application broke everywhere else, the
branch was rejected, the agent thought it had succeeded."
CRS picks up automatically on next CP poll.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The agent-runtimes repo is moving to a CRS-only harness model: dispatcher
images stop baking in /app/harnesses/ and discover them via CRS at runtime.
cp-harness was the only harness that lived only in the agent-runtimes
checkout (created in M16 Phase 1 D4, commit c1df4cc) — copied here so
nothing is lost in the cutover.
cp-harness has no current consumers (no composite or template references
it); it is infrastructure for the future M16 self-orchestration api_key
submission flow.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The airouter ESO ExternalSecret materialises a single key `api_key`
(matching the provider schema in agent-runtimes M22 Phase 8e cutover,
acct-59b7fb0b). The harness init script was checking for `auth_token` +
`base_url` (an Anthropic-compat shape that never existed in real Vault
state) and failing on every dispatch:
ERROR: /run/agent/secrets/airouter/auth_token not readable.
Check ESO ExternalSecret for airouter.
Surfaced as the second blocker for the M16 Wave A1 dogfood (the first
was the airouter dispatcher missing CRS sync; that fix went into
agent-runtimes-deploy@0f11cd1).
Same shape of bug as the minimax + gitea-ssh init scripts that landed
during the same M22 phase — those were fixed at the time, airouter was
not.
Changes:
- init.sh: verify the single `api_key` file (root-only, 0400 ESO mount).
Stage to /var/agent-secrets/airouter/api_key with mode 0600 agent-owned
(mirrors minimax pattern). H-SECRET-4 compliant — no exports.
- harness.yaml: add OPENAI_API_KEY_FILE pointing at the staged path.
Agentic runner reads the file at request time per
entrypoint/runners/agentic.py:146 (OPENAI_API_KEY_FILE precedence).
- Delete dead bin/anthropic-compat-wrapper.sh — confirmed unused per
agent-runtimes/memory/log/2026-05-07.214249.md (post-M22-Phase-9 cleanup
found these per-provider wrappers were never invoked; runner only
prepends /opt/agent/claude-wrapper.sh).
CRS picks this up automatically on next CP poll; no agent-runtimes image
rebuild needed.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Removes harnesses/contexts/z-ai/v1/. Mirrors the agent-runtimes
companion PR — the z-ai harness was never cut over to ESO and
there's no active subscription. Revival path documented in
agent-runtimes planning/future/providers F49.
Removed:
- harnesses/contexts/z-ai/v1/{harness.yaml, init.sh}
Updated:
- CLAUDE.md — drops z-ai from the 'no secrets in this repo'
context list. Retirement note added.
- gitea-admin/v1/init.sh — comment ref to z-ai removed.
- gitea-https/v1/init.sh — comment ref to z-ai removed.
No composites layer z-ai (verified via grep across composites/);
no model registry entries reference it.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Phase 8 cutover (M22) renamed the anthropic harness from anthropic-cloud
→ anthropic-cloud-paul-oauth in agent-runtimes, but the equivalent
rename in this framework's composites and the deletion of the stale
SOPS-era anthropic-cloud/v1 dir were never done. Every cloud-Anthropic
composite (haiku/sonnet/opus code, planning, security-review, spec-
writing, test-writing, best-practices) layered the dead context and
silently never resolved a credential.
- Update 14 composites to layer anthropic-cloud-paul-oauth/v1
- Delete the orphaned harnesses/contexts/anthropic-cloud/v1/ directory
(used pre-Phase-9 SOPS file delivery; now obsolete)
Closes BUG-22b. Pairs with the live PATCH /v1/accounts/{id} rename
shipped under BUG-22a.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Same bug as upstream agent-runtimes: scoped form never matched
because the dispatcher does literal scope.id compare and
owner_principal_id stores the user UUID, not 'paul'.
Pairs with PATCH /v1/accounts/{id} renaming the live account_label
to 'anthropic-cloud-paul-oauth' so the bare ref resolves uniquely.
Closes BUG-22a.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Mirrors agent-runtimes audit + fix. All three M22 Phase 9 wrapper-script
auth paths were broken since the cutover: each one ran as the agent UID
trying to `cat` a 0400 root-only ESO mount. Same root cause we hit on
minimax/z-ai earlier today.
Changes:
1. anthropic-cloud-paul-oauth/v1 (NEW in framework)
- Mirrors agent-runtimes — was previously only present there.
- init.sh stages oauth_token into ~/.claude/.credentials.json (Claude
Code's native subscription-OAuth schema). No env, no apiKeyHelper,
no wrapper. Restores the equivalent of what harness_init.py used to
do for the legacy SOPS path.
- The legacy `bin/anthropic-wrapper.sh` was dead code (never wired).
2. gitea-https/v1
- init.sh stages the token to $HOME/.config/git/gitea-https-token
(0600 agent:agent) and points the per-host git credential helper at
the staged copy. Previously the helper `cat`d the ESO mount path
and silently failed at every git invocation.
3. gitea-admin/v1
- init.sh stages the token to $HOME/.config/gitea-admin/token
(0600 agent:agent). Wrapper updated to read from the staged copy.
- Removes stale `requires: anthropic-cloud/v1` (the only anthropic
harness in agent-runtimes is anthropic-cloud-paul-oauth/v1).
Pattern matches gitea-ssh / minimax / z-ai: ESO mount stays root-only,
init.sh runs as root and `install -m 0600 -o agent -g agent`s a single
explicit copy. Per-secret enumeration; future ESO Secret keys remain
inaccessible by default.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
