diff --git a/CLAUDE.md b/CLAUDE.md index 7c469b9..ee70601 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -49,8 +49,12 @@ task-templates/ # dispatch-task --template YAML files workflows/ # Workflow DAG definitions model-registry/ # Model scoring YAMLs (capability dimensions) models/ # Model endpoint YAMLs (URLs, auth, parameters) +benchmarks/ # Reference data for the scores/costs pass (NOT read by any loader): + # artificial-analysis/ snapshot + README, unverified/estimate files ``` +**Model slugs:** a registry slug is not a provider model id. MiniMax silently maps unknown model names to MiniMax-M3, so each `models/.yaml` must carry the real id in `default_flags.model` (agent-runtimes translates slug -> id on every runner). One `models/` + `model-registry/` file per effort level (e.g. `minimax-m31-flash-{max,medium}`). + ## Harness Conventions ### harness.yaml fields diff --git a/CONTEXT.md b/CONTEXT.md new file mode 100644 index 0000000..09c51f3 --- /dev/null +++ b/CONTEXT.md @@ -0,0 +1,7 @@ +# CONTEXT.md — agent-runtime-framework + +## Active + +- **MiniMax Flash migration done; scores/costs are placeholders.** `minimax-m31-flash-{max,medium}` (models/ + model-registry/) and context `minimax-m31-flash/v1` are live on the CP (d7decc1). Registry scores are copies of the M3 entries with costs just below them. Planner FUTURE `629829d3` (agent-runtimes) tracks replacing them and the stop-gap `benchmarks/estimates.csv` M3.1-Flash row (0.9x M3 cost, index 61.5 — implausible). See `benchmarks/artificial-analysis/README.md` for the AA snapshot, slug mapping and caveats. +- **Unverified end-to-end:** dispatch a real hatchmate task and check the model the API reports is `MiniMax-M3.1-Flash-Preview` (agent-runtimes 0.0.5.1817 translates slug -> provider id on all runners). +- Untracked, left alone: `.claude/` (worktrees), `tests/__init__.py`. diff --git a/memory/log/2026-05-04.225112.md b/memory/log/2026-05-04.225112.md deleted file mode 100644 index e8bc286..0000000 --- a/memory/log/2026-05-04.225112.md +++ /dev/null @@ -1,27 +0,0 @@ -# Session Log -- 2026-05-04 - -## Summary -Implemented 5 planned fixes for BUG-5 (`.agent-output/` silent drops) and BUG-20 (MiniMax finalize no-push): pre_test.sh harness hook, output validation in finalize, push retry with debug tracing, spec-planning workflow output tags, and wrong-path detection with agent correction re-invoke. Fork cleanup PR merged, agent-monitor PR #44 merged. - -## Decisions -- Fork cleanup workflow (PR #1 on agent-runtimes-agents): weekly cron deletes `task-*` branches inactive for 7+ days; main reset to orphan commit if no commits for 7+ days. Supports dry-run via workflow_dispatch. **Merged** (2026-05-04 10:39 UTC by admin). -- Gitea API token auth requires HTTP Basic Auth (`auth=("user","pass")`) not Bearer token — Bearer returns 401 even with valid access token. Use `httpx.get(..., auth=("user","pass"))` pattern. -- Shallow clone (`--depth 1`) for task branches in fork cleanup to avoid bloating with full history. - -## Gotchas Discovered -- **[Gitea Gotchas]** Gitea API endpoint `/login` is `404 Not Found` — use `/users//access_tokens` to create tokens, or use HTTP Basic Auth directly against any `/api/v1/` endpoint with `auth=("user","pass")`. -- **[Gitea Gotchas]** PR state `closed` with `merged: true` means the PR was merged (not just closed). Gitea distinguishes merged vs closed states. -- **[Gitea Gotchas]** Fork PRs use the parent repo's Actions (has_actions=true on parent, `has_actions: false` on fork). Fork cleanup workflow runs on the fork's GHA because fork enables Actions independently. -- **[tokens]** Token file at `~/.config/agent-runtimes/tokens.json` uses flat structure: `{"access_token": "...", "expires_at": N}`. Token was valid (not expired at session continuation time: now 1777891716 vs expiry 1777891804). - -## Key Context -- **agent-runtime-framework** `8bbf6cb`: pre_test.sh created (harness-level test folder revert), finalize.sh gains `set -x` tracing, push retry (`AGENT_PUSH_RETRIES`), output validation (`AGENT_EXPECTED_OUTPUT`), wrong-path detection + `.correction-prompt.txt` sentinel for entrypoint re-invoke. All 17 spec-planning workflow nodes tagged with `output: {path, min_bytes}`. -- **agent-runtimes-agents** fork: `fork-cleanup` branch merged to main (PR #1 by ai_enablement, merged by admin 2026-05-04). Workflow file at `.gitea/workflows/cleanup.yaml`. -- **agent-runtimes** `148d9d3`: PR #44 merged — `scripts/agent-monitor` gains `provider` and `model_full` columns. -- **agent-runtimes** local uncommitted: M22 Phase 3 manifests work (`controlplane/api/manifests.py`, `controlplane/db/manifest_store.py`, migration `20260504_0017_m16_phase3_manifests.py`, `controlplane/manifests/`). Not related to this session's focus. -- Two ops gates pending before M22 Phase 9 live cutover: CI build `agent-runtimes-init:1.0.0` image; `agent-session` Role deployed via `homelab/agent-runtimes-deploy` + ArgoCD sync. - -## Process Notes -- When verifying PR state via Gitea API: parse JSON directly (don't assume specific keys — use `json.dumps(..., indent=2)` to inspect full structure). -- Fork cleanup workflow commits to `agent-runtimes-agents` fork main; parent `agent-runtimes` main unchanged. Fork's main tracked via `agent-runtimes-agents` remote in local `agent-runtime-framework` checkout. -- Fork cleanup's `fetch-depth: 0` is needed for commit date inspection even with shallow clones elsewhere. diff --git a/memory/log/2026-10-13.144722.md b/memory/log/2026-10-13.144722.md new file mode 100644 index 0000000..227d75e --- /dev/null +++ b/memory/log/2026-10-13.144722.md @@ -0,0 +1,32 @@ +# Session Log -- 2026-10-13 + +## Summary +Moved the framework's MiniMax content to MiniMax-M3.1-Flash-Preview (new context, Flash models/registry, bare `minimax`/`minimax-m2` slugs removed; d7decc1), then added a `benchmarks/` directory with an Artificial Analysis snapshot, unverified M3.1 Flash figures, and a flagged placeholder estimate (418d634). Both pushed to `main`. + +## Decisions +- Decision: new context `harnesses/contexts/minimax-m31-flash/v1` (env pin `MiniMax-M3.1-Flash-Preview`, no init.sh) replaces `minimax-m3-max/v1` in scaffolding-repo, integration-direct, test-writing-repo, best-practices-minimax-repo; the old context stays for rollback -- Rationale: Flash ignores MAX_THINKING_TOKENS; effort comes from `--effort` via `default_flags.reasoning_effort`. +- Decision: deleted `model-registry/minimax.yaml` (endpoint `minimax` never existed) and repointed the six `task-templates/minimax-*.yaml` and the three `*-review-loop@1` `model: minimax-m2` nodes to `minimax-m31-flash-max` -- Rationale: MiniMax silently maps unknown model names to M3. +- Decision: benchmark data lives in `benchmarks/` beside the registry, not inside registry entries -- Rationale: `RegistryEntry` (pydantic default extra=ignore) silently drops unknown fields, and the loader never reads this dir. +- Decision: cost factor recorded is "cost per intelligence point" = AA cost per task / AA Intelligence Index (derived; the site has no such column) -- Rationale: operator request. +- Decision: M3.1 Flash placeholder = 0.9x M3 cost per task ($0.46) and M3's index scaled by the KingBench-3 ratio (29 x 2.12 = 61.5), flagged `ESTIMATE-TODO` -- Rationale: operator stop-gap; 61.5 exceeds Opus 5.5 max (58), so implausible. Planner FUTURE item 629829d3 (agent-runtimes) tracks replacing it. + +## Gotchas Discovered +- **[minimax]** Symptom: unknown model names (`minimax`, `minimax-m31-flash-max`) return HTTP 200 with `model: MiniMax-M3` -- Fix: send the provider id; translation now done by agent-runtimes ef822703 (deployed 0.0.5.1817). Check the echoed `model`. +- **[artificial-analysis]** Symptom: WebFetch of artificialanalysis.ai/models returns only headline numbers (tables are JS-rendered) -- Fix: open the page in Chrome (claude-in-chrome), wait ~4 s, read `document.querySelector('table')` rows via javascript_tool (269 rows on /leaderboards/models; columns: model, context, creator, index, cost per task, tokens/s, latency). +- **[artificial-analysis]** MiniMax-M3.1-Flash-Preview is not on the leaderboard (no row; /models/minimax-m3-1-* 404 as of 2026-10-13); only `MiniMax-M3` (index 29, $0.51/task) exists for MiniMax. +- **[benchmarks]** Cost vs intelligence correlates strongly only within one model family across effort levels (Claude: r 0.93-0.99, cost x4-6 per +10 index points); across vendors it is weak (r 0.53) and Qwen3.8 27B is inverted (cost does not fall with lower effort), so cross-vendor cost estimates for M3.1 Flash are unreliable. +- **[git]** The framework checkout was on the user's feature branch `feat/hatchmate-per-project-fork-key`; moving my uncommitted work to `main` needed `git stash push -- ` (untracked new files stay in place), `checkout main`, ff-pull, `stash pop`. + +## Open Questions +- Which Qwen3.8 size do `airouter-qwen38*` models actually call? (AA has 27B, 2.4T A95B, Max, Flash-Next.) +- Registry model ids `claude-sonnet-5` / `claude-opus-5` / `claude-haiku-4-5` predate the 5.5 releases AA lists -- verify before mapping AA rows to them. +- Source for the KingBench 3 figures; scoring rule to map AA index / cost per intelligence point onto the registry's 0-10 dimensions. + +## Key Context +- Files: `benchmarks/artificial-analysis/{2026-10-13.csv,README.md}` (34 rows: Anthropic, Qwen, MiniMax-M3), `benchmarks/minimax-m3-1-flash-unverified.md`, `benchmarks/estimates.csv`. +- Live: CRS checkout for this repo is at d7decc1 on the controlplane (verified); 418d634 adds only benchmarks (nothing reads it). +- Registry scores/costs for `minimax-m31-flash-{max,medium}` are still placeholders (copies of the M3 entries; costs just under them). + +## Process Notes +- Before rebasing onto a feature-branch checkout, check `git branch --show-current`; land work on `main` explicitly. +- Record derived metrics with their derivation (README) and mark stop-gap numbers with a TODO in the data file name/status column AND a durable planner item.