skynet/agent-runtime-framework
3
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: end-session 2026-10-13 (CONTEXT.md, session log, benchmarks/ + model-slug note in CLAUDE.md)

Browse Source
This commit is contained in:
Paul O'Reilly
2026-10-13 14:47:57 +13:00
parent 418d634afa
commit 44334a2143
4 changed files with 43 additions and 27 deletions
Download Patch File Download Diff File Expand all files Collapse all files

32
memory/log/2026-10-13.144722.md Normal file
View File

@@ -0,0 +1,32 @@
# Session Log -- 2026-10-13
## Summary
Moved the framework's MiniMax content to MiniMax-M3.1-Flash-Preview (new context, Flash models/registry, bare `minimax`/`minimax-m2` slugs removed; d7decc1), then added a `benchmarks/` directory with an Artificial Analysis snapshot, unverified M3.1 Flash figures, and a flagged placeholder estimate (418d634). Both pushed to `main`.
## Decisions
- Decision: new context `harnesses/contexts/minimax-m31-flash/v1` (env pin `MiniMax-M3.1-Flash-Preview`, no init.sh) replaces `minimax-m3-max/v1` in scaffolding-repo, integration-direct, test-writing-repo, best-practices-minimax-repo; the old context stays for rollback -- Rationale: Flash ignores MAX_THINKING_TOKENS; effort comes from `--effort` via `default_flags.reasoning_effort`.
- Decision: deleted `model-registry/minimax.yaml` (endpoint `minimax` never existed) and repointed the six `task-templates/minimax-*.yaml` and the three `*-review-loop@1` `model: minimax-m2` nodes to `minimax-m31-flash-max` -- Rationale: MiniMax silently maps unknown model names to M3.
- Decision: benchmark data lives in `benchmarks/` beside the registry, not inside registry entries -- Rationale: `RegistryEntry` (pydantic default extra=ignore) silently drops unknown fields, and the loader never reads this dir.
- Decision: cost factor recorded is "cost per intelligence point" = AA cost per task / AA Intelligence Index (derived; the site has no such column) -- Rationale: operator request.
- Decision: M3.1 Flash placeholder = 0.9x M3 cost per task ($0.46) and M3's index scaled by the KingBench-3 ratio (29 x 2.12 = 61.5), flagged `ESTIMATE-TODO` -- Rationale: operator stop-gap; 61.5 exceeds Opus 5.5 max (58), so implausible. Planner FUTURE item 629829d3 (agent-runtimes) tracks replacing it.
## Gotchas Discovered
- **[minimax]** Symptom: unknown model names (`minimax`, `minimax-m31-flash-max`) return HTTP 200 with `model: MiniMax-M3` -- Fix: send the provider id; translation now done by agent-runtimes ef822703 (deployed 0.0.5.1817). Check the echoed `model`.
- **[artificial-analysis]** Symptom: WebFetch of artificialanalysis.ai/models returns only headline numbers (tables are JS-rendered) -- Fix: open the page in Chrome (claude-in-chrome), wait ~4 s, read `document.querySelector('table')` rows via javascript_tool (269 rows on /leaderboards/models; columns: model, context, creator, index, cost per task, tokens/s, latency).
- **[artificial-analysis]** MiniMax-M3.1-Flash-Preview is not on the leaderboard (no row; /models/minimax-m3-1-* 404 as of 2026-10-13); only `MiniMax-M3` (index 29, $0.51/task) exists for MiniMax.
- **[benchmarks]** Cost vs intelligence correlates strongly only within one model family across effort levels (Claude: r 0.93-0.99, cost x4-6 per +10 index points); across vendors it is weak (r 0.53) and Qwen3.8 27B is inverted (cost does not fall with lower effort), so cross-vendor cost estimates for M3.1 Flash are unreliable.
- **[git]** The framework checkout was on the user's feature branch `feat/hatchmate-per-project-fork-key`; moving my uncommitted work to `main` needed `git stash push -- <paths>` (untracked new files stay in place), `checkout main`, ff-pull, `stash pop`.
## Open Questions
- Which Qwen3.8 size do `airouter-qwen38*` models actually call? (AA has 27B, 2.4T A95B, Max, Flash-Next.)
- Registry model ids `claude-sonnet-5` / `claude-opus-5` / `claude-haiku-4-5` predate the 5.5 releases AA lists -- verify before mapping AA rows to them.
- Source for the KingBench 3 figures; scoring rule to map AA index / cost per intelligence point onto the registry's 0-10 dimensions.
## Key Context
- Files: `benchmarks/artificial-analysis/{2026-10-13.csv,README.md}` (34 rows: Anthropic, Qwen, MiniMax-M3), `benchmarks/minimax-m3-1-flash-unverified.md`, `benchmarks/estimates.csv`.
- Live: CRS checkout for this repo is at d7decc1 on the controlplane (verified); 418d634 adds only benchmarks (nothing reads it).
- Registry scores/costs for `minimax-m31-flash-{max,medium}` are still placeholders (copies of the M3 entries; costs just under them).
## Process Notes
- Before rebasing onto a feature-branch checkout, check `git branch --show-current`; land work on `main` explicitly.
- Record derived metrics with their derivation (README) and mark stop-gap numbers with a TODO in the data file name/status column AND a durable planner item.