skynet/agent-runtime-framework
3
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: end-session 2026-10-13 (CONTEXT.md, session log, benchmarks/ + model-slug note in CLAUDE.md)

Browse Source
This commit is contained in:
Paul O'Reilly
2026-10-13 14:47:57 +13:00
parent 418d634afa
commit 44334a2143
4 changed files with 43 additions and 27 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
CLAUDE.md
View File

@@ -49,8 +49,12 @@ task-templates/ # dispatch-task --template <name> YAML files
workflows/ # Workflow DAG definitions
model-registry/ # Model scoring YAMLs (capability dimensions)
models/ # Model endpoint YAMLs (URLs, auth, parameters)
benchmarks/ # Reference data for the scores/costs pass (NOT read by any loader):
# artificial-analysis/ snapshot + README, unverified/estimate files
```
**Model slugs:** a registry slug is not a provider model id. MiniMax silently maps unknown model names to MiniMax-M3, so each `models/<slug>.yaml` must carry the real id in `default_flags.model` (agent-runtimes translates slug -> id on every runner). One `models/` + `model-registry/` file per effort level (e.g. `minimax-m31-flash-{max,medium}`).
## Harness Conventions
### harness.yaml fields

7
CONTEXT.md Normal file
View File

@@ -0,0 +1,7 @@
# CONTEXT.md — agent-runtime-framework
## Active
- **MiniMax Flash migration done; scores/costs are placeholders.** `minimax-m31-flash-{max,medium}` (models/ + model-registry/) and context `minimax-m31-flash/v1` are live on the CP (d7decc1). Registry scores are copies of the M3 entries with costs just below them. Planner FUTURE `629829d3` (agent-runtimes) tracks replacing them and the stop-gap `benchmarks/estimates.csv` M3.1-Flash row (0.9x M3 cost, index 61.5 — implausible). See `benchmarks/artificial-analysis/README.md` for the AA snapshot, slug mapping and caveats.
- **Unverified end-to-end:** dispatch a real hatchmate task and check the model the API reports is `MiniMax-M3.1-Flash-Preview` (agent-runtimes 0.0.5.1817 translates slug -> provider id on all runners).
- Untracked, left alone: `.claude/` (worktrees), `tests/__init__.py`.

27
memory/log/2026-05-04.225112.md
View File

@@ -1,27 +0,0 @@
# Session Log -- 2026-05-04
## Summary
Implemented 5 planned fixes for BUG-5 (`.agent-output/` silent drops) and BUG-20 (MiniMax finalize no-push): pre_test.sh harness hook, output validation in finalize, push retry with debug tracing, spec-planning workflow output tags, and wrong-path detection with agent correction re-invoke. Fork cleanup PR merged, agent-monitor PR #44 merged.
## Decisions
- Fork cleanup workflow (PR #1 on agent-runtimes-agents): weekly cron deletes `task-*` branches inactive for 7+ days; main reset to orphan commit if no commits for 7+ days. Supports dry-run via workflow_dispatch. **Merged** (2026-05-04 10:39 UTC by admin).
- Gitea API token auth requires HTTP Basic Auth (`auth=("user","pass")`) not Bearer token — Bearer returns 401 even with valid access token. Use `httpx.get(..., auth=("user","pass"))` pattern.
- Shallow clone (`--depth 1`) for task branches in fork cleanup to avoid bloating with full history.
## Gotchas Discovered
- **[Gitea Gotchas]** Gitea API endpoint `/login` is `404 Not Found` — use `/users/<username>/access_tokens` to create tokens, or use HTTP Basic Auth directly against any `/api/v1/` endpoint with `auth=("user","pass")`.
- **[Gitea Gotchas]** PR state `closed` with `merged: true` means the PR was merged (not just closed). Gitea distinguishes merged vs closed states.
- **[Gitea Gotchas]** Fork PRs use the parent repo's Actions (has_actions=true on parent, `has_actions: false` on fork). Fork cleanup workflow runs on the fork's GHA because fork enables Actions independently.
- **[tokens]** Token file at `~/.config/agent-runtimes/tokens.json` uses flat structure: `{"access_token": "...", "expires_at": N}`. Token was valid (not expired at session continuation time: now 1777891716 vs expiry 1777891804).
## Key Context
- **agent-runtime-framework** `8bbf6cb`: pre_test.sh created (harness-level test folder revert), finalize.sh gains `set -x` tracing, push retry (`AGENT_PUSH_RETRIES`), output validation (`AGENT_EXPECTED_OUTPUT`), wrong-path detection + `.correction-prompt.txt` sentinel for entrypoint re-invoke. All 17 spec-planning workflow nodes tagged with `output: {path, min_bytes}`.
- **agent-runtimes-agents** fork: `fork-cleanup` branch merged to main (PR #1 by ai_enablement, merged by admin 2026-05-04). Workflow file at `.gitea/workflows/cleanup.yaml`.
- **agent-runtimes** `148d9d3`: PR #44 merged — `scripts/agent-monitor` gains `provider` and `model_full` columns.
- **agent-runtimes** local uncommitted: M22 Phase 3 manifests work (`controlplane/api/manifests.py`, `controlplane/db/manifest_store.py`, migration `20260504_0017_m16_phase3_manifests.py`, `controlplane/manifests/`). Not related to this session's focus.
- Two ops gates pending before M22 Phase 9 live cutover: CI build `agent-runtimes-init:1.0.0` image; `agent-session` Role deployed via `homelab/agent-runtimes-deploy` + ArgoCD sync.
## Process Notes
- When verifying PR state via Gitea API: parse JSON directly (don't assume specific keys — use `json.dumps(..., indent=2)` to inspect full structure).
- Fork cleanup workflow commits to `agent-runtimes-agents` fork main; parent `agent-runtimes` main unchanged. Fork's main tracked via `agent-runtimes-agents` remote in local `agent-runtime-framework` checkout.
- Fork cleanup's `fetch-depth: 0` is needed for commit date inspection even with shallow clones elsewhere.

32
memory/log/2026-10-13.144722.md Normal file
View File

@@ -0,0 +1,32 @@
# Session Log -- 2026-10-13
## Summary
Moved the framework's MiniMax content to MiniMax-M3.1-Flash-Preview (new context, Flash models/registry, bare `minimax`/`minimax-m2` slugs removed; d7decc1), then added a `benchmarks/` directory with an Artificial Analysis snapshot, unverified M3.1 Flash figures, and a flagged placeholder estimate (418d634). Both pushed to `main`.
## Decisions
- Decision: new context `harnesses/contexts/minimax-m31-flash/v1` (env pin `MiniMax-M3.1-Flash-Preview`, no init.sh) replaces `minimax-m3-max/v1` in scaffolding-repo, integration-direct, test-writing-repo, best-practices-minimax-repo; the old context stays for rollback -- Rationale: Flash ignores MAX_THINKING_TOKENS; effort comes from `--effort` via `default_flags.reasoning_effort`.
- Decision: deleted `model-registry/minimax.yaml` (endpoint `minimax` never existed) and repointed the six `task-templates/minimax-*.yaml` and the three `*-review-loop@1` `model: minimax-m2` nodes to `minimax-m31-flash-max` -- Rationale: MiniMax silently maps unknown model names to M3.
- Decision: benchmark data lives in `benchmarks/` beside the registry, not inside registry entries -- Rationale: `RegistryEntry` (pydantic default extra=ignore) silently drops unknown fields, and the loader never reads this dir.
- Decision: cost factor recorded is "cost per intelligence point" = AA cost per task / AA Intelligence Index (derived; the site has no such column) -- Rationale: operator request.
- Decision: M3.1 Flash placeholder = 0.9x M3 cost per task ($0.46) and M3's index scaled by the KingBench-3 ratio (29 x 2.12 = 61.5), flagged `ESTIMATE-TODO` -- Rationale: operator stop-gap; 61.5 exceeds Opus 5.5 max (58), so implausible. Planner FUTURE item 629829d3 (agent-runtimes) tracks replacing it.
## Gotchas Discovered
- **[minimax]** Symptom: unknown model names (`minimax`, `minimax-m31-flash-max`) return HTTP 200 with `model: MiniMax-M3` -- Fix: send the provider id; translation now done by agent-runtimes ef822703 (deployed 0.0.5.1817). Check the echoed `model`.
- **[artificial-analysis]** Symptom: WebFetch of artificialanalysis.ai/models returns only headline numbers (tables are JS-rendered) -- Fix: open the page in Chrome (claude-in-chrome), wait ~4 s, read `document.querySelector('table')` rows via javascript_tool (269 rows on /leaderboards/models; columns: model, context, creator, index, cost per task, tokens/s, latency).
- **[artificial-analysis]** MiniMax-M3.1-Flash-Preview is not on the leaderboard (no row; /models/minimax-m3-1-* 404 as of 2026-10-13); only `MiniMax-M3` (index 29, $0.51/task) exists for MiniMax.
- **[benchmarks]** Cost vs intelligence correlates strongly only within one model family across effort levels (Claude: r 0.93-0.99, cost x4-6 per +10 index points); across vendors it is weak (r 0.53) and Qwen3.8 27B is inverted (cost does not fall with lower effort), so cross-vendor cost estimates for M3.1 Flash are unreliable.
- **[git]** The framework checkout was on the user's feature branch `feat/hatchmate-per-project-fork-key`; moving my uncommitted work to `main` needed `git stash push -- <paths>` (untracked new files stay in place), `checkout main`, ff-pull, `stash pop`.
## Open Questions
- Which Qwen3.8 size do `airouter-qwen38*` models actually call? (AA has 27B, 2.4T A95B, Max, Flash-Next.)
- Registry model ids `claude-sonnet-5` / `claude-opus-5` / `claude-haiku-4-5` predate the 5.5 releases AA lists -- verify before mapping AA rows to them.
- Source for the KingBench 3 figures; scoring rule to map AA index / cost per intelligence point onto the registry's 0-10 dimensions.
## Key Context
- Files: `benchmarks/artificial-analysis/{2026-10-13.csv,README.md}` (34 rows: Anthropic, Qwen, MiniMax-M3), `benchmarks/minimax-m3-1-flash-unverified.md`, `benchmarks/estimates.csv`.
- Live: CRS checkout for this repo is at d7decc1 on the controlplane (verified); 418d634 adds only benchmarks (nothing reads it).
- Registry scores/costs for `minimax-m31-flash-{max,medium}` are still placeholders (copies of the M3 entries; costs just under them).
## Process Notes
- Before rebasing onto a feature-branch checkout, check `git branch --show-current`; land work on `main` explicitly.
- Record derived metrics with their derivation (README) and mark stop-gap numbers with a TODO in the data file name/status column AND a durable planner item.