skynet/agent-runtime-framework
3
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

benchmarks: add Artificial Analysis snapshot and placeholder M3.1-Flash estimate

Browse Source 
- artificial-analysis/2026-10-13.csv: Anthropic, Qwen, MiniMax-M3 rows with
  cost per task and derived cost per intelligence point
- minimax-m3-1-flash-unverified.md: operator-supplied KingBench figures, unverified
- estimates.csv: TODO placeholder for M3.1 Flash (0.9x M3 cost, index scaled by
  KingBench ratio) -- replace with measured/published values
This commit is contained in:
Paul O'Reilly
2026-10-13 14:33:53 +13:00
parent d7decc1dec
commit 418d634afa
4 changed files with 88 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

23
benchmarks/minimax-m3-1-flash-unverified.md Normal file
View File

@@ -0,0 +1,23 @@
# MiniMax-M3.1-Flash-Preview — ad-hoc, UNVERIFIED figures
Supplied by the operator on 2026-10-13 with no source URL. Not from Artificial Analysis (the model is not on
that leaderboard yet — see `artificial-analysis/README.md`). Do not use for registry scores until confirmed.
| Item | Figure | Note |
|---|---|---|
| KingBench 3, M3.1 Flash | 66.25% (53/80) | "community and third-party testing"; arithmetic checks (53/80 = 66.25%) |
| KingBench 3, M3 baseline | 31.25% (25/80) | arithmetic checks; the source also mentions "alternative baseline figures depending on the evaluation set" |
| M3 SWE-bench Pro | ~59% | reference baseline for M3, described as official |
| M3 "Browse.com" | 83.5 | benchmark name looks mistyped (BrowseComp?) — confirm before citing |
| M3 Terminal Bench 2.1 | "strong" | no number given |
Caveats: unknown effort level for the KingBench runs (Flash always thinks; server default is max); KingBench 3 is
small (80 items) so one item is 1.25 points; none of these figures share a scale with the AA Intelligence Index
(M3 = 29 there), so they cannot be mixed into one cost-per-intelligence figure.
## TODO(revisit) — placeholder estimate in use (2026-10-13)
`estimates.csv` carries a stop-gap row for M3.1 Flash: cost per task = 90% of M3's ($0.51 -> $0.46) and
intelligence index = M3's AA index (29) scaled by the KingBench-3 ratio (66.25/31.25 = 2.12) = 61.5. That index
would rank it above Claude Opus 5.5 max (58), which is implausible for a Flash-class model — treat both numbers as
rough placeholders. Replace with (a) the Artificial Analysis row when the model is listed, or (b) our own measured
cost per task at each effort level (fixed task set via `ask-minimax-run --output-format json`).