diff --git a/benchmarks/artificial-analysis/2026-10-13.csv b/benchmarks/artificial-analysis/2026-10-13.csv new file mode 100644 index 0000000..146473f --- /dev/null +++ b/benchmarks/artificial-analysis/2026-10-13.csv @@ -0,0 +1,35 @@ +model,creator,context,intelligence_index,cost_per_task_usd,cost_per_intelligence_point_usd,median_tokens_per_s +Claude Opus 5.5 (max with fallback),Anthropic,1M,58,5.98,0.1031,96 +Claude Sonnet 5.5 (max with fallback),Anthropic,1M,56,5.46,0.0975,135 +Claude Opus 5.5 (xhigh with fallback),Anthropic,1M,56,3.46,0.0618,81 +Claude Opus 5.5 (high with fallback),Anthropic,1M,54,1.82,0.0337,82 +Claude Fable 5.1 (max with fallback),Anthropic,1M,53,7.63,0.1440,67 +Claude Fable 5.1 (xhigh with fallback),Anthropic,1M,53,5.98,0.1128,62 +Claude Sonnet 5.5 (xhigh with fallback),Anthropic,1M,52,2.01,0.0387,103 +Claude Opus 5.5 (medium with fallback),Anthropic,1M,51,1.34,0.0263,79 +Claude Fable 5.1 (high with fallback),Anthropic,1M,51,3.91,0.0767,52 +Claude Fable 5.1 (medium with fallback),Anthropic,1M,49,2.98,0.0608,54 +Claude Fable 5.1 (low with fallback),Anthropic,1M,47,2.37,0.0504,56 +Claude Sonnet 5.5 (high with fallback),Anthropic,1M,47,0.88,0.0187,96 +Claude Haiku 5.5 (max),Anthropic,1M,43,0.21,0.0049,236 +Claude Opus 5.5 (low with fallback),Anthropic,1M,42,0.55,0.0131,80 +Claude Haiku 5.5 (xhigh),Anthropic,1M,41,0.12,0.0029,187 +Claude Sonnet 5.5 (medium with fallback),Anthropic,1M,41,0.48,0.0117,95 +Claude Haiku 5.5 (high),Anthropic,1M,38,0.08,0.0021,175 +Claude Sonnet 5.5 (low with fallback),Anthropic,1M,36,0.35,0.0097,100 +Claude Haiku 5.5 (medium),Anthropic,1M,34,0.05,0.0015,171 +Claude Haiku 5.5 (low),Anthropic,1M,29,0.02,0.0007,164 +MiniMax-M3,MiniMax,1M,29,0.51,0.0176,107 +Qwen3.8 Max (0902),Alibaba,984k,45,5.41,0.1202,36 +Qwen3.8 2.4T A95B,Alibaba,262k,40,2.16,0.0540,37 +Qwen3.8-Flash-Next,Alibaba,256k,40,0.37,0.0092,51 +Qwen3.8 27B (xhigh),Alibaba,256k,34,1.01,0.0297,46 +Qwen3.8 27B (medium),Alibaba,256k,28,1.13,0.0404,50 +Qwen3.8 27B (low),Alibaba,256k,26,1.05,0.0404,53 +Qwen3.7 Plus,Alibaba,1M,25,0.22,0.0088,55 +Qwen3.8 27B (non-reasoning),Alibaba,256k,20,2.49,0.1245,52 +Qwen3.5 397B A17B,Alibaba,262k,18,0.47,0.0261,88 +Qwen3.6 35B A3B,Alibaba,262k,18,0.48,0.0267,129 +Qwen3.5 122B A10B,Alibaba,262k,16,0.32,0.0200,129 +Qwen3.5 9B,Alibaba,262k,11,0.21,0.0191,58 +Qwen3 Coder Next,Alibaba,256k,9,0.78,0.0867,73 diff --git a/benchmarks/artificial-analysis/README.md b/benchmarks/artificial-analysis/README.md new file mode 100644 index 0000000..63131b6 --- /dev/null +++ b/benchmarks/artificial-analysis/README.md @@ -0,0 +1,28 @@ +# Artificial Analysis benchmark snapshot + +Reference data for the model-registry scores/costs pass. NOT read by any loader +(RegistryEntry ignores unknown fields, so this lives beside the registry, not in it). + +- **Source:** https://artificialanalysis.ai/leaderboards/models, extracted from the rendered + table on 2026-10-13 (Anthropic, Alibaba/Qwen and MiniMax rows only; 34 rows with a score and a cost). +- **File:** `2026-10-13.csv` — `intelligence_index` is the AA Intelligence Index; `cost_per_task_usd` is the + site's "Cost per Task (USD)". +- **`cost_per_intelligence_point_usd`** = `cost_per_task_usd / intelligence_index`. Derived here, not a site + column. Chosen as the cost factor to record (user decision, 2026-10-13). +- Rows the site marks with `*` (partial results, no cost shown) are omitted. +- Names are the site's labels. "with fallback" on most Claude rows is the site's wording; effort level is the + bracketed suffix. +- Attribution: data © Artificial Analysis. Re-extract rather than editing by hand. + +## Mapping to registry slugs (informational, not authoritative) + +| Registry slug | AA row(s) | Note | +|---|---|---| +| `minimax-m3`, `minimax-m3-max` | MiniMax-M3 (index 29, $0.51, 0.0176) | AA lists one M3 row with no effort level; both slugs share it | +| `minimax-m31-flash-{max,medium}` | none (see `../estimates.csv`) | MiniMax-M3.1-Flash-Preview is NOT on the leaderboard yet (checked 2026-10-13; no model page). **TODO(revisit):** `../estimates.csv` holds a placeholder estimate (cost 0.9x M3, index scaled by the KingBench ratio) — replace with measured values | +| `airouter-qwen38`, `airouter-qwen38-max` | Qwen3.8 family | `default_flags.model: "Qwen3.8"` does not say which size; AA has Qwen3.8 27B (low/medium/xhigh), 2.4T A95B, Max (0902), Flash-Next — confirm before mapping | +| `claude-fable-5-1` | Claude Fable 5.1 (low..max) | registry sets `reasoning_effort: low` (inert on the claude runner) | +| `claude-sonnet-5`, `claude-opus-5`, `claude-haiku-4-5` | Sonnet 5.5, Opus 5.5, Haiku 5.5 | slug/model-id names predate the 5.5 releases (`default_flags.model` is `claude-sonnet-5` etc.); verify which API model each actually calls before using these rows | + +Cost per task is not comparable to `cost_per_1k_tokens`: it folds in how many tokens the model spends. Use it +for `cost_efficiency` scoring, not for the per-token price fields. diff --git a/benchmarks/estimates.csv b/benchmarks/estimates.csv new file mode 100644 index 0000000..1f12daa --- /dev/null +++ b/benchmarks/estimates.csv @@ -0,0 +1,2 @@ +model,creator,context,intelligence_index,cost_per_task_usd,cost_per_intelligence_point_usd,status,basis +MiniMax-M3.1-Flash-Preview (effort unspecified),MiniMax,1M,61.5,0.46,0.0075,ESTIMATE-TODO,"cost = 0.9 x MiniMax-M3 AA cost per task ($0.51); index = M3 AA index (29) x KingBench-3 ratio (66.25/31.25 = 2.12); both inputs unverified/placeholder" diff --git a/benchmarks/minimax-m3-1-flash-unverified.md b/benchmarks/minimax-m3-1-flash-unverified.md new file mode 100644 index 0000000..ffcc7fd --- /dev/null +++ b/benchmarks/minimax-m3-1-flash-unverified.md @@ -0,0 +1,23 @@ +# MiniMax-M3.1-Flash-Preview — ad-hoc, UNVERIFIED figures + +Supplied by the operator on 2026-10-13 with no source URL. Not from Artificial Analysis (the model is not on +that leaderboard yet — see `artificial-analysis/README.md`). Do not use for registry scores until confirmed. + +| Item | Figure | Note | +|---|---|---| +| KingBench 3, M3.1 Flash | 66.25% (53/80) | "community and third-party testing"; arithmetic checks (53/80 = 66.25%) | +| KingBench 3, M3 baseline | 31.25% (25/80) | arithmetic checks; the source also mentions "alternative baseline figures depending on the evaluation set" | +| M3 SWE-bench Pro | ~59% | reference baseline for M3, described as official | +| M3 "Browse.com" | 83.5 | benchmark name looks mistyped (BrowseComp?) — confirm before citing | +| M3 Terminal Bench 2.1 | "strong" | no number given | + +Caveats: unknown effort level for the KingBench runs (Flash always thinks; server default is max); KingBench 3 is +small (80 items) so one item is 1.25 points; none of these figures share a scale with the AA Intelligence Index +(M3 = 29 there), so they cannot be mixed into one cost-per-intelligence figure. + +## TODO(revisit) — placeholder estimate in use (2026-10-13) +`estimates.csv` carries a stop-gap row for M3.1 Flash: cost per task = 90% of M3's ($0.51 -> $0.46) and +intelligence index = M3's AA index (29) scaled by the KingBench-3 ratio (66.25/31.25 = 2.12) = 61.5. That index +would rank it above Claude Opus 5.5 max (58), which is implausible for a Flash-class model — treat both numbers as +rough placeholders. Replace with (a) the Artificial Analysis row when the model is listed, or (b) our own measured +cost per task at each effort level (fixed task set via `ask-minimax-run --output-format json`).