benchmarks: add Artificial Analysis snapshot and placeholder M3.1-Flash estimate
- artificial-analysis/2026-10-13.csv: Anthropic, Qwen, MiniMax-M3 rows with cost per task and derived cost per intelligence point - minimax-m3-1-flash-unverified.md: operator-supplied KingBench figures, unverified - estimates.csv: TODO placeholder for M3.1 Flash (0.9x M3 cost, index scaled by KingBench ratio) -- replace with measured/published values
This commit is contained in:
28
benchmarks/artificial-analysis/README.md
Normal file
28
benchmarks/artificial-analysis/README.md
Normal file
@@ -0,0 +1,28 @@
|
||||
# Artificial Analysis benchmark snapshot
|
||||
|
||||
Reference data for the model-registry scores/costs pass. NOT read by any loader
|
||||
(RegistryEntry ignores unknown fields, so this lives beside the registry, not in it).
|
||||
|
||||
- **Source:** https://artificialanalysis.ai/leaderboards/models, extracted from the rendered
|
||||
table on 2026-10-13 (Anthropic, Alibaba/Qwen and MiniMax rows only; 34 rows with a score and a cost).
|
||||
- **File:** `2026-10-13.csv` — `intelligence_index` is the AA Intelligence Index; `cost_per_task_usd` is the
|
||||
site's "Cost per Task (USD)".
|
||||
- **`cost_per_intelligence_point_usd`** = `cost_per_task_usd / intelligence_index`. Derived here, not a site
|
||||
column. Chosen as the cost factor to record (user decision, 2026-10-13).
|
||||
- Rows the site marks with `*` (partial results, no cost shown) are omitted.
|
||||
- Names are the site's labels. "with fallback" on most Claude rows is the site's wording; effort level is the
|
||||
bracketed suffix.
|
||||
- Attribution: data © Artificial Analysis. Re-extract rather than editing by hand.
|
||||
|
||||
## Mapping to registry slugs (informational, not authoritative)
|
||||
|
||||
| Registry slug | AA row(s) | Note |
|
||||
|---|---|---|
|
||||
| `minimax-m3`, `minimax-m3-max` | MiniMax-M3 (index 29, $0.51, 0.0176) | AA lists one M3 row with no effort level; both slugs share it |
|
||||
| `minimax-m31-flash-{max,medium}` | none (see `../estimates.csv`) | MiniMax-M3.1-Flash-Preview is NOT on the leaderboard yet (checked 2026-10-13; no model page). **TODO(revisit):** `../estimates.csv` holds a placeholder estimate (cost 0.9x M3, index scaled by the KingBench ratio) — replace with measured values |
|
||||
| `airouter-qwen38`, `airouter-qwen38-max` | Qwen3.8 family | `default_flags.model: "Qwen3.8"` does not say which size; AA has Qwen3.8 27B (low/medium/xhigh), 2.4T A95B, Max (0902), Flash-Next — confirm before mapping |
|
||||
| `claude-fable-5-1` | Claude Fable 5.1 (low..max) | registry sets `reasoning_effort: low` (inert on the claude runner) |
|
||||
| `claude-sonnet-5`, `claude-opus-5`, `claude-haiku-4-5` | Sonnet 5.5, Opus 5.5, Haiku 5.5 | slug/model-id names predate the 5.5 releases (`default_flags.model` is `claude-sonnet-5` etc.); verify which API model each actually calls before using these rows |
|
||||
|
||||
Cost per task is not comparable to `cost_per_1k_tokens`: it folds in how many tokens the model spends. Use it
|
||||
for `cost_efficiency` scoring, not for the per-token price fields.
|
||||
Reference in New Issue
Block a user