benchmarks: add Artificial Analysis snapshot and placeholder M3.1-Flash estimate
- artificial-analysis/2026-10-13.csv: Anthropic, Qwen, MiniMax-M3 rows with cost per task and derived cost per intelligence point - minimax-m3-1-flash-unverified.md: operator-supplied KingBench figures, unverified - estimates.csv: TODO placeholder for M3.1 Flash (0.9x M3 cost, index scaled by KingBench ratio) -- replace with measured/published values
This commit is contained in:
35
benchmarks/artificial-analysis/2026-10-13.csv
Normal file
35
benchmarks/artificial-analysis/2026-10-13.csv
Normal file
@@ -0,0 +1,35 @@
|
||||
model,creator,context,intelligence_index,cost_per_task_usd,cost_per_intelligence_point_usd,median_tokens_per_s
|
||||
Claude Opus 5.5 (max with fallback),Anthropic,1M,58,5.98,0.1031,96
|
||||
Claude Sonnet 5.5 (max with fallback),Anthropic,1M,56,5.46,0.0975,135
|
||||
Claude Opus 5.5 (xhigh with fallback),Anthropic,1M,56,3.46,0.0618,81
|
||||
Claude Opus 5.5 (high with fallback),Anthropic,1M,54,1.82,0.0337,82
|
||||
Claude Fable 5.1 (max with fallback),Anthropic,1M,53,7.63,0.1440,67
|
||||
Claude Fable 5.1 (xhigh with fallback),Anthropic,1M,53,5.98,0.1128,62
|
||||
Claude Sonnet 5.5 (xhigh with fallback),Anthropic,1M,52,2.01,0.0387,103
|
||||
Claude Opus 5.5 (medium with fallback),Anthropic,1M,51,1.34,0.0263,79
|
||||
Claude Fable 5.1 (high with fallback),Anthropic,1M,51,3.91,0.0767,52
|
||||
Claude Fable 5.1 (medium with fallback),Anthropic,1M,49,2.98,0.0608,54
|
||||
Claude Fable 5.1 (low with fallback),Anthropic,1M,47,2.37,0.0504,56
|
||||
Claude Sonnet 5.5 (high with fallback),Anthropic,1M,47,0.88,0.0187,96
|
||||
Claude Haiku 5.5 (max),Anthropic,1M,43,0.21,0.0049,236
|
||||
Claude Opus 5.5 (low with fallback),Anthropic,1M,42,0.55,0.0131,80
|
||||
Claude Haiku 5.5 (xhigh),Anthropic,1M,41,0.12,0.0029,187
|
||||
Claude Sonnet 5.5 (medium with fallback),Anthropic,1M,41,0.48,0.0117,95
|
||||
Claude Haiku 5.5 (high),Anthropic,1M,38,0.08,0.0021,175
|
||||
Claude Sonnet 5.5 (low with fallback),Anthropic,1M,36,0.35,0.0097,100
|
||||
Claude Haiku 5.5 (medium),Anthropic,1M,34,0.05,0.0015,171
|
||||
Claude Haiku 5.5 (low),Anthropic,1M,29,0.02,0.0007,164
|
||||
MiniMax-M3,MiniMax,1M,29,0.51,0.0176,107
|
||||
Qwen3.8 Max (0902),Alibaba,984k,45,5.41,0.1202,36
|
||||
Qwen3.8 2.4T A95B,Alibaba,262k,40,2.16,0.0540,37
|
||||
Qwen3.8-Flash-Next,Alibaba,256k,40,0.37,0.0092,51
|
||||
Qwen3.8 27B (xhigh),Alibaba,256k,34,1.01,0.0297,46
|
||||
Qwen3.8 27B (medium),Alibaba,256k,28,1.13,0.0404,50
|
||||
Qwen3.8 27B (low),Alibaba,256k,26,1.05,0.0404,53
|
||||
Qwen3.7 Plus,Alibaba,1M,25,0.22,0.0088,55
|
||||
Qwen3.8 27B (non-reasoning),Alibaba,256k,20,2.49,0.1245,52
|
||||
Qwen3.5 397B A17B,Alibaba,262k,18,0.47,0.0261,88
|
||||
Qwen3.6 35B A3B,Alibaba,262k,18,0.48,0.0267,129
|
||||
Qwen3.5 122B A10B,Alibaba,262k,16,0.32,0.0200,129
|
||||
Qwen3.5 9B,Alibaba,262k,11,0.21,0.0191,58
|
||||
Qwen3 Coder Next,Alibaba,256k,9,0.78,0.0867,73
|
||||
|
28
benchmarks/artificial-analysis/README.md
Normal file
28
benchmarks/artificial-analysis/README.md
Normal file
@@ -0,0 +1,28 @@
|
||||
# Artificial Analysis benchmark snapshot
|
||||
|
||||
Reference data for the model-registry scores/costs pass. NOT read by any loader
|
||||
(RegistryEntry ignores unknown fields, so this lives beside the registry, not in it).
|
||||
|
||||
- **Source:** https://artificialanalysis.ai/leaderboards/models, extracted from the rendered
|
||||
table on 2026-10-13 (Anthropic, Alibaba/Qwen and MiniMax rows only; 34 rows with a score and a cost).
|
||||
- **File:** `2026-10-13.csv` — `intelligence_index` is the AA Intelligence Index; `cost_per_task_usd` is the
|
||||
site's "Cost per Task (USD)".
|
||||
- **`cost_per_intelligence_point_usd`** = `cost_per_task_usd / intelligence_index`. Derived here, not a site
|
||||
column. Chosen as the cost factor to record (user decision, 2026-10-13).
|
||||
- Rows the site marks with `*` (partial results, no cost shown) are omitted.
|
||||
- Names are the site's labels. "with fallback" on most Claude rows is the site's wording; effort level is the
|
||||
bracketed suffix.
|
||||
- Attribution: data © Artificial Analysis. Re-extract rather than editing by hand.
|
||||
|
||||
## Mapping to registry slugs (informational, not authoritative)
|
||||
|
||||
| Registry slug | AA row(s) | Note |
|
||||
|---|---|---|
|
||||
| `minimax-m3`, `minimax-m3-max` | MiniMax-M3 (index 29, $0.51, 0.0176) | AA lists one M3 row with no effort level; both slugs share it |
|
||||
| `minimax-m31-flash-{max,medium}` | none (see `../estimates.csv`) | MiniMax-M3.1-Flash-Preview is NOT on the leaderboard yet (checked 2026-10-13; no model page). **TODO(revisit):** `../estimates.csv` holds a placeholder estimate (cost 0.9x M3, index scaled by the KingBench ratio) — replace with measured values |
|
||||
| `airouter-qwen38`, `airouter-qwen38-max` | Qwen3.8 family | `default_flags.model: "Qwen3.8"` does not say which size; AA has Qwen3.8 27B (low/medium/xhigh), 2.4T A95B, Max (0902), Flash-Next — confirm before mapping |
|
||||
| `claude-fable-5-1` | Claude Fable 5.1 (low..max) | registry sets `reasoning_effort: low` (inert on the claude runner) |
|
||||
| `claude-sonnet-5`, `claude-opus-5`, `claude-haiku-4-5` | Sonnet 5.5, Opus 5.5, Haiku 5.5 | slug/model-id names predate the 5.5 releases (`default_flags.model` is `claude-sonnet-5` etc.); verify which API model each actually calls before using these rows |
|
||||
|
||||
Cost per task is not comparable to `cost_per_1k_tokens`: it folds in how many tokens the model spends. Use it
|
||||
for `cost_efficiency` scoring, not for the per-token price fields.
|
||||
Reference in New Issue
Block a user