Evidence coverage
2666 published observations across 285 measures. Capability scoring groups compatible evidence into 43 benchmark families.
A published result contributes only when a matching reference-panel result is available. Missing evidence stays unknown. Coverage describes breadth, separately from performance.
Coverage across our labs
Models and contributing families
Raw published counts include all source types. Contributing and independent family counts follow the selected evidence policy.
451 models match.
Family coverage
| Benchmark family | Capability | Models contributing | With independent support |
|---|
| OSWorld | Agentic | 19 | 6 |
|---|
| Webarena Verified | Agentic | 4 | 0 |
|---|
| Browsecomp | Agentic | 22 | 0 |
|---|
| Toolathlon | Agentic | 15 | 0 |
|---|
| MCP-Atlas | Agentic | 19 | 0 |
|---|
| AutomationBench | Agentic | 19 | 0 |
|---|
| DeepSearchQA | Agentic | 10 | 0 |
|---|
| GDPval | Agentic | 60 | 41 |
|---|
| Bfcl V4 | Agentic | 12 | 0 |
|---|
| Apex Agents | Agentic | 10 | 0 |
|---|
| OfficeQA | Agentic | 11 | 0 |
|---|
| Humanity’s Last Exam | Hard reasoning | 31 | 8 |
|---|
| GPQA | Hard reasoning | 48 | 23 |
|---|
| ARC-AGI | Hard reasoning | 48 | 48 |
|---|
| Aime | Hard reasoning | 22 | 0 |
|---|
| FrontierMath | Hard reasoning | 10 | 0 |
|---|
| Imo Answerbench | Hard reasoning | 5 | 0 |
|---|
| Terminal-Bench Science | Hard reasoning | 5 | 0 |
|---|
| SWE-bench Verified | Coding | 38 | 19 |
|---|
| SWE-bench Pro | Coding | 20 | 4 |
|---|
| SWE-bench Multilingual | Coding | 9 | 0 |
|---|
| DeepSWE | Coding | 31 | 28 |
|---|
| LiveCodeBench | Coding | 25 | 9 |
|---|
| Terminal-Bench | Coding | 41 | 2 |
|---|
| Scicode | Coding | 7 | 0 |
|---|
| Human preference (Arena) | Human pref | 115 | 115 |
|---|
| MMLU-Pro | Knowledge | 68 | 52 |
|---|
| Gmmlu | Knowledge | 3 | 0 |
|---|
| Milu | Knowledge | 3 | 0 |
|---|
| Simpleqa Verified | Knowledge | 2 | 0 |
|---|
| Mmmu | Multimodal | 2 | 0 |
|---|
| Mmmu Pro | Multimodal | 24 | 0 |
|---|
| CharXiv | Multimodal | 16 | 0 |
|---|
| Mathvista | Multimodal | 2 | 0 |
|---|
| Chartography | Multimodal | 3 | 0 |
|---|
| Lvbench | Multimodal | 6 | 0 |
|---|
| ScreenSpot | Multimodal | 12 | 0 |
|---|
| Ocrbench V2 Average Accuracy | Multimodal | 12 | 0 |
|---|
| MRCR | Long context | 12 | 0 |
|---|
| Longbench | Long context | 8 | 0 |
|---|
| Ruler 1M | Long context | 2 | 0 |
|---|
| Graphwalks | Long context | 7 | 0 |
|---|
| Aa Lcr | Long context | 5 | 0 |
|---|
Model evidence matrix
Showing the same 226–250 models as the list above.
| Model | OSWorld | Webarena Verified | Browsecomp | Toolathlon | MCP-Atlas | AutomationBench | DeepSearchQA | GDPval | Bfcl V4 | Apex Agents | OfficeQA | Humanity’s Last Exam | GPQA | ARC-AGI | Aime | FrontierMath | Imo Answerbench | Terminal-Bench Science | SWE-bench Verified | SWE-bench Pro | SWE-bench Multilingual | DeepSWE | LiveCodeBench | Terminal-Bench | Scicode | Human preference (Arena) | MMLU-Pro | Gmmlu | Milu | Simpleqa Verified | Mmmu | Mmmu Pro | CharXiv | Mathvista | Chartography | Lvbench | ScreenSpot | Ocrbench V2 Average Accuracy | MRCR | Longbench | Ruler 1M | Graphwalks | Aa Lcr |
|---|
| DeepSeek-R1-0528-Qwen3-8B | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown |
|---|
| DeepSeek-R1-Distill-Llama-70B | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown |
|---|
| DeepSeek-R1-Distill-Llama-8B | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown |
|---|
| DeepSeek-R1-Distill-Qwen-1.5B | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown |
|---|
| DeepSeek-R1-Distill-Qwen-14B | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown |
|---|
| DeepSeek-R1-Distill-Qwen-32B | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown |
|---|
| DeepSeek-R1-Distill-Qwen-7B | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown |
|---|
| DeepSeek-V3.2-Speciale | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown |
|---|
| DeepSeek-V4-Flash-Vision-Exp | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown |
|---|
| Devstral Medium 1.0 | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown |
|---|
| Devstral Small 1.1 | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown |
|---|
| Doubao Seed Code | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown |
|---|
| ERNIE-4.5-0.3B-PT | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown |
|---|
| ERNIE-4.5-21B-A3B-PT | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown |
|---|
| ERNIE-4.5-21B-A3B-Thinking | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown |
|---|
| ERNIE-4.5-300B-A47B-PT | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown |
|---|
| ERNIE-4.5-turbo-128k | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown |
|---|
| ERNIE-4.5-turbo-20260402 | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown |
|---|
| ERNIE-4.5-turbo-32k | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown |
|---|
| ERNIE-4.5-turbo-vl | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown |
|---|
| ERNIE-4.5-turbo-vl-32k | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown |
|---|
| ERNIE-4.5-VL-28B-A3B-PT | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown |
|---|
| ERNIE-4.5-VL-28B-A3B-Thinking | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown |
|---|
| ERNIE-4.5-VL-424B-A47B-PT | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown |
|---|
| ERNIE-5.0 | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown | Unknown |
|---|
Family definitions and scoring rules · Compare capabilities · Compare published evidence