Filled points are supported; hollow points are preliminary. Lines stop at unknown capabilities. Exact values and sources follow below.
Agentic2 families · 0 with independent evidence · Supported40.0
2 core families; 5 direct opponents across 3 labs.
Estimated against the same complete panel: gpt-5.6-sol claude-opus-4-8 gemini-3.1-pro . Indirect comparisons assume performance can be summarized by one strength within this capability.
Refit without one family: 36.4–44.5; 2/13 scenarios unsupported. Without one publisher: 29.9–44.7; 1/15 unsupported. Smoothing check: 37.6–43.2; 0/3 unsupported. These are sensitivity checks, not confidence intervals.
Audit: family and publisher holdouts improved on neutral predictions within evaluated coverage; source sensitivity still requires inspection. Validation audit.
Matched family win shares below describe source evidence; they are not averaged to produce the adjusted score.
- GDPval: 0.0 observed win share
Google · Source 1 - GDP.pdf: 40.0 observed win share
Google · Source 1
Hard reasoning3 families · 2 with independent evidence · Supported63.5
3 core families; 62 direct opponents across 10 labs.
Estimated against the same complete panel: gpt-5.6-sol claude-opus-4-8 gemini-3.1-pro . Indirect comparisons assume performance can be summarized by one strength within this capability.
Refit without one family: 57.4–77.9; 0/6 scenarios unsupported. Without one publisher: 52.7–73.1; 0/16 unsupported. Smoothing check: 59.2–66.1; 0/3 unsupported. These are sensitivity checks, not confidence intervals.
Audit: family and publisher holdouts improved on neutral predictions within evaluated coverage. Validation audit.
Matched family win shares below describe source evidence; they are not averaged to produce the adjusted score.
- GPQA: 70.0 observed win share
artificialanalysis.ai · Source 1 - ARC-AGI: 80.5 observed win share
arcprize.org · Source 1 - Humanity’s Last Exam: 40.0 observed win share
Google · Source 1
Coding2 families · 2 with independent evidence · Supported27.6
2 core families; 35 direct opponents across 9 labs.
Estimated against the same complete panel: gpt-5.6-sol claude-opus-4-8 gemini-3.1-pro . Indirect comparisons assume performance can be summarized by one strength within this capability.
Refit without one family: 27.7–35.6; 2/6 scenarios unsupported. Without one publisher: 24.8–33.7; 0/19 unsupported. Smoothing check: 24.6–33.8; 0/3 unsupported. These are sensitivity checks, not confidence intervals.
Audit: family and publisher holdouts improved on neutral predictions within evaluated coverage. Validation audit.
Matched family win shares below describe source evidence; they are not averaged to produce the adjusted score.
Human prefNo comparable evidenceUnknown
0 core families; 0 direct opponents across 0 labs. No admitted core comparisons
Estimated against the same complete panel: gpt-5.6-sol claude-opus-4-8 gemini-3.1-pro . Indirect comparisons assume performance can be summarized by one strength within this capability.
Refit without one family: No supported estimate; 1/1 scenarios unsupported. Without one publisher: No supported estimate; 1/1 unsupported. Smoothing check: No supported estimate; 3/3 unsupported. These are sensitivity checks, not confidence intervals.
Audit: cross-family and cross-publisher predictions could not be evaluated from the available comparison network. Validation audit.
Matched family win shares below describe source evidence; they are not averaged to produce the adjusted score.
KnowledgeNo comparable evidenceUnknown
0 core families; 0 direct opponents across 0 labs. No admitted core comparisons
Estimated against the same complete panel: gpt-5.6-sol claude-opus-4-8 gemini-3.1-pro . Indirect comparisons assume performance can be summarized by one strength within this capability.
Refit without one family: No supported estimate; 4/4 scenarios unsupported. Without one publisher: No supported estimate; 5/5 unsupported. Smoothing check: No supported estimate; 3/3 unsupported. These are sensitivity checks, not confidence intervals.
Audit: almost no publisher-held-out comparisons remained estimable. Cross-source validity is unresolved. Validation audit.
Matched family win shares below describe source evidence; they are not averaged to produce the adjusted score.
Multimodal2 families · 0 with independent evidence · Supported75.7
2 core families; 5 direct opponents across 3 labs.
Estimated against the same complete panel: gpt-5.6-sol claude-opus-4-8 gemini-3.1-pro . Indirect comparisons assume performance can be summarized by one strength within this capability.
Refit without one family: 74.0–79.5; 2/8 scenarios unsupported. Without one publisher: 51.9–82.9; 1/8 unsupported. Smoothing check: 69.5–78.9; 0/3 unsupported. These are sensitivity checks, not confidence intervals.
Audit: limited publisher-held-out coverage. Cross-source validity is unresolved. Validation audit.
Matched family win shares below describe source evidence; they are not averaged to produce the adjusted score.
- CharXiv: 40.0 observed win share
Google · Source 1 - Lvbench: 66.7 observed win share
Google · Source 1
Long contextNo comparable evidenceUnknown
0 core families; 0 direct opponents across 0 labs. No admitted core comparisons
Estimated against the same complete panel: gpt-5.6-sol claude-opus-4-8 gemini-3.1-pro . Indirect comparisons assume performance can be summarized by one strength within this capability.
Refit without one family: No supported estimate; 5/5 scenarios unsupported. Without one publisher: No supported estimate; 5/5 unsupported. Smoothing check: No supported estimate; 3/3 unsupported. These are sensitivity checks, not confidence intervals.
Audit: family and publisher holdouts underperformed neutral predictions. Treat this as an exploratory estimate. Validation audit.
Matched family win shares below describe source evidence; they are not averaged to produce the adjusted score.
Scores estimate outcomes against a shared reference panel; they are not accuracy percentages. Sparse or disconnected evidence cannot qualify an overall profile. Open a capability to inspect its evidence.
/badge/gemini-3.7-flash.svg