RankingQwen3.5-397B-A17B

Qwen3.5-397B-A17B

Data updated 22 Oct 2026

43 published benchmark measures · 14 benchmark families contribute across 6 task areas. 3 capability estimates available in the full profile. See all results ↓ · Compare published benchmarks →

Current configurationSources

Compare with

Choose a second model and its effort to see both profiles and benchmark differences below.

Compare withModel and configuration

Choose a configuration to compare. The overview below combines evidence across settings and has no model-wide rank.

Model evidence summary

This profile combines published settings. It is not a runnable configuration or a leaderboard rank. Compare measured configurations →

Performance profile

Capabilities

Adjusted comparison score · 0–100

Filled points are supported; hollow points are preliminary. Lines stop at unknown capabilities. Exact values and sources follow below.

Agentic2 families · 0 with independent evidence · Supported1.6

2 core families; 7 direct opponents across 6 labs.

Estimated against the same complete panel: gpt-5.6-sol claude-opus-4-8 gemini-3.1-pro . Indirect comparisons assume performance can be summarized by one strength within this capability.

Refit without one family: 1.3–2.1; 2/13 scenarios unsupported. Without one publisher: 0.6–50.8; 1/15 unsupported. Smoothing check: 0.5–6.1; 0/3 unsupported. These are sensitivity checks, not confidence intervals.

Audit: family and publisher holdouts improved on neutral predictions within evaluated coverage; source sensitivity still requires inspection. Validation audit.

Matched family win shares below describe source evidence; they are not averaged to produce the adjusted score.

Browsecomp: 37.5 observed win share
Mistral, NVIDIA · Source 1 Source 2
GDPval: 0.0 observed win share
NVIDIA · Source 1

Hard reasoning4 families · 0 with independent evidence · Supported3.9

4 core families; 8 direct opponents across 6 labs.

Estimated against the same complete panel: gpt-5.6-sol claude-opus-4-8 gemini-3.1-pro . Indirect comparisons assume performance can be summarized by one strength within this capability.

Refit without one family: 2.8–6.6; 0/6 scenarios unsupported. Without one publisher: 1.1–31.5; 1/16 unsupported. Smoothing check: 1.0–13.2; 0/3 unsupported. These are sensitivity checks, not confidence intervals.

Audit: family and publisher holdouts improved on neutral predictions within evaluated coverage. Validation audit.

Matched family win shares below describe source evidence; they are not averaged to produce the adjusted score.

Aime: 0.0 observed win share
Mistral · Source 1
GPQA: 75.0 observed win share
NVIDIA · Source 1
Humanity’s Last Exam: 54.2 observed win share
NVIDIA · Source 1 Source 2
Imo Answerbench: 25.0 observed win share
NVIDIA · Source 1 Source 2

Coding3 families · 0 with independent evidence · Supported1.6

3 core families; 7 direct opponents across 6 labs.

Estimated against the same complete panel: gpt-5.6-sol claude-opus-4-8 gemini-3.1-pro . Indirect comparisons assume performance can be summarized by one strength within this capability.

Refit without one family: 0.5–9.3; 0/6 scenarios unsupported. Without one publisher: 1.0–7.8; 1/19 unsupported. Smoothing check: 0.7–5.2; 0/3 unsupported. These are sensitivity checks, not confidence intervals.

Audit: family and publisher holdouts improved on neutral predictions within evaluated coverage. Validation audit.

Matched family win shares below describe source evidence; they are not averaged to produce the adjusted score.

SWE-bench Verified: 12.5 observed win share
Mistral, NVIDIA · Source 1 Source 2
LiveCodeBench: 25.0 observed win share
NVIDIA · Source 1
SWE-bench Multilingual: 25.0 observed win share
NVIDIA · Source 1

Human pref1 families · 1 with independent evidence · PreliminaryUnknown

1 core families; 76 direct opponents across 17 labs. No connected comparison to the complete reference panel

Estimated against the same complete panel: gpt-5.6-sol claude-opus-4-8 gemini-3.1-pro . Indirect comparisons assume performance can be summarized by one strength within this capability.

Refit without one family: No supported estimate; 1/1 scenarios unsupported. Without one publisher: No supported estimate; 1/1 unsupported. Smoothing check: No supported estimate; 3/3 unsupported. These are sensitivity checks, not confidence intervals.

Audit: cross-family and cross-publisher predictions could not be evaluated from the available comparison network. Validation audit.

Matched family win shares below describe source evidence; they are not averaged to produce the adjusted score.

Human preference (Arena): 67.1 observed win share
lmarena.ai · Source 1

Knowledge1 families · 1 with independent evidence · PreliminaryUnknown

1 core families; 55 direct opponents across 14 labs. No connected comparison to the complete reference panel

Estimated against the same complete panel: gpt-5.6-sol claude-opus-4-8 gemini-3.1-pro . Indirect comparisons assume performance can be summarized by one strength within this capability.

Refit without one family: No supported estimate; 4/4 scenarios unsupported. Without one publisher: No supported estimate; 5/5 unsupported. Smoothing check: No supported estimate; 3/3 unsupported. These are sensitivity checks, not confidence intervals.

Audit: almost no publisher-held-out comparisons remained estimable. Cross-source validity is unresolved. Validation audit.

Matched family win shares below describe source evidence; they are not averaged to produce the adjusted score.

MMLU-Pro: 98.0 observed win share
NVIDIA, huggingface.co · Source 1 Source 2 Source 3

MultimodalNo comparable evidenceUnknown

0 core families; 0 direct opponents across 0 labs. No admitted core comparisons

Estimated against the same complete panel: gpt-5.6-sol claude-opus-4-8 gemini-3.1-pro . Indirect comparisons assume performance can be summarized by one strength within this capability.

Refit without one family: No supported estimate; 8/8 scenarios unsupported. Without one publisher: No supported estimate; 8/8 unsupported. Smoothing check: No supported estimate; 3/3 unsupported. These are sensitivity checks, not confidence intervals.

Audit: limited publisher-held-out coverage. Cross-source validity is unresolved. Validation audit.

Matched family win shares below describe source evidence; they are not averaged to produce the adjusted score.

Long context3 families · 0 with independent evidence · PreliminaryUnknown

3 core families; 4 direct opponents across 4 labs. No connected comparison to the complete reference panel

Estimated against the same complete panel: gpt-5.6-sol claude-opus-4-8 gemini-3.1-pro . Indirect comparisons assume performance can be summarized by one strength within this capability.

Refit without one family: No supported estimate; 5/5 scenarios unsupported. Without one publisher: No supported estimate; 5/5 unsupported. Smoothing check: No supported estimate; 3/3 unsupported. These are sensitivity checks, not confidence intervals.

Audit: family and publisher holdouts underperformed neutral predictions. Treat this as an exploratory estimate. Validation audit.

Matched family win shares below describe source evidence; they are not averaged to produce the adjusted score.

Aa Lcr: 50.0 observed win share
NVIDIA · Source 1
Longbench: 100.0 observed win share
NVIDIA · Source 1
Ruler 1M: 0.0 observed win share
NVIDIA · Source 1

Scores estimate outcomes against a shared reference panel; they are not accuracy percentages. Sparse or disconnected evidence cannot qualify an overall profile. Open a capability to inspect its evidence.

Reported effort · Not specified

Settings reported in this model's published benchmark results, including results outside the aggregate. Effort names are provider-specific. These are not API defaults or equal compute budgets.

Not specified: 47 observations

Mixed settings means multiple settings occur in the evidence. Best across efforts means the source selected its best reported result across settings; it does not mean Max. Unspecified settings stay unknown. This model-summary chart combines reported settings. The leaderboard keeps identified configurations separate and excludes unknown effort.

Inspect each result and its source ↓ · Download effort evidence

Score contributions and missing evidence

14 contributing families across 6 capabilities. Fixed reference panels do not change when the catalog expands.

Capability is fitted jointly across families. Capability estimates below describe different task areas; their weighted sum is not the Capability score.

Results without reviewed compatibility or a reference match remain in the raw evidence below. Coverage counts only contributing results.

Model information & shareable badge

Lab: Qwen
Catalog status: open-weights-available
Availability: Open weights available for self-hosting
Family: Qwen3.5
Released: —
Context: —
License: apache-2.0
Model card: https://huggingface.co/Qwen/Qwen3.5-397B-A17B
Default Capability family coverage: /badge/qwen3.5-397b-a17b.svg

Benchmark scores & sources

Original results, evaluation harnesses, and evidence behind this model.

Benchmark	Bucket	Score	Harness	Evidence	Source-recorded date
AA-LCR · source release snapshot; version not specified	Long context	68.3% Reported settings & source NVIDIA evaluation harness/settings per benchmark in model card; comparator scores are NVIDIA-reported under that protocol. First-party reported result. Original cell: 68.3 nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16 · Performance table: AA-LCR · reviewed 2026-09-06	Published configuration Effort: Not specified contributes to capability	lab self-report	Reviewed 2026-09-06
AIME 2025 avg@16 · source release snapshot; version not specified	Hard reasoning	83.1% Reported settings & source Maximum reasoning; Sonnet4.6 external API truncation caveat. First-party reported result; comparator results retain the source evaluation setup. Mistral Medium 3.5 model card performance charts · images/image1.png · reviewed 2026-09-06	Published configuration Effort: Not specified contributes to capability	lab self-report	Reviewed 2026-09-06
AllenAI IFBench · source release snapshot; version not specified	Supporting evidence	76.5% Reported settings & source Maximum reasoning; Sonnet4.6 external API truncation caveat. First-party reported result; comparator results retain the source evaluation setup. Mistral Medium 3.5 model card performance charts · images/image1.png · reviewed 2026-09-06	Published configuration Effort: Not specified	lab self-report Supporting evidence outside the reviewed capability core	Reviewed 2026-09-06
Apex-Shortlist (no tools) · source release snapshot; version not specified	Supporting evidence	61.4% Reported settings & source NVIDIA evaluation harness/settings per benchmark in model card; comparator scores are NVIDIA-reported under that protocol. First-party reported result. Original cell: 61.4 nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16 · Performance table: Apex-Shortlist (no tools) · reviewed 2026-09-06	Published configuration Effort: Not specified	lab self-report Supporting evidence outside the reviewed capability core	Reviewed 2026-09-06
Apex-Shortlist (with tools) · source release snapshot; version not specified	Supporting evidence	60.4% Reported settings & source NVIDIA evaluation harness/settings per benchmark in model card; comparator scores are NVIDIA-reported under that protocol. First-party reported result. Original cell: 60.4 nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16 · Performance table: Apex-Shortlist (with tools) · reviewed 2026-09-06	Published configuration Effort: Not specified	lab self-report Supporting evidence outside the reviewed capability core	Reviewed 2026-09-06
BeyondAIME avg@16 · source release snapshot; version not specified	Supporting evidence	72.3% Reported settings & source Maximum reasoning; Sonnet4.6 external API truncation caveat. First-party reported result; comparator results retain the source evaluation setup. Mistral Medium 3.5 model card performance charts · images/image1.png · reviewed 2026-09-06	Published configuration Effort: Not specified	lab self-report Supporting evidence outside the reviewed capability core	Reviewed 2026-09-06
BrowseComp · source release snapshot; version not specified	Agentic	78.6% Reported settings & source Maximum reasoning; tau3 4 trials, GPT5.2 low simulator except Sierra reported comparisons; BrowseComp discard-all context at100k. First-party reported result; comparator results retain the source evaluation setup. Mistral Medium 3.5 model card performance charts · images/image4.png · reviewed 2026-09-06	Published configuration Effort: Not specified contributes to capability	lab self-report	Reviewed 2026-09-06
BrowseComp · source release snapshot; version not specified	Agentic	40.5% Reported settings & source NVIDIA evaluation harness/settings per benchmark in model card; comparator scores are NVIDIA-reported under that protocol. First-party reported result. Original cell: 40.5 nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16 · Performance table: BrowseComp · reviewed 2026-09-06	Published configuration Effort: Not specified contributes to capability	lab self-report	Reviewed 2026-09-06
Collie · source release snapshot; version not specified	Supporting evidence	88.9% Reported settings & source Maximum reasoning; Sonnet4.6 external API truncation caveat. First-party reported result; comparator results retain the source evaluation setup. Mistral Medium 3.5 model card performance charts · images/image1.png · reviewed 2026-09-06	Published configuration Effort: Not specified	lab self-report Supporting evidence outside the reviewed capability core	Reviewed 2026-09-06
CritPt (no tools) · source release snapshot; version not specified	Supporting evidence	2.4% Reported settings & source NVIDIA evaluation harness/settings per benchmark in model card; comparator scores are NVIDIA-reported under that protocol. First-party reported result. Original cell: 2.4 nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16 · Performance table: CritPt (no tools) · reviewed 2026-09-06	Published configuration Effort: Not specified	lab self-report Supporting evidence outside the reviewed capability core	Reviewed 2026-09-06
GDPVal · source release snapshot; version not specified	Agentic	34.6% Reported settings & source NVIDIA evaluation harness/settings per benchmark in model card; comparator scores are NVIDIA-reported under that protocol. First-party reported result. Original cell: 34.6 nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16 · Performance table: GDPVal · reviewed 2026-09-06	Published configuration Effort: Not specified contributes to capability	lab self-report	Reviewed 2026-09-06
GPQA (no tools) · source release snapshot; version not specified	Hard reasoning	87.1% Reported settings & source NVIDIA evaluation harness/settings per benchmark in model card; comparator scores are NVIDIA-reported under that protocol. First-party reported result. Original cell: 87.1 nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16 · Performance table: GPQA (no tools) · reviewed 2026-09-06	Published configuration Effort: Not specified contributes to capability	lab self-report	Reviewed 2026-09-06
HLE (no tools) · source release snapshot; version not specified	Hard reasoning	28.5% Reported settings & source NVIDIA evaluation harness/settings per benchmark in model card; comparator scores are NVIDIA-reported under that protocol. First-party reported result. Original cell: 28.5 nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16 · Performance table: HLE (no tools) · reviewed 2026-09-06	Published configuration Effort: Not specified contributes to capability	lab self-report	Reviewed 2026-09-06
HLE (with tools) · source release snapshot; version not specified	Hard reasoning	48.3% Reported settings & source NVIDIA evaluation harness/settings per benchmark in model card; comparator scores are NVIDIA-reported under that protocol. First-party reported result. Original cell: 48.3 nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16 · Performance table: HLE (with tools) · reviewed 2026-09-06	Published configuration Effort: Not specified contributes to capability	lab self-report	Reviewed 2026-09-06
IFBench (prompt loose) · source release snapshot; version not specified	Supporting evidence	78.2% Reported settings & source NVIDIA evaluation harness/settings per benchmark in model card; comparator scores are NVIDIA-reported under that protocol. First-party reported result. Original cell: 78.2 nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16 · Performance table: IFBench (prompt loose) · reviewed 2026-09-06	Published configuration Effort: Not specified	lab self-report Supporting evidence outside the reviewed capability core	Reviewed 2026-09-06
IMOAnswerBench (no tools) · source release snapshot; version not specified	Hard reasoning	83.1% Reported settings & source NVIDIA evaluation harness/settings per benchmark in model card; comparator scores are NVIDIA-reported under that protocol. First-party reported result. Original cell: 83.1 nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16 · Performance table: IMOAnswerBench (no tools) · reviewed 2026-09-06	Published configuration Effort: Not specified contributes to capability	lab self-report	Reviewed 2026-09-06
IMOAnswerBench (with tools) · source release snapshot; version not specified	Hard reasoning	84.51% Reported settings & source NVIDIA evaluation harness/settings per benchmark in model card; comparator scores are NVIDIA-reported under that protocol. First-party reported result. Original cell: 84.51 nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16 · Performance table: IMOAnswerBench (with tools) · reviewed 2026-09-06	Published configuration Effort: Not specified contributes to capability	lab self-report	Reviewed 2026-09-06
IOI 2025 · source release snapshot; version not specified	Supporting evidence	441.3 points Reported settings & source NVIDIA evaluation harness/settings per benchmark in model card; comparator scores are NVIDIA-reported under that protocol. First-party reported result. Original cell: 441.3 nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16 · Performance table: IOI 2025 · reviewed 2026-09-06	Published configuration Effort: Not specified	lab self-report Supporting evidence outside the reviewed capability core	Reviewed 2026-09-06
LiveCodeBench (v6) · source release snapshot; version not specified	Coding	79.3% Reported settings & source NVIDIA evaluation harness/settings per benchmark in model card; comparator scores are NVIDIA-reported under that protocol. First-party reported result. Original cell: 79.3 nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16 · Performance table: LiveCodeBench (v6) · reviewed 2026-09-06	Published configuration Effort: Not specified contributes to capability	lab self-report	Reviewed 2026-09-06
LMArena Text Arena	Human pref	1441	LMArena Text Effort: Not specified contributes to capability	official board	2026-10-20
Longbench v2 (≤ 1M) · source release snapshot; version not specified	Long context	68.9% Reported settings & source NVIDIA evaluation harness/settings per benchmark in model card; comparator scores are NVIDIA-reported under that protocol. First-party reported result. Original cell: 68.9 nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16 · Performance table: Longbench v2 (≤ 1M) · reviewed 2026-09-06	Published configuration Effort: Not specified contributes to capability	lab self-report	Reviewed 2026-09-06
MMLU-Pro	Knowledge	87.8%	MMLU-Pro reported Effort: Not specified contributes to capability	official board	2026-10-13
MMLU-Pro · source release snapshot; version not specified	Knowledge	88.3% Reported settings & source NVIDIA evaluation harness/settings per benchmark in model card; comparator scores are NVIDIA-reported under that protocol. First-party reported result. Original cell: 88.3 nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16 · Performance table: MMLU-Pro · reviewed 2026-09-06	Published configuration Effort: Not specified contributes to capability	lab self-report	Reviewed 2026-09-06
MMLU-ProX (avg en/de/fr/es/it/ja/zh/hi/pt/ko) · source release snapshot; version not specified	Knowledge	86.4% Reported settings & source NVIDIA evaluation harness/settings per benchmark in model card; comparator scores are NVIDIA-reported under that protocol. First-party reported result. Original cell: 86.4 nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16 · Performance table: MMLU-ProX (avg en/de/fr/es/it/ja/zh/hi/pt/ko) · reviewed 2026-09-06	Published configuration Effort: Not specified contributes to capability	lab self-report	Reviewed 2026-09-06
Multi-Challenge · source release snapshot; version not specified	Supporting evidence	63.9% Reported settings & source NVIDIA evaluation harness/settings per benchmark in model card; comparator scores are NVIDIA-reported under that protocol. First-party reported result. Original cell: 63.9 nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16 · Performance table: Multi-Challenge · reviewed 2026-09-06	Published configuration Effort: Not specified	lab self-report Supporting evidence outside the reviewed capability core	Reviewed 2026-09-06