Immanuel Giulea

We've launched our HLE-Diamond leaderboard, our independent evaluation of the refined 1,000-question version of Humanity's Last Exam, as part of v5 of the Artificial Analysis Intelligence Index HLE-Diamond was released by @CAIS and @scale_AI after a year of cleaning and refining the original Humanity's Last Exam dataset with input from the research community. It has 1,000 closed-ended questions, split evenly between reasoning and knowledge, and 272 of them include an image. We run every question once without tools, using the official HLE system prompt. GPT-5.6 Luna (medium) grades each answer against the reference answer with the official HLE judge prompt, and we report the share of questions judged correct (pass@1). Key takeaways: ➤ Only 17 model variants (~13%) score above 50% today: Gemini 4 Carbon (high) leads at 69%, followed by Claude Opus 5.5 (max) at 68% ➤ Models score higher on reasoning than on knowledge: Over 90% of the models we measured do better on the reasoning portion of the benchmark. Gemini 4 Carbon (high) leads both, at 80% on reasoning and 59% on knowledge. Claude Opus 5.5 (max) and GPT-6 Astra (max) both score 78% on reasoning, and Opus 5.5 is ahead on knowledge, 58% vs 50% ➤ Open weights models trail by more than 30 points: Kimi K3.1 Pro (weights not yet released) is the top open weights model at 37%, followed by MiniMax-M3.1-Flash at 28% and Kimi K3 (max) at 27%

Thank you to @cognition, @StevenDillmann and the @terminalbench team, @CAIS, @scale_AI, and @MiniHui_zhu, @MinyangTian1, @haopeng_uiuc, and the CritPt team for the benchmarks behind this update. Explore the full results: artificialanalysis.ai Methodology: artificialanalysis.ai/method…

Artificial Analysis Intelligence Index v5 weights: ➤ Knowledge Work (25%): AA-Briefcase (15%), AutomationBench-AA (5%), and GDP.pdf (5%) ➤ Coding (25%): FrontierCode 2.0 (15%) and Terminal-Bench 4.0 (10%) ➤ Science (25%): Terminal-Bench-Science (15%) and CritPt v2 hard subset (10%) ➤ General (25%): AA-Omniscience v1.1 15% (10% accuracy, 5% non-hallucination) and HLE-Diamond (10%)

Alongside v5, we're launching the Artificial Analysis Progress Index v1. It places language models released since November 2022 on one capability scale, across a wide span of benchmarks. Use the Progress Index to compare how underlying capability has changed over time, including between models released years apart. To compare models for current use cases and understand the present frontier, use the Intelligence Index, which remains our primary measure of current model intelligence. Thank you to @EpochAIResearch and @AnthropicAI - we built on learnings from the Epoch Capabilities Index in designing the Progress Index Explore the Progress Index: artificialanalysis.ai/trends Methodology: artificialanalysis.ai/method…

v5 tasks demand substantially more work from models. Every model we measured generates more output tokens per Intelligence Index task than in v4.3, with the median rising by two thirds, from 35k to 59k. ➤ Agentic evaluations account for most token use: Terminal-Bench-Science, Terminal-Bench 4.0, AA-Briefcase and FrontierCode 2.0 make up 55% of the Index weight but a median 88% of output tokens ➤ Terminal-Bench-Science is the most token-intensive: it accounts for about a third of output Tokens per Task on its own

Effort settings matter more in v5 than in v4.3. Claude Opus 5.5 scores 35 at low effort and 58 at max, at 10x the cost per Intelligence Index task ($0.98 vs $10.04). Claude Sonnet 5.5 spans 29 points from low to max, up from 20 in v4.3, and GPT-6.1 Sol's spread doubles from 10 to 20 points.

Cost varies widely among the leading models. Claude Opus 5.5 (max) costs $10.04 per Intelligence Index task, GPT-6 Astra (max) $6.24 and GPT-6.1 Sol (max) $1.27. GPT-6.1 Sol (max) scores within 4 points of Claude Opus 5.5 (max) at about an eighth of the cost per task.

Open weights models fall further behind in v5. The leader, Kimi K3.1 Pro (weights not yet released), scores 46, 12 points behind the leader, compared with a 3-point gap in v4.3. Of models with released weights, DeepSeek V4.1 Flash 1017 (max) and GLM-5.3 (max) score highest at 32, and Claude Haiku 5.5 (max) outscores all of them. The gap is widest on HLE-Diamond and Terminal-Bench-Science, where Kimi K3.1 Pro scores 37% and 34%, compared with 68% for Claude Opus 5.5 (max) and 63% for GPT-6 Astra (max).

The Index scores CritPt v2 on a private 40-question hard subset: the 70 research-level physics questions, minus the 30 that seven frontier models solved in all five attempts at launch. Models work in an offline Python sandbox in Stirrup, our agentic harness. GPT-6 Astra leads at 72% on high effort, followed by GPT-6.1 Sol at 69% on max effort. Claude Opus 5.5 (max), the top model in the Index overall, scores 61%. Kimi K3.1 Pro (weights not yet released) is the highest-scoring open weights model at 45%.

AA-Omniscience v1.1 is a cleaned version of our knowledge and hallucination benchmark, with saturated and public set questions removed, new questions added, and ambiguous or erroneous questions revised. Scores are mostly aligned with v1, with models scoring on average 2 points lower on the AA-Omniscience v1.1 Index (median 0.6). We’ve also released a new public subset on @huggingface, AA-Omniscience-Public-v1.1, recalibrated to ensure accuracy, hallucination rate, and overall performance are consistent between the public and private sets.

HLE-Diamond replaces Humanity's Last Exam in the Index. It is the refined 1,000-question version of Humanity's Last Exam from @CAIS and @scale_AI, split evenly between reasoning and knowledge. Claude Opus 5.5 (max) leads at 68%, followed by GPT-6 Astra (max) at 64% and Claude Fable 5.1 (max) at 60%. Knowledge questions are harder than reasoning questions for the leading models: Claude Opus 5.5 (max) and GPT-6 Astra (max) both score 78% on reasoning respectively, compared with 58% and 50% on knowledge.

Terminal-Bench-Science tests whether agents can carry out scientific research workflows in a terminal. Experts curate and review each task, and a task passes only if every one of its tests passes. GPT-6 Astra (max) leads at 63%, followed by Claude Opus 5.5 (max) at 59% and GPT-6.1 Sol (max) at 58%. Scores drop off sharply below the leaders: the best open weights model, Kimi K3.1 Pro (weights not yet released), scores 34%, and Claude Haiku 5.5 (max) scores 20%. The best open weights model with released weights, DeepSeek V4.1 Flash 1017 (max), scores 13%, and most open weights models we benchmarked score under 10%.