EK

We’ve released AA-Omniscience v1.1, updating our knowledge and reasoning benchmark with a refreshed public set, removal of saturated questions, and ambiguity fixes AA-Omniscience is our proprietary benchmark measuring model knowledge and hallucinations, including whether models admit when they don’t know an answer. Our primary metric, Net Score, rewards correct answers, penalizes incorrect answers, and treats abstentions neutrally. AA-Omniscience was updated to v1.1 as part of the Artificial Analysis Intelligence Index v5 release. Key changes in AA-Omniscience v1.1: ➤ Updated 3,000-question dataset: Down from 6,000 questions, with previously public and saturated questions removed, errors and ambiguities corrected, and new questions added ➤ Newly calibrated 300-question public set: AA-Omniscience v1.1 has a new public set on @huggingface calibrated to match the difficulty of the private set. At release, public and private scores have a 0.99 Pearson correlation and 0.99 Spearman rank correlation for AA-Omniscience Net Score, with a median difference of 0.0 points. Accuracy and hallucination rate also have 0.99 correlations, with median differences of -0.07 and -0.29 percentage points, respectively ➤ Slightly updated domain distribution: Questions span the same 6 domains and 42 topics as version 1, with a revised distribution: Science, Engineering and Mathematics (600), Software Engineering (600), Finance (450), Health (450), Humanities and Social Sciences (450), and Law (450) Key results for AA-Omniscience v1.1: ➤ Rank order largely preserved: AA-Omniscience v1.1 has a 0.98 Spearman rank correlation with v1. The recently released Gemini 4 Carbon (high) leads with a Net Score of 54, followed by Claude Opus 5.5 (48, max with fallback) and Claude Fable 5.1 (46, max with fallback). Muse Spark Open (xhigh) is the leading open weights model, 28 points behind Gemini 4 Carbon ➤ Scores average ~2 points lower on the refreshed set: Models score 1.9 points lower on Net Score on average compared to v1, with a median decrease of 0.6 points. Some models see much larger declines, particularly MiniMax M3.1 Flash, down 22 points (16 to -6), and MiMo V2.6 Pro, down 13 points (8 to -5) ➤ Slightly lower accuracy: AA-Omniscience Accuracy measures the percentage of questions the model answers correctly. Models answer a median of 1.5 percentage points fewer questions correctly on average compared to v1. Claude Fable 5.1 (max with fallback) has the highest accuracy at 68%, followed by Claude Opus 5.5 (max with fallback) at 66% ➤ Hallucination rates decrease: AA-Hallucination measures how often models choose to answer when they do not know the answer. The median Hallucination Rate falls 1.4 percentage points, with Gemini 4 Argon (high) recording the lowest hallucination rate at 4%

We've launched our HLE-Diamond leaderboard, our independent evaluation of the refined 1,000-question version of Humanity's Last Exam, as part of v5 of the Artificial Analysis Intelligence Index HLE-Diamond was released by @CAIS and @scale_AI after a year of cleaning and refining the original Humanity's Last Exam dataset with input from the research community. It has 1,000 closed-ended questions, split evenly between reasoning and knowledge, and 272 of them include an image. We run every question once without tools, using the official HLE system prompt. GPT-5.6 Luna (medium) grades each answer against the reference answer with the official HLE judge prompt, and we report the share of questions judged correct (pass@1). Key takeaways: ➤ Only 17 model variants (~13%) score above 50% today: Gemini 4 Carbon (high) leads at 69%, followed by Claude Opus 5.5 (max) at 68% ➤ Models score higher on reasoning than on knowledge: Over 90% of the models we measured do better on the reasoning portion of the benchmark. Gemini 4 Carbon (high) leads both, at 80% on reasoning and 59% on knowledge. Claude Opus 5.5 (max) and GPT-6 Astra (max) both score 78% on reasoning, and Opus 5.5 is ahead on knowledge, 58% vs 50% ➤ Open weights models trail by more than 30 points: Kimi K3.1 Pro (weights not yet released) is the top open weights model at 37%, followed by MiniMax-M3.1-Flash at 28% and Kimi K3 (max) at 27%

We’re renaming our Max plans: Max 5x → Max Max 20x → Ultra Max is still $100/month, and Ultra is still $200. If you’re a subscriber, there’s nothing you need to do. We’re also removing the 5-hour limit for verified Ultra accounts, starting today with most existing subscribers. Newer accounts keep a temporary limit until they’re verified, which helps us prevent fraud and abuse. You'll get an email when it's removed. Weekly limits remain unchanged. View our plans here: claude.com/pricing