Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

Story

Artificial Analysis overhauls Intelligence Index with private evaluations

AI summary

Artificial Analysis launched Intelligence Index v5, with 60% of its evaluations now based on private test sets or held-out answers. The update adds FrontierCode 2.0 and Terminal-Bench-Science, upgrades three existing benchmarks, and introduces Progress Index v1 to track model capability over time; Claude Opus 5.5 leads the initial v5 ranking with 58 points.

In-depth insight

The overhaul shifts leading model rankings toward more realistic coding and scientific tasks while making a majority of the composite score impossible to reproduce independently.

RT Artificial Analysis Announcing Artificial Analysis Intelligence Index v5, featuring 2 new benchmarks and 3 upgrades. For the first time, the majority of the Index is built on private datasets The Artificial Analysis Intelligence Index is our composite measure of model capabilities across tasks reflecting real-world usage today. v5 is our biggest upgrade since v4 launched in January. With v5, 60% of the Index is now based on evaluations with private test sets or held-out answers. Alongside Intelligence Index v5, we’re launching the Artificial Analysis Progress Index v1, which tracks model capability over time on one continuous scale. New to the Intelligence Index: ➤ FrontierCode 2.0: Cognition's newly-launched private benchmark of real-world coding tasks, graded on whether a repository maintainer would merge the agent's change ➤ Terminal-Bench-Science: 70 agentic scientific research tasks, run in a terminal environment Upgraded: ➤ HLE-Diamond (replacing Humanity's Last Exam): a refined 1,000-question version of Humanity's Last Exam, split evenly between reasoning and knowledge questions ➤ AA-Omniscience v1.1: An updated dataset with saturated and public set questions removed, new questions added, and ambiguous or erroneous questions revised ➤ CritPt v2: a private, 40-question hard subset of CritPt v2, revised by the CritPt authors and run with a code-execution tool Staying in the Index: AA-Briefcase, AutomationBench-AA, GDP.pdf and Terminal-Bench 4.0 Removed from the Index: GDPval-AA v2.1, SciCode, and AA-LCR v1.1 Key results: ➤ Claude Opus 5.5 (max) leads at 58, ahead of GPT-6 Astra (max, 57) and GPT-6.1 Sol (max, 55): Anthropic and OpenAI hold the top five models.

#Artificial Analysis#Artificial Analysis Intelligence Index v5#Artificial Analysis Progress Index v1#FrontierCode 2.0#Terminal-Bench-Science#Cognition#Claude Opus 5.5#GPT-6 Astra#GPT-6.1 Sol#Kimi K3.1 Pro#DeepSeek V4.1 Flash 1017#GLM-5.3

Original · 2 min readRead the original
Group chat

No comments yet. Start the conversation.