Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

Story

Artificial Analysis releases Decision Index v2

AI summary

Artificial Analysis released Decision Index v2, whose new Decision Score combines answer correctness with confidence calibration. Private datasets now make up 45% of the index, up from 25% in v1. The update adds the 1,500-document private AA-Paperclip benchmark and the AA-Symmetry consistency test, adds an implementation of the ONET benchmark, removes WildGuardTest because of high saturation, and reduces When2Call’s weighting from 25% to 10%. Claude Haiku 5.5 (Max) and GPT-6 Luna (Max) scored 77 and 76, respectively, while Drex Max v1.1 and GLiDE v1.2 tied at 73 among specialist decision models. GPT-6 Luna cost $1,530 per million tasks and took 11.34 seconds per task, compared with $102 and 275 milliseconds for Drex Max v1.1. GLiDE v1.2 cost $780 per million tasks, nearly eight times Drex Max v1.1’s cost, because it bills the document input for every question.

Introducing the Artificial Analysis Decision Index v2: a new proprietary evaluation, updated benchmarks, and a headline metric that rewards model calibration

We’ve released the second version of the Artificial Analysis Decision Index to better reflect real-world utility and guard against benchmark gaming. 45% of the Index is now built on private datasets, up from 25% in v1. Drex Max v1.1 from @NaceAI narrowly leads among decision models at 73, within 1 point of GLiDE v1.2 (@fastinoAI) and Mammoth 27B (@baselabs)

Updates in Decision Index v2:

➤ Decision Score: We adopt a new headline metric that rewards a model for both correctness and calibrated confidence. Each question scores from 0 to 100: right answers score above 50, wrong answers below 50, and confidence in its top answer moves the score within each half

➤ AA-Paperclip (30%): We added a new private benchmark of 1,500 documents including resumes, receipts, invoices, and AI and machine learning research papers. Models make decisions and extract facts from each document and are scored jointly as a single decision, correct only if all 15 answers are right

#Artificial Analysis#Artificial Analysis Decision Index v2#AA-Paperclip#AA-Symmetry#Claude Haiku 5.5#GPT-6 Luna#Drex Max v1.1#GLiDE v1.2#Mammoth 27B#Nace AI#Fastino Labs#Baseten

Original · 2 min readRead the original
Group chat

No comments yet. Start the conversation.