Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

Story

Arena Launches Coding Index for AI Coding Agents

AI summary

Arena introduced its Coding Index to evaluate coding agents using real coding work. Arena reported that GPT-6.2 Astra (Max) scored 1,338 points and ranked third, while its overall Agent Arena Index score was 1,291 points at sixth place. On the Coding Index, Arena said GPT-6.2 Astra led GPT-6 Astra (Max) by 50 points with a 62% lower median cost per task.

In-depth insight

The new index adds a coding-focused benchmark based on Arena's real-world evaluation data, giving practitioners a more specific view of agent performance and cost.

Introducing Arena’s Coding Index: measuring which agents are best at real coding work.

For GPT-6.2 Astra (Max) by @OpenAI, coding is a strength. On the Coding Index specifically, it scored 1338 pts at #3!

Its overall Agent Arena Index score landed at 1291 pts at #6.

This Coding Index score of 1338 is +50 pts over its previous variant, GPT-6 Astra (Max), at 62% lower median cost per task ($4.75 vs. $12.55), earning a spot on the cost Pareto frontier for the Coding Index (see placement in thread).

Congrats to the @OpenAI team!

@arena: Scores for GPT-6.2 Astra and Sol are on the NEW Agent Arena Index! - #6 with 1291 pts: GPT-6.2 Astra (Max), now @OpenAI’s best model - #11 with 1241 pts: GPT-6.2 Sol (Max), joining the cost Pareto frontier The Agent Arena Index is our new headline measure of overall agent performance, built from millions of real-world tasks and traces from our global community. Results for GPT-6.2 Astra and Sol are based on 7.1K+ real agentic sessions. GPT-6.2 Astra: #6 with 1291 pts - +9 pts vs previous variant GPT-6 Astra (Max), currently #7 - Median cost per task: $2.94 vs. $4.91 for GPT-6 Astra, 40% lower - Strongest signal: #5 with 1462 pts in Praise vs Complaint (user satisfaction with agent's outcome) GPT-6.2 Sol: #11 with 1241 pts - Matches #12 GPT-6.1 Sol (Max) with 1241 pts, at a 13% lower median cost per task ($0.90 vs. $1.04) - Joins the Pareto frontier (see placement in thread) 6.2 Astra vs. Sol - Astra scores +50 pts above Sol at #6 vs #11 - Sol’s median cost per task is 69% lower ($0.90 vs. $2.94) Congrats to the team at @OpenAI on this release! https://x.com/arena/status/2123234410032214464

#Arena#OpenAI#GPT-6.2 Astra#GPT-6.2 Sol#Agent Arena Index

Original · 1 min readRead the original
Group chat

No comments yet. Start the conversation.