Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

Story

UniPat AI's PaperBenchX benchmark: GPT-6 Astra fully reproduces only 13.98% of papers

AI summary

UniPat AI released PaperBenchX, which it calls the first multidisciplinary end-to-end benchmark for reproducing published paper results: 93 reproduction tasks built from 93 papers span 12 research directions and 10 domain-native scientific environments (including Ansys HFSS, Ansys Lumerical, Meep, PySCF and ABACUS), with 3,168 expert-verified scoring items. Across the 93 tasks, the strongest tested configuration, GPT-6 Astra, achieved only a 13.98% full-reproduction rate; averaged over tested configurations, modeling and execution scored about 62% each while validation reached just 42.8%. Per-task budgets run 4–24 hours (median 7 hours), and 12 tasks are open-sourced with 81 held out.

In-depth insight

It turns “can an agent reliably redo a published scientific result with reproducible evidence?” into a measurable, comparable question, giving a benchmark closer to real research practice.

< img id="wx_img" src="https://www.qbitai.com/wp-content/uploads/imgs/qbitai-logo-1.png" width="400" height="400">

2026-10-08 17:28:44 来源:量子位

PaperBenchX为代表的基准或许能更好地衡量AI的科研实力

允中 发自 凹非寺 量子位 | 公众号QbitAI

OpenAI最近真是频频往数学界扔重磅炸弹。

Image 1

#UniPat AI#PaperBenchX#GPT-6 Astra#OpenAI#Ansys HFSS#Ansys Lumerical#Meep#PySCF#ABACUS

Original · 12 min readRead the original
Group chat

No comments yet. Start the conversation.