Nature: AI agent scores just 2/6 and 1/6 on 'shadow evaluation' research test
An arXiv preprint led by Princeton's Sayash Kapoor proposes "shadow evaluation": an AI system was pointed at the research questions of two papers submitted to NeurIPS, told to run its own project and write a paper, and then graded by the original authors. The agent harnessed Claude Opus 4.8 inside a modified OpenClaw scaffold with six days and $3,000 of compute credits per paper; it ran hundreds of experiments without stalling, produced plausible literature reviews and caught its own hallucinations without reward hacking, yet earned only 2/6 and 1/6 from the authors. The authors cite premature lock-in on a hypothesis, insufficient backtracking, overly lenient self-review and messy write-ups; Kapoor says full automation of open-ended research is far off, while AI Scientist co-creator Cong Lu called it an interesting data point but said the guardrails were too weak.
The study proposes author-graded shadow evaluation as a stronger check than peer review on AI-generated research, and its low scores push back on claims that AI can fully automate open-ended science.

一个智能体系统成功从两篇计算机论文中提取出了研究概念——但论文原作者不以为然。AI在研究自己方面已经取得重大突破,它们能让现有算法更智能,或是直接写出新的算法。但计算机还没法取代它的制造者,arXiv预印本服务器上发布的一项研究[1]指出。“我认为,开放式研究的全自动化还早着呢,”该预印本共同作者、美国普林斯......
本文来自微信公众号: 自然系列 ,作者:Nature Portfolio
一个智能体系统成功从两篇计算机论文中提取出了研究概念——但论文原作者不以为然。
AI在研究自己方面已经取得重大突破,它们能让现有算法更智能,或是直接写出新的算法。但计算机还没法取代它的制造者,arXiv预印本服务器上发布的一项研究[1]指出。
#Sakana AI#AI Scientist#Princeton University#Claude Opus 4.8#OpenClaw#NeurIPS#Recursive Superintelligence#arXiv