Embodied First Exam benchmarks coding agents on robot tasks
Researchers introduced Embodied First Exam (EFE), a unified, agent-native benchmark for evaluating LLM-driven coding agents on robotics. It contains 35 task suites with 1,156 tasks across 47 robots, including spatial puzzles, dexterous manipulation, multi-robot interaction and dynamic whole-body control; agents may output actions, write control programs or train policies. In the reported evaluation of eight agents, GPT-6 Astra in Codex achieved a 33.6% overall success rate under the standard setting, while unlimited environment resets proved more helpful than exact object-state information in closing the gap to privileged evaluation.
EFE provides a common framework for comparing coding agents across robot tasks, control methods, observation conditions and failure stages, while highlighting how reset access and physical constraints affect performance.
让机器人掌握一项新技能,通常有两条路径:人类为它编写程序,或者让它从数据中学习。如今,让由 LLM 驱动的 coding agent 直接控制机器人,成为了第三种可能。然而,coding agent 能否在物理世界的约束下可靠地控制多种机器人?现有基准尚未回答这个问题。
为此,华人科学家李曼玲、Ruohan Zhang、石冠亚联合团队及其合作者(包括李飞飞、Dawn Song、Pieter Abbeel 等)提出了“具身的第一场考试”(Embodied First Exam,EFE),一个统一、全面且 agent 原生的评测框架,旨在考察通用 coding agent 能解决哪些机器人任务、物理世界约束如何影响表现,以及失败发生在哪一步。
论文链接:https://arxiv.org/abs/2611.12177
据论文介绍,EFE 包含 35 个任务套件中的 1156 项任务,涉及 47 款机器人。其中 26 个套件改编自已有机器人基准,9 个新增套件将评测从抓取和放置拓展至空间谜题、灵巧操作、多机器人交互、动态全身控制,以及不同仿真环境之间的工程任务。