Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

Story

Embodied First Exam benchmarks coding agents on robot tasks

AI summary

Researchers introduced Embodied First Exam (EFE), a unified, agent-native benchmark for evaluating LLM-driven coding agents on robotics. It contains 35 task suites with 1,156 tasks across 47 robots, including spatial puzzles, dexterous manipulation, multi-robot interaction and dynamic whole-body control; agents may output actions, write control programs or train policies. In the reported evaluation of eight agents, GPT-6 Astra in Codex achieved a 33.6% overall success rate under the standard setting, while unlimited environment resets proved more helpful than exact object-state information in closing the gap to privileged evaluation.

In-depth insight

EFE provides a common framework for comparing coding agents across robot tasks, control methods, observation conditions and failure stages, while highlighting how reset access and physical constraints affect performance.

让机器人掌握一项新技能,通常有两条路径:人类为它编写程序,或者让它从数据中学习。如今,让由 LLM 驱动的 coding agent 直接控制机器人,成为了第三种可能。然而,coding agent 能否在物理世界的约束下可靠地控制多种机器人?现有基准尚未回答这个问题。

为此,华人科学家李曼玲、Ruohan Zhang、石冠亚联合团队及其合作者(包括李飞飞、Dawn Song、Pieter Abbeel 等)提出了“具身的第一场考试”(Embodied First Exam,EFE),一个统一、全面且 agent 原生的评测框架,旨在考察通用 coding agent 能解决哪些机器人任务、物理世界约束如何影响表现,以及失败发生在哪一步。

论文链接:https://arxiv.org/abs/2611.12177

据论文介绍,EFE 包含 35 个任务套件中的 1156 项任务,涉及 47 款机器人。其中 26 个套件改编自已有机器人基准,9 个新增套件将评测从抓取和放置拓展至空间谜题、灵巧操作、多机器人交互、动态全身控制,以及不同仿真环境之间的工程任务。

#Embodied First Exam#GPT-6 Astra#Codex

Original · 5 min readRead the original
Group chat

No comments yet. Start the conversation.