Skip to content

论文HF Daily Papers · arXiv 2611.1217713 天前

Embodied First Exam发布:47款机器人基准测试代码智能体控制能力 ​

本文由 AI 根据公开内容整理,可能有误,请以原文为准:Embodied First Exam(HF Daily Papers · arXiv 2611.12177)

核心内容 ​

为验证代码智能体(Coding Agents)替代传统视觉运动策略控制机器人的可行性,研究团队推出Embodied First Exam评测框架。该基准包含35个测试套件、1156项任务,覆盖47款异构机器人,新增空间推理、灵巧操作、多机交互及全身动态控制等9类非抓取场景,旨在评估智能体在直接控制、代码生成控制及策略训练三种路径下的综合表现。

评测设标准、实时与特权三种环境,其中标准设置模拟真实物理约束,禁止环境重置且仅提供传感器观测;特权设置开放全状态访问以测定性能上界。实测显示,在456项核心任务的标准设置中,最强模型GPT-6 Astra成功率仅为34%。研究通过对失败案例进行阶段级归因标注,量化对比了不同智能体在感知、规划与执行各环节的短板,为具身代码智能体研发提供明确改进坐标。

关键看点 ​

  • 评测规模:涵盖47款机器人与1156项任务,新增9类超越抓取的复杂操作套件
  • 控制范式:支持直接控制、代码生成控制及策略训练三种Agent原生控制路径
  • 环境约束:标准设置模拟真实世界无重置条件,特权设置提供全状态作为性能上界
  • 性能基线:GPT-6 Astra在标准设置核心任务成功率34%,暴露物理落地差距

工程与产业观察 ​

该基准填补了代码智能体在物理机器人控制领域的评测空白,34%的成功率表明当前大模型从代码生成到可靠物理执行仍存在显著鸿沟。其阶段级失败归因机制为算法工程师提供了精细化Debug工具,提示具身智能研发需重点关注Sim-to-Sim工程化验证与长时序物理约束下的代码鲁棒性,而非单纯追求语义理解能力。

英文摘要 (Abstract) ​

Early successes on agentic robotics point to a potential paradigm shift toward using coding agents to control robots. Existing robotics benchmarks are built around visuomotor policy learning, leaving it open whether coding agents can reliably control diverse robots under physical-world constraints. We therefore introduce the Embodied First Exam, a unified, comprehensive, and agent-native evaluation framework where coding agents can choose their own control approach: direct control, code to control, or train a policy to control. We hope it serves as a first exam for coding agents before they enter the physical world, with 1,156 tasks in 35 suites on 47 robots, including 9 new suites that go beyond pick-and-place to spatial puzzles, dexterous manipulation, multi-robot interaction, dynamic whole-body control, and sim-to-sim engineering. Each task can be run in up to three evaluation settings. The Standard Setting mimics the real world: the agent receives only sensor observations and, by default, cannot reset the environment. Its Realtime variant keeps the environment running while the agent reasons. The Privileged Setting gives the agent access to the full simulator state and unlimited resets, allowing it to search for an oracle solution for training purposes. Success rates in this setting serve as an approximate upper bound. On the 456 core tasks, the strongest of eight agents, GPT-6 Astra in Codex, achieves a success rate of 34% in the Standard Setting, leaving much room for improvement. To see where agents fail, we label the stage at which failed attempts go wrong and compare failures across agents, settings, and stage categories.

Hugging Face Daily Papers 热度:19 upvotes 作者:Kangrui Wang, Qineng Wang, Changhe Chen, Pingyue Zhang, Chi Gui, Feishi Wang, Canyu Chen, Wenli Xiao, Xiaoyu Huang, Tonghe Zhang, Linjie Li, Zihan Wang, Minyeong Kim, Yiyou Sun, Mingkai Deng, Jinyu Ho…

基于本页技术方向关联的整机参数、核心零部件、实测报告与同赛道选型横评。

外部来源

本页事实对应的公开页面,以原文为准。(点击将离开本站)

  1. HF Daily Papers · arXiv 2611.12177:Embodied First Exam ↗访问 2026-11-12