外观
论文arXiv 2610.2347912 天前
HexaShell：用于探索 VLM 智能体作为机器人编码智能体极限的轻量框架
原文链接：HexaShell: A Thin Harness for Exploring the Limits of VLM Agents as Robot Coding Agents（arXiv 2610.23479 · 发布日期：2026-10-16）
HexaShell: A Thin Harness for Exploring the Limits of VLM Agents as Robot Coding Agents
HexaShell以轻量框架让VLM编码智能体通过命令行控制双臂机器人，并在RoboDojo 42项任务上验证能力边界。
核心要点
- 研究VLM编码智能体在控制机器人而非电脑时能走多远、边界在哪里。
- 框架仅将命令转为运动，智能体看三路RGB和标定，失败后读日志并编写工具。
- 工具只用机器人观测，通过多视角三角化或射线与已知平面求交定位物体。
- 在42个RoboDojo任务上成功率39.7%、进度分48.0；已开源已演化工具。
验证范围与局限：工具在每个任务一个种子的10种布局上构建并冻结，在同种子的10种新布局及42个RoboDojo任务上测试。
英文摘要 (Abstract)
Agents built on vision language models (VLMs) now compete with learned policies on robot manipulation leaderboards. We ask how far a coding agent can go when it controls a robot instead of a computer, and where it stops. Our system, HexaShell, keeps the harness thin: it only turns the agent's commands into robot motion. The agent drives a robot with two arms through one command line. It sees three RGB cameras and knows their calibration. After each failed attempt, the agent reads the log, finds the cause, and writes or fixes small programs called tools. In this way it keeps what it learns as code, as a coding agent does when it fixes a failing test. The tools may use only what the robot observes. They use triangulation to find where objects are, relative to each other and to the gripper. If two cameras see the same point, the point lies where the two camera rays meet. If a point lies on a known plane, such as the table or the top of a block, it lies where its ray meets that plane. When the head camera is not enough, the robot moves a wrist camera to get a second view. We build the tools on only ten layouts of one seed per task, then freeze them and test them on ten new layouts of the same seed. We use Codex with GPT-6-Astra at medium reasoning effort. On all 42 RoboDojo tasks, HexaShell reaches a success rate of 39.7% and a progress score of 48.0, averaged in the same way as the leaderboard. This is a new state of the art on RoboDojo. The limit is contact, not reasoning: pushing reaches 92.5% and memory tasks 65.0%, but tight insertion only 8.3%. When the agent fails, it usually sees the failure but cannot make the motion work. We open source the tools evolved so far at https://github.com/HexaFuture/HexaShell.
作者：Hongcheng Gao, Alex Zhou, Jay Zhu, Zelin Zheng, Shijia Ge, Xiping Li, Di Wu, Yazhe Wang, Jianshu Zeng, Xuan Shangguan
来源
本页事实对应的公开页面，以原文为准。