外观
论文arXiv 2307.158182023-07-28
RT-2视觉语言动作模型：文本Token化实现端到端机器人语义推理
本文由 AI 根据公开内容整理，可能有误，请以原文为准：RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control（arXiv 2307.15818）
核心内容
Google Research提出Vision-Language-Action（VLA）模型RT-2，旨在解决机器人策略泛化难题。研究团队将SOTA视觉语言模型在机器人轨迹数据与互联网级视觉问答任务上进行联合微调，使单一端到端模型同时掌握观测到动作的映射及大规模预训练知识。该方法核心在于将机器人动作直接表达为文本Token，与自然语言Token统一格式纳入训练集，从而让模型继承Web数据的语义理解优势。
在6000次评估试验中，RT-2展现出显著的涌现能力：不仅能泛化至训练集外的新物体，还能执行未见过指令（如按数字或图标放置物体）及基础语义推理（如抓取最大/最小或最近物体）。引入思维链（Chain of Thought）后，模型可完成多阶段推理，例如识别石头作为临时锤子或为疲劳者选择能量饮料。这种将动作空间离散化为语言符号的范式，为大模型驱动通用机器人控制提供了标准化技术路径。
关键看点
- VLA架构创新：将机器人动作序列化为文本Token，与自然语言统一编码，实现互联网知识与机器人控制的端到端联合微调
- 涌现语义推理：6k次测试验证模型能解析训练集外指令，并基于属性（大小、距离）执行零样本物体操作与泛化
- 思维链多步决策：集成CoT推理机制，支持跨模态常识推断，如根据“疲劳”语义自主决策抓取能量饮料等复杂任务
- 数据融合范式：打破感知与控制边界，证明在VLM训练集中混入机器人轨迹数据可直接提升物理世界操作策略性能
工程与产业观察
RT-2确立了VLA模型的技术基线，通过将动作空间对齐至语言Token空间，解决了大模型落地机器人的语义接地难题。其验证的CoT推理与零样本泛化能力，表明高质量互联网语料可作为具身智能的‘常识引擎’，为后续端到端通用机器人大模型的训练数据配比与架构设计提供了关键实证参考。
英文摘要 (Abstract)
We study how vision-language models trained on Internet-scale data can be incorporated directly into end-to-end robotic control to boost generalization and enable emergent semantic reasoning. Our goal is to enable a single end-to-end trained model to both learn to map robot observations to actions and enjoy the benefits of large-scale pretraining on language and vision-language data from the web. To this end, we propose to co-fine-tune state-of-the-art vision-language models on both robotic trajectory data and Internet-scale vision-language tasks, such as visual question answering. In contrast to other approaches, we propose a simple, general recipe to achieve this goal: in order to fit both natural language responses and robotic actions into the same format, we express the actions as text tokens and incorporate them directly into the training set of the model in the same way as natural language tokens. We refer to such category of models as vision-language-action models (VLA) and instantiate an example of such a model, which we call RT-2. Our extensive evaluation (6k evaluation trials) shows that our approach leads to performant robotic policies and enables RT-2 to obtain a range of emergent capabilities from Internet-scale training. This includes significantly improved generalization to novel objects, the ability to interpret commands not present in the robot training data (such as placing an object onto a particular number or icon), and the ability to perform rudimentary reasoning in response to user commands (such as picking up the smallest or largest object, or the one closest to another object). We further show that incorporating chain of thought reasoning allows RT-2 to perform multi-stage semantic reasoning, for example figuring out which object to pick up for use as an improvised hammer (a rock), or which type of drink is best suited for someone who is tired (an energy drink).
作者：Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choromanski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, Pete Florence, Chuyuan Fu, Montse Gonzalez Arenas…
来源
本页事实对应的公开页面，以原文为准。