# OE-VLA模型支持图像视频等五类开放指令扩展具身交互

> 原文链接：https://robotics.hellobike.com/papers/2025-05-16-2505-11214-79f0774d
> 频道：papers | 更新日期：2025-05-16 | 标签：auto, arxiv, cs.RO
> 核心摘要：针对现有VLA模型仅依赖文本指令的局限，研究提出OE-VLA架构，支持图像、视频、白板文字等多模态开放指令输入。实验验证其在保持语言任务性能的同时，成功覆盖四类非文本交互场景，显著拓展具身智能应用边界。

# OE-VLA模型支持图像视频等五类开放指令扩展具身交互

::: v-pre
> 本文由 AI 根据公开内容整理，可能有误，请以原文为准：[Unveiling the Potential of Vision-Language-Action Models with Open-Ended Multimodal Instructions](https://arxiv.org/abs/2505.11214)（arXiv 2505.11214）

## 核心内容

针对当前视觉-语言-动作（VLA）模型在人机交互中过度依赖单一文本提示的瓶颈，arXiv论文2505.11214提出OE-VLA框架。该研究旨在解决用户通过展示图片、指向白板文字或演示视频等非语言方式下达指令的需求，突破了传统端到端神经网络仅接受语言输入的约束，为开放式人机协作提供了新的技术范式。

OE-VLA基于大规模互联网数据训练的视觉语言基座模型构建，实现从多模态观测到机器人动作的直接生成。实验结果表明，该模型在传统语言指令任务上性能对标主流VLA，并在图像检索、白板指令识别、视频行为模仿等四类开放任务中取得有效结果。这一架构验证了统一多模态指令空间在具身智能中的可行性，降低了自然语言描述复杂意图的认知门槛。

## 关键看点

- **多模态指令扩展**：突破纯文本限制，新增支持图像目标、白板文字、视频演示及混合模态四类开放式指令输入
- **端到端架构继承**：沿用VLA单网络范式，直接将多模态视觉观测与人类非语言提示映射为机器人底层动作序列
- **性能基准对齐**：在引入多模态能力同时，传统语言指令任务表现与现有主流VLA模型持平，无精度回退
- **交互场景泛化**：验证了非结构化环境下的意图理解能力，支撑物体指代、行为克隆等日常复杂交互任务

## 工程与产业观察

OE-VLA将具身指令空间从文本扩展至全模态，解决了复杂意图难以语言化的工程痛点。这标志着VLA研发重心正从单一语义对齐转向多模态意图统一表征，为降低真机数据采集标注成本、提升非专家用户交互效率提供了关键算法基础，加速具身智能从实验室走向开放场景。

## 英文摘要 (Abstract)

Vision-Language-Action (VLA) models have recently become highly prominent in the field of robotics. Leveraging vision-language foundation models trained on large-scale internet data, the VLA model can generate robotic actions directly from visual observations and human instructions through a single end-to-end neural network. Despite their effectiveness, current VLA models usually accept only one form of human prompting, language instructions, which may constrain their applicability in open-ended human-robot interactions. For example, a user might expect the robot to retrieve an object shown in an image, follow an instruction written on the whiteboard, or imitate a behavior demonstrated in a video, rather than relying solely on language-based descriptions. To address this gap, we introduce OE-VLA, which explores the potential of VLA models for open-ended multimodal instructions. Extensive results demonstrate that our OE-VLA not only achieves comparable performance to traditional VLA models with linguistic input but also delivers impressive results across four additional categories of open-ended tasks. The proposed methodology could significantly expand the applications of VLA models across various everyday scenarios and facilitate human-robot interaction.

作者：Wei Zhao, Gongsheng Li, Zhefei Gong, Pengxiang Ding, Han Zhao, Donglin Wang
:::

## 原始参考来源 (Sources)

- [arXiv 2505.11214：Unveiling the Potential of Vision-Language-Action Models with Open-Ended Multimodal Instructions](https://arxiv.org/abs/2505.11214)