外观
论文arXiv 2407.077752024-07-10
Mobility VLA：长上下文VLM结合拓扑图实现多模态指令导航
本文由 AI 根据公开内容整理，可能有误，请以原文为准：Mobility VLA: Multimodal Instruction Navigation with Long-Context VLMs and Topological Graphs（arXiv 2407.07775）
核心内容
针对多模态指令导航（MINT）任务中环境先验获取难的问题，研究团队提出Mobility VLA系统。该方法将预先录制的演示视频作为环境先验输入，解决了传统导航依赖精确地图或大量在线探索的痛点。实验在836m²真实物理空间进行，验证了系统在“手持塑料箱时询问归还位置”等包含视觉状态与语言意图的复合指令下的有效性，实现了此前未解决的多模态导航任务的高成功率。
技术架构采用分层解耦设计：高层策略由长上下文VLM构成，同时处理演示Tour视频流与用户多模态指令，通过推理精准检索出对应的目标视频帧；底层策略基于离线构建的拓扑图，以目标帧为锚点生成逐时刻机器人控制动作。这种设计规避了VLM直接输出连续动作的不稳定性，将语义理解能力与拓扑导航的鲁棒性结合，为具身智能体在非结构化环境中的零样本或少样本导航提供了可复用的工程范式。
关键看点
- 分层VLA架构：高层长上下文VLM负责从演示视频中检索目标帧，底层基于拓扑图生成时序动作，解耦语义推理与运动控制
- MINT任务定义：确立以预录演示视频为环境先验的多模态指令导航范式，降低对实时建图与大规模在线数据采集的依赖
- 真实环境验证：在836m²物理空间中测试，成功执行含物体操作状态的复合语义指令，端到端导航成功率达实用水平
- 视频锚点机制：利用VLM将自然语言与图像指令映射至演示视频特定帧，作为拓扑导航的语义目标点，提升泛化能力
工程与产业观察
该工作验证了“视频即地图”在具身导航中的可行性，通过长上下文VLM与拓扑图的异构融合，有效弥合了开放词汇语义理解与闭环运动控制间的鸿沟。对于缺乏高精度先验地图的场景，利用低成本演示视频构建导航策略，为服务机器人在动态非结构化环境中的快速部署提供了低数据依赖的技术路径。
英文摘要 (Abstract)
An elusive goal in navigation research is to build an intelligent agent that can understand multimodal instructions including natural language and image, and perform useful navigation. To achieve this, we study a widely useful category of navigation tasks we call Multimodal Instruction Navigation with demonstration Tours (MINT), in which the environment prior is provided through a previously recorded demonstration video. Recent advances in Vision Language Models (VLMs) have shown a promising path in achieving this goal as it demonstrates capabilities in perceiving and reasoning about multimodal inputs. However, VLMs are typically trained to predict textual output and it is an open research question about how to best utilize them in navigation. To solve MINT, we present Mobility VLA, a hierarchical Vision-Language-Action (VLA) navigation policy that combines the environment understanding and common sense reasoning power of long-context VLMs and a robust low-level navigation policy based on topological graphs. The high-level policy consists of a long-context VLM that takes the demonstration tour video and the multimodal user instruction as input to find the goal frame in the tour video. Next, a low-level policy uses the goal frame and an offline constructed topological graph to generate robot actions at every timestep. We evaluated Mobility VLA in a 836m^2 real world environment and show that Mobility VLA has a high end-to-end success rates on previously unsolved multimodal instructions such as "Where should I return this?" while holding a plastic bin. A video demonstrating Mobility VLA can be found here: https://youtu.be/-Tof__Q8_5s
作者：Hao-Tien Lewis Chiang, Zhuo Xu, Zipeng Fu, Mithun George Jacob, Tingnan Zhang, Tsang-Wei Edward Lee, Wenhao Yu, Connor Schenck, David Rendleman, Dhruv Shah, Fei Xia, Jasmine Hsu, Jonathan Hoech, Pete…
来源
本页事实对应的公开页面，以原文为准。