外观
论文arXiv 2607.145863 个月前
SoftNav：将3D场景Token注入VLM用于具身导航
原文链接：SoftNav: Injecting 3D Scene Tokens into VLMs for Embodied Navigation（arXiv 2607.14586 · 发布日期：2026-07-16）
SoftNav: Injecting 3D Scene Tokens into VLMs for Embodied Navigation
SoftNav将3D场景软token注入VLM，在HM3D-OVON、迁移基准和真实机器人上验证目标导航。
核心要点
- 当前方法以文本向VLM传递3D场景信息，存在表示鸿沟。
- SoftNav以检测物体或frontier为单位，向VLM隐藏空间注入连续3D软token。
- 冻结3D编码器和VLM，仅训练轻量投影器，约1200样本和17M参数。
- 在HM3D-OVON超越既有方法，并零样本迁移到多个基准和真机。
验证范围与局限：验证包括HM3D-OVON、GOAT-Bench、SG3D和真实机器人部署；摘要未说明更多真机细节。
英文摘要 (Abstract)
In goal-directed embodied navigation, where an agent must locate a specified target in an unseen environment, 3D scene understanding and navigation reasoning must work in concert. Current approaches transmit 3D scene information to vision-language models (VLMs) through text, suggesting a representation gap in our tested configurations; a controlled ablation confirms that direct embedding-level transfer significantly outperforms the evaluated text serialization formats. We introduce SoftNav, which injects entity-level 3D continuous representations -- one token per detected object or frontier -- into a VLM's hidden space as soft tokens through a lightweight projector. With the 3D encoder and VLM frozen, only ~1,200 samples and ~17M trainable parameters are needed. On HM3D-OVON, SoftNav achieves 74.2%/68.3%/66.7% SR across three splits, surpassing all prior methods in both SR and SPL; the same navigation policy transfers zero-shot to GOAT-Bench (67.2% SR), SG3D (47.2% s-SR), and real-world robot deployment without retraining or architectural modification. Injecting 3D scene tokens directly into VLMs bridges the representation gap, enabling transferable navigation with minimal training.
作者：Yi Wu, Junjie An, Xiao Liu, Yiqun Zhou, Yuechen Wu, Xiaoqing Guan, Shuyang Yu, You Wang, Guang Li
来源
本页事实对应的公开页面，以原文为准。