外观
论文arXiv 2609.3718729 天前
InsightMap：用于具身多模态推理的结构化空间建模
InsightMap以俯视地图作显式空间记忆和动作条件预测目标，在导航、问答、推理、3D定位及Unitree Go2上验证。
宇树 Go2原文链接：InsightMap: Structured Spatial Modeling for Embodied Multimodal Reasoning（arXiv 2609.37187 · 发布日期：2026-09-29）
InsightMap: Structured Spatial Modeling for Embodied Multimodal Reasoning
InsightMap以俯视地图作显式空间记忆和动作条件预测目标，在导航、问答、推理、3D定位及Unitree Go2上验证。
核心要点
- 语言引导导航需将局部观测连接到持久空间参照，并学习动作如何改变表示。
- 框架把历史视角关联到带标签地图位置，共享多模态骨干联合学习动作预测与后动作地图生成。
- R2R-CE/RxR-CE验证未见划分SR为56.9%/54.9%，地图监督提升R2R-CE表现。
- 静态空间任务报告ScanQA、SQA3D和ScanRefer指标；Unitree Go2上优于NaVid和NaVILA。
验证范围与局限：验证包括R2R-CE与RxR-CE验证未见划分、静态空间任务，以及Unitree Go2在走廊、实验室和办公室环境中的测试。
英文摘要 (Abstract)
Language-guided navigation requires connecting partial observations to a persistent spatial reference and learning how actions change that representation. We introduce InsightMap, a framework that uses top-down maps as both explicit spatial memory and action-conditioned prediction targets. Historical views are linked to labeled map locations, and a shared multimodal backbone jointly learns navigation action prediction and post-action map generation. Map prediction provides auxiliary training supervision, while navigation inference decodes actions from the observed spatial context. An aligned RGB-D data pipeline supports a common interface for navigation, visual question answering, situated reasoning, and 3D grounding. On the validation-unseen splits of R2R-CE and RxR-CE, InsightMap achieves success rates (SR) of 56.9% and 54.9%, respectively. Adding map-prediction supervision improves R2R-CE SR by 4.3 and success weighted by path length (SPL) by 3.2 percentage points. On static spatial tasks, InsightMap achieves 103.7 CIDEr on ScanQA, 60.1% exact-match accuracy on SQA3D, and 53.1% grounding accuracy at 0.5 IoU on ScanRefer with detected object proposals. On Unitree Go2, it outperforms NaVid and NaVILA in hallway, lab, and office environments.
作者：Hongpei Zheng, Hujun Yin
来源
本页事实对应的公开页面，以原文为准。