外观
论文arXiv 2608.162342 个月前
GaussianDWM++：用于统一场景理解、编辑和多模态生成的语言锚定3D Gaussian驾驶世界模型
原文链接：GaussianDWM++: Language-Grounded 3D Gaussian Driving World Model for Unified Scene Understanding, Editing, and Multi-Modal Generation（arXiv 2608.16234 · 发布日期：2026-08-17）
GaussianDWM++: Language-Grounded 3D Gaussian Driving World Model for Unified Scene Understanding, Editing, and Multi-Modal Generation
提出基础特征Gaussian驾驶世界模型，在驾驶基准上统一验证场景理解、语言推理、可控4D编辑与多模态生成。
核心要点
- 现有DWM多关注条件场景生成，缺少显式3D理解、语言推理和可控4D编辑。
- 方法将Qwen/SigLIP视觉语言特征蒸馏到3D Gaussian，构建开放词汇语义场。
- 通过几何感知适配器和KL对齐，将Gaussian世界令牌与基础图像令牌对齐。
- 在更广泛驾驶基准上多任务达到SOTA表现，并计划在Github公开代码和数据集。
验证范围与局限：验证范围为更广泛驾驶基准上的场景理解、视觉定位、规划推理和可控4D生成任务。
英文摘要 (Abstract)
Driving World Models (DWMs) have recently advanced rapidly with generative models, yet most existing methods mainly focus on conditional scene generation and lack explicit 3D scene understanding, language-grounded reasoning, and controllable 4D editing capabilities. Moreover, commonly used point cloud, occupancy, or BEV representations make it difficult to achieve fine-grained alignment between textual information and the underlying 3D scene structure. To address these limitations, we propose a foundation-feature Gaussian driving world model that unifies scene understanding, language-grounded reasoning, controllable 4D editing, and multi-modal generation within a single framework. Specifically, we introduce a foundation-feature Gaussian tokenizer that directly distills Qwen/SigLIP visual-language features into 3D Gaussian primitives, building a compact open-vocabulary Gaussian semantic field. We further design a geometry-aware Gaussian adapter that combines importance-aware hierarchical selection with text-conditioned Perceiver-style cross-attention to aggregate dense Gaussian primitives into compact world tokens. To improve representation compatibility, we introduce a KL-based Gaussian--image distribution alignment objective that aligns Gaussian world tokens with foundation image tokens. Based on the aligned Gaussian representation, our framework further supports instruction-controllable scene editing, including weather-conditioned generation and dynamic vehicle manipulation. Extensive experiments on broader driving benchmarks demonstrate that our method achieves state-of-the-art performance across scene understanding, visual grounding, planning-oriented reasoning, and controllable 4D generation tasks. We will release the code and datasets publicly on Github.
作者：Tianchen Deng, Xuefeng Chen, Shuang Wu, Qu Chen, Jiajun Zhu, Bo Dai, Jianfei Yang, Hesheng Wang
来源
本页事实对应的公开页面，以原文为准。