外观
论文arXiv 2610.1784415 天前
什么造就了 MLLMs 空间理解的良好视觉表征？
原文链接：What Makes a Good Visual Representation for Spatial Understanding in MLLMs?（arXiv 2610.17844 · 发布日期：2026-10-13）
What Makes a Good Visual Representation for Spatial Understanding in MLLMs?
研究MLLM空间理解的良好视觉表征，比较蒸馏与融合迁移，并在静态3D和动态4D空间推理基准验证。
核心要点
- MLLM视觉预训练数据几乎全为2D，视觉token常缺少内在3D结构信息。
- 分离迁移内容与迁移方式，考察几何、自监督、视觉语言和生成模型表征。
- 将潜特征或显式深度、点图通过蒸馏作为监督，或通过融合作为额外输入。
- 发现3D结构与增益强相关；联合蒸馏后在两类空间推理基准优于基座模型。
验证范围与局限：仅以RGB视频输入，在静态3D和动态4D空间推理基准上验证。
英文摘要 (Abstract)
Spatial understanding is a necessary step in moving multimodal large language models (MLLMs) toward reasoning about the 3D physical world. Their visual pretraining data, however, is almost entirely 2D, so the visual tokens processed by MLLMs often lack inherent 3D structural information. A growing number of methods inject such structure by transferring representations from external vision or geometry models, and collectively establish that such transfer helps. Behind these methods lies a largely underexplored question: what makes a visual representation good for spatial understanding? To answer it, we first separate what is transferred from how it is transferred, and then trace the resulting differences back to properties of the representations themselves. What we transfer comes from 3D geometry models, self-supervised image and video models, vision-language models, and generative models, in the form of latent features or explicit depth and point maps. We transfer it either by distillation, using it as a supervision target, or by fusion, feeding it as an additional input. We find that a representation's 3D structure correlates strongly with how much it helps, measured by cross-view consistency for latent features and by depth and reconstruction accuracy for explicit geometry. We further show that the form of a representation dictates the mechanism it needs. Latent features gain as much from fusion as from distillation; explicit geometry gains little from fusion, and most of its distillation gain depends on a geometry decoding head. Guided by these findings, we distill both latent features and explicit geometry into a single MLLM. Taking only RGB video as input, it consistently improves over its base model on both static 3D and dynamic 4D spatial reasoning benchmarks.
作者：Xiaohu Huang, Yumin Guo, Kai Han
来源
本页事实对应的公开页面，以原文为准。