# DimensionX：单图生成可控3D/4D场景，解耦时空维度赋能具身仿真

> 原文链接：https://robotics.hellobike.com/papers/2024-11-07-2411-04928-c9086a39
> 频道：papers | 更新日期：2024-11-07 | 标签：auto, arxiv, cs.RO, landmark
> 核心摘要：提出DimensionX框架，通过ST-Director解耦视频扩散模型时空因子，实现单图到3D/4D场景的可控生成。引入轨迹感知与身份保持策略，解决生成视频与真实场景对齐难题，在多项数据集上验证了优于现有方法的生成质量与可控性。

# DimensionX：单图生成可控3D/4D场景，解耦时空维度赋能具身仿真

::: v-pre
> 本文由 AI 根据公开内容整理，可能有误，请以原文为准：[DimensionX: Create Any 3D and 4D Scenes from a Single Image with Controllable Video Diffusion](https://arxiv.org/abs/2411.04928)（arXiv 2411.04928）

## 核心内容

针对视频扩散模型在3D/4D重建中时空可控性不足的问题，研究团队提出DimensionX框架。该方法核心洞察在于利用视频帧序列统一表征3D空间结构与4D时间演化，并通过ST-Director模块学习维度感知LoRA，从维度变异数据中解耦空间与时间因子。实验表明，该框架在真实世界及合成数据集上的3D/4D场景生成效果显著优于既往方法，为具身智能提供了高保真环境生成新范式。

技术架构上，ST-Director实现了视频扩散过程中空间结构与时间动态的精确独立操控。为弥合生成视频与物理世界的鸿沟，系统集成了两大关键机制：面向3D生成的轨迹感知模块确保几何结构一致性，以及面向4D生成的身份保持去噪策略以维持时序连贯性。这种可控生成能力使得从单张图像直接重构具备物理属性的3D/4D表示成为可能，直接支撑机器人仿真训练与视觉导航算法验证。

## 关键看点

- **ST-Director解耦**：通过学习维度感知LoRA分离时空因子，实现视频扩散中3D结构与4D动态的独立精确控制
- **轨迹感知机制**：专为3D生成设计，约束生成过程以对齐真实场景几何轨迹，提升空间结构重建精度
- **身份保持去噪**：针对4D生成引入时序一致性策略，消除帧间闪烁与形变，确保动态场景的时间连续性
- **单图多维生成**：仅凭单张输入图像即可输出照片级3D/4D场景，大幅降低具身仿真环境构建的数据门槛

## 工程与产业观察

该工作解决了生成式AI在具身仿真中“好看但不可用”的痛点，通过时空解耦与几何约束将视频生成转化为可靠的3D/4D资产管线。对于机器人算法工程师，这意味着可利用海量互联网图片快速构建带物理一致性的训练环境；对产业侧而言，降低了Sim-to-Real迁移中高质量合成数据的获取成本，加速视觉导航与操作策略的迭代验证。

## 英文摘要 (Abstract)

In this paper, we introduce \textbf{DimensionX}, a framework designed to generate photorealistic 3D and 4D scenes from just a single image with video diffusion. Our approach begins with the insight that both the spatial structure of a 3D scene and the temporal evolution of a 4D scene can be effectively represented through sequences of video frames. While recent video diffusion models have shown remarkable success in producing vivid visuals, they face limitations in directly recovering 3D/4D scenes due to limited spatial and temporal controllability during generation. To overcome this, we propose ST-Director, which decouples spatial and temporal factors in video diffusion by learning dimension-aware LoRAs from dimension-variant data. This controllable video diffusion approach enables precise manipulation of spatial structure and temporal dynamics, allowing us to reconstruct both 3D and 4D representations from sequential frames with the combination of spatial and temporal dimensions. Additionally, to bridge the gap between generated videos and real-world scenes, we introduce a trajectory-aware mechanism for 3D generation and an identity-preserving denoising strategy for 4D generation. Extensive experiments on various real-world and synthetic datasets demonstrate that DimensionX achieves superior results in controllable video generation, as well as in 3D and 4D scene generation, compared with previous methods.

作者：Wenqiang Sun, Shuo Chen, Fangfu Liu, Zilong Chen, Yueqi Duan, Jun Zhang, Yikai Wang
:::

## 原始参考来源 (Sources)

- [arXiv 2411.04928：DimensionX: Create Any 3D and 4D Scenes from a Single Image with Controllable Video Diffusion](https://arxiv.org/abs/2411.04928)