论文HF Daily Papers · arXiv 2611.05377昨天

DreamX复用视频扩散模型构建通用潜动作空间实现零样本跨具身迁移

DreamX将预训练视频扩散模型转化为通用潜动作学习器，单帧压缩300倍至4 tokens。该方法在LARY-Bench达SOTA，支持人-机跨具身零样本动作迁移与任意视频转换，显著降低基于视频扩散策略的计算开销。

AI 整理

仿真与训练
数据与采集
具身模型

本文由 AI 根据公开内容整理，可能有误，请以原文为准：DreamX: Video Diffusion Model is a Universal Latent Action Learner（HF Daily Papers · arXiv 2611.05377）

核心内容

针对现有潜动作模型难以解耦动作与视觉变化、限制跨具身迁移的痛点，研究提出DreamX框架。该方法复用预训练视频扩散模型作为通用潜动作学习器，从丰富时空特征中提取紧凑潜动作表示。实验显示其将每帧潜变量从1200 tokens压缩至4 tokens（300倍压缩率），并在LARY-Bench基准测试中取得当前最优性能，验证了其在统一动作表征方面的有效性。

技术核心在于利用视频生成先验解耦动作相关动态与外观干扰，构建跨越人类/机器人本体、环境及参数化方式的共享动作接口。该潜空间不仅支持任意场景间的视频转换，更实现了跨具身零样本动作迁移。此外，紧凑的潜动作空间使策略学习可直接在低维流形上进行，有效规避了高维像素空间计算瓶颈，大幅降低了视频扩散策略的训练与推理成本。

关键看点

极致压缩：从视频模型提取潜动作，单帧由1200 tokens压缩至4 tokens，压缩率达300倍
跨体迁移：构建人机共享动作接口，支持零样本动作迁移与任意视频转换，解耦外观干扰
性能验证：在LARY-Bench基准测试中刷新SOTA，证明通用潜动作表征的有效性
计算优化：策略学习直接在紧凑潜空间进行，显著降低视频扩散策略的计算与训练成本

工程与产业观察

DreamX证明了预训练视频大模型可作为通用的具身动作编码器，通过300倍压缩构建与外观解耦的共享潜空间。这为解决异构机器人技能迁移提供了新范式，表明利用生成式先验提取低维动作表征是降低具身策略训练成本、实现Zero-shot跨本体泛化的关键技术路径。

英文摘要 (Abstract)

Learning a universal action representation from video requires modeling dynamics that are consistent across different executions of the same action yet invariant to changes in embodiment and appearance. Existing latent action models infer actions from visual transitions, but their representations often entangle the underlying action with irrelevant visual variations, limiting transfer across embodiments. We introduce DreamX, which repurposes a pretrained video diffusion model as a universal latent action learner. Our approach extracts compact latent actions from the video model's rich spatiotemporal features, compressing each latent video frame by 300times (1{,200 rightarrow 4} tokens). In addition, by leveraging the video model's generative prior, these latent actions capture action-relevant dynamics disentangled from visual appearance. The resulting latent space serves as a shared action interface across embodiments (human and robot), environments, and action parameterizations. On LARY-Bench, DreamX achieves state-of-the-art performance among latent action models. Moreover, actions encoded in one setting remain effective in another: the latent space supports any-to-any video transfer and zero-shot action transfer across diverse embodiments and environments. Furthermore, this shared interface also enables policy learning in a compact latent action space, reducing the computational cost of video-diffusion-based policies.

Hugging Face Daily Papers 热度：43 upvotes 作者：JisooKim, Chaehyun Kim, Junsu Kim, Jiho Park, Seongchan Kim, Eunbeen Hong, Soowon Son, Fu-En Yang, Seungryong Kim