Skip to content

论文HF Daily Papers · arXiv 2611.266173 天前

O²LaF复用视频DiT先验实现无标签视频自监督潜在动作学习 ​

本文由 AI 根据公开内容整理,可能有误,请以原文为准:One Prior, Two Dynamics: Repurposing Video Diffusion Transformers for Self-Supervised Latent Action Learning(HF Daily Papers · arXiv 2611.26617)

核心内容 ​

针对潜在动作学习依赖从头训练动力学模型的问题,研究提出O²LaF(One Prior, Two Dynamics)框架。该方法核心在于复用同一个预训练视频Diffusion Transformer(DiT)作为逆动力学与正动力学的共享先验,而非分别独立建模。通过在导航和机器人操作任务上的验证,O²LaF在无标签视频数据上实现了自监督学习,并在少样本动作探测(few-shot action probing)基准上取得了新的State-of-the-Art性能,同时显著增强了跨视觉上下文与不同具身形态的零样本动作迁移能力。

技术架构上,O²LaF将同一DiT适配为两个角色:逆动力学采用带可学习动作查询的前馈转换编码器,正动力学采用因果式动作条件视频生成器。系统通过控制信息访问路径并利用紧凑瓶颈层进行端到端联合优化,迫使模型从视觉转换中提取可迁移的潜在动作表征。此外,该框架成功扩展至V-JEPA2等预测性表征先验并保持竞争力,证实了将预训练视频生成模型重参数化为动作发现与后果预测工具的通用范式可行性。

关键看点 ​

  • 统一先验架构:复用同一预训练视频DiT初始化逆/正动力学模型,通过紧凑瓶颈端到端优化提取可迁移潜在动作
  • SOTA性能验证:在导航与机械臂操作任务中刷新少样本动作探测记录,实现跨具身与视觉上下文的零样本迁移
  • 双动力学适配:逆动力学用可学习查询前馈编码,正动力学用因果动作条件生成,通过信息流控制解耦角色
  • 先验泛化能力:框架兼容V-JEPA2等预测性表征模型,验证生成式视频先验用于动作发现与后果预测的通用性

工程与产业观察 ​

O²LaF证明了预训练视频生成模型可作为具身智能动作表征学习的通用基座,无需海量标注数据即可提取跨具身可迁移动作空间。这为利用互联网规模视频数据解决机器人技能冷启动、降低真机数据采集成本提供了新范式,对构建通用具身基础模型具有重要工程参考价值。

英文摘要 (Abstract) ​

Videos reveal how the world changes, but rarely the actions that cause those changes. Latent action learning recovers such supervision from visual transitions, yet often learns the underlying dynamics largely from scratch. We ask whether a pretrained video generator can instead provide the prior for both inverse and forward dynamics. We introduce O^{2}LaF, built on a simple principle: one prior, two dynamics. We initialize both models from the same pretrained video DiT and adapt them to their distinct roles: a feed-forward transition encoder with learnable action queries for inverse dynamics, and a causal, action-conditioned video generator for forward dynamics. By controlling information access and optimizing both dynamics end-to-end through a compact bottleneck, O^{2}LaF learns transferable actions from unlabeled videos. Across navigation and robotic manipulation, O^{2}LaF sets a new state of the art in few-shot action probing and enables stronger zero-shot action transfer across visual contexts and embodiments. Beyond generative video priors, we further extend our framework to predictive representation priors such as V-JEPA2, achieving competitive performance. Our findings suggest a broader paradigm for latent-action learning: repurposing pretrained video models to discover transferable actions and predict their consequences.

Hugging Face Daily Papers 热度:1 upvotes 作者:Yuxin Jiang, Wenqiang Sun, Jiaqi Li, Yuchao Gu, Lvmin Zhang, Ivor Tsang, Mike Zheng Shou

基于本页技术方向关联的整机参数、核心零部件、实测报告与同赛道选型横评。

外部来源

本页事实对应的公开页面,以原文为准。(点击将离开本站)

  1. HF Daily Papers · arXiv 2611.26617:One Prior, Two Dynamics: Repurposing Video Diffusion Transformers for Self-Supervised Latent Action Learning ↗访问 2026-11-21