论文HF Daily Papers · arXiv 2611.1287013 天前
SWAM世界动作模型:4.7万小时数据对齐语言视频与动作控制
本文由 AI 根据公开内容整理,可能有误,请以原文为准:SWAM: Scaling World Action Models with Cross-Modal Alignment(HF Daily Papers · arXiv 2611.12870)
核心内容
针对世界动作模型(WAM)引入动作学习导致语言-视频对齐退化的问题,研究团队构建了包含通用视频、第一视角视频、UMI及机器人演示的约4.7万小时数据集。通过系统对比不同监督信号组合,发现混合训练能有效维持预训练对齐能力。最终模型在固定算力预算下,于RoboTwin Clean任务的Easy和Hard条件下分别取得88.10%和61.56%的成功率。
技术验证表明,混合第一视角视频与UMI演示数据可显著提升控制泛化性,且模型架构直接影响经验利用效率。在三个真实世界未见任务中,SWAM实现40.00%的操作成功率,较最强基线23.33%提升近一倍,证实了高质量标注与联合训练配方对具身智能迁移的关键作用。项目组已完整开源清洗后的数据集、训练代码及模型权重供社区复现。
关键看点
- 数据规模:构建约4.7万小时混合数据集,涵盖通用视频、Ego视频、UMI及机器人演示以支撑跨模态对齐研究
- 仿真性能:仅用RoboTwin Clean数据后训练,在Easy与Hard条件下成功率分别达88.10%和61.56%
- 真机泛化:在三个真实世界未见任务中取得40.00%成功率,较最强基线23.33%提升16.67个百分点
- 开源资产:完整发布清洗数据集、训练代码及模型权重,提供可复现的世界动作模型联合训练配方
工程与产业观察
该工作揭示了WAM训练中语言-视频对齐与动作学习的冲突机制,证明混合Ego视频与UMI数据是低成本提升泛化的关键。其开源的4.7万小时对齐数据集与联合训练Recipe,为具身大模型解决Sim-to-Real鸿沟提供了高价值基准与工程范式。
英文摘要 (Abstract)
World action models jointly predict video and actions conditioned on language. Existing studies focus primarily on architectures and training mechanisms, typically treating language-video alignment acquired through video pretraining as an established foundation. However, introducing action learning may impair this alignment. Principled guidance on data and supervision design for maintaining language-video alignment during action learning remains limited. To investigate this, we curate a pool of approximately 47,000 hours of general videos, egocentric videos, UMI demonstrations, and robot demonstrations. Building on this, we systematically study how language-video supervision, language-video-action supervision, and the joint use of these two forms affect WAM performance. In particular, we examine how language annotation quality affects pretrained language-video capabilities, and how action supervision from UMI and robot demonstrations extends these capabilities to language-video-action modeling. To better scale the model, we further conduct detailed studies on joint training recipe and model architecture. Under a fixed training budget, mixing egocentric videos and UMI demonstrations improves control generalization. Model architecture affects how effectively these experiences are used, while scaling pretraining under the selected configuration brings further gains. The resulting recipe, post-trained on RoboTwin Clean demonstrations only, achieves success rates of 88.10% and 61.56% under Easy (Clean-to-Clean) and Hard (Clean-to-Random) conditions, respectively. Our model achieves 40.00% SR across three real-world tasks, compared with 23.33% for the strongest baseline, indicating better generalization under unseen conditions. We release the curated data, training code, and model weights.
Hugging Face Daily Papers 热度:3 upvotes 作者:zhou yu, Xinyuan Xia, Xuankun Yang, Yinghao Yang, Wendi Chen, Bowen Liu, Jianhan Ma, Weicheng Duan, Jun Lv, Cewu Lu, Yanwei Li
外部来源
本页事实对应的公开页面,以原文为准。(点击将离开本站)