论文HF Daily Papers · arXiv 2611.247894 天前
HINT-WAM:基于后见自蒸馏的世界动作模型在线训练框架
本文由 AI 根据公开内容整理,可能有误,请以原文为准:HINT-WAM: Hindsight-Informed Training for World Action Models(HF Daily Papers · arXiv 2611.24789)
核心内容
针对世界动作模型(WAMs)在机器人操作中难以将泛化能力转化为可靠执行的问题,研究提出HINT-WAM框架。该方法旨在克服离线监督微调仅依赖噪声演示、无法覆盖自身采样中间状态,以及在线强化学习因奖励稀疏导致探索困难的瓶颈。作为首个On-policy自蒸馏方案,它允许模型在部署时输入下,利用自身生成的去噪轨迹进行自我监督训练,显著提升了从预训练到任务执行的迁移效率。
技术上,HINT-WAM将未来演示视频作为“后见”特权信息赋予教师模型,使其在学生模型自身采样的状态点上提供条件动作分布作为密集训练目标。区别于传统速度场回归,该框架采用能量分数(Energy Score)作为分布级优化目标,将教师的高维动作分布直接蒸馏至学生的流匹配动作专家中。整个过程无需环境奖励函数或仿真器交互,项目主页及代码已在GitHub开源,为具身智能算法训练提供了新范式。
关键看点
- 自蒸馏机制:首创On-policy自蒸馏框架,学生模型基于自身去噪轨迹生成动作,教师模型利用未来视频后见信息提供条件动作分布作为密集监督信号
- 能量分数优化:摒弃传统速度场回归,采用能量分数作为分布级目标函数,直接将教师动作分布匹配至学生模型,提升流匹配动作专家的训练稳定性
- 零交互训练:训练过程仅需演示数据与模型自身采样,完全脱离环境奖励函数与仿真器交互,解决了RL中信用分配难与SFT中分布偏移问题
- 特权信息利用:将未来演示视频定义为WAM自身的后见特权信息,使教师能在学生实际访问的状态上提供精准指导,弥补了纯离线数据的覆盖盲区
工程与产业观察
HINT-WAM验证了利用视频生成模型内部表征作为自监督信号的可行性,为具身大模型摆脱对昂贵真机RL试错和高保真仿真的依赖提供了新路径。其能量分数蒸馏策略对流匹配架构的适配,对解决机器人基础模型从“看懂”到“做对”的工程落地具有直接参考价值,尤其适用于缺乏密集奖励定义的复杂操作场景。
英文摘要 (Abstract)
World action models (WAMs) have shown promising generalization and data efficiency in robot manipulation by coupling action generation with predictions of future visual dynamics. Translating these capabilities into reliable task execution, however, remains a challenge for existing post-training methods. Offline supervised fine-tuning provides dense supervision on expert trajectories, but its flow-matching action expert is trained only on noised demonstration actions and never on the intermediate states produced by its own sampler; task-specific adaptation can also compromise pretrained capabilities. On-policy reinforcement learning trains on the policy's own behavior, yet sparse task rewards make exploration and credit assignment difficult, while dense reward shaping still provides scalar evaluations rather than direct action targets. We introduce HINT-WAM, the first on-policy self-distillation framework that supervises the action expert on its own sampling trajectories using hindsight, the future video of the demonstration, as privileged information available to the WAM itself. For each demonstration observation, the student generates its action through its own denoising trajectory, and at the states along this trajectory the same model acts as a teacher conditioned additionally on this hindsight, providing a conditional action distribution as a dense training target, while the student sees only deployment-time inputs and needs no environmental rewards or simulator interaction. To transfer this supervision to a flow-matching action expert, HINT-WAM distills the teacher into the student with an energy score, a distribution-level objective that matches the teacher's action distribution from the student's own samples rather than regressing its velocity field. The project web is available in https://tuandattt.github.io/hintwam/.
Hugging Face Daily Papers 热度:22 upvotes 作者:Phuong Tuan Dat, Qi Li, Xinchao Wang
外部来源
本页事实对应的公开页面,以原文为准。(点击将离开本站)