论文HF Daily Papers · arXiv 2611.171107 天前
SALA框架实现具身模型双轴扩展,利用非配对视频提升机器人任务成功率
本文由 AI 根据公开内容整理,可能有误,请以原文为准:SALA: Semantically Aligned Latent Actions Enable Dual-Axis Scaling of Embodied Foundation Models(HF Daily Papers · arXiv 2611.17110)
核心内容
针对现有潜动作模型(LAMs)仅依赖视频重建导致表征按视觉而非动作聚类、难以跨具身对齐的痛点,研究提出SALA框架。该方法利用非配对人机视频的语言描述估计语义相似度,将其转化为软邻域结构以引导人类潜动作向语义相关的机器人潜动作靠拢,同时通过机器人动作重建锚定控制空间,从而在无配对数据条件下建立统一的共享潜动作空间。
SALA实现了具身基础模型的双轴扩展能力:在预训练阶段,增加通用人类视频数据量可持续降低下游机器人动作预测误差;在后训练阶段,增加任务特定人类视频数据能显著提升真实世界任务成功率。相比世界动作模型(WAMs),SALA避免了策略学习与执行中的显式像素预测,在保持跨模态语义对齐的同时提供了更高的训练效率与更低的推理成本。
关键看点
- 语义对齐机制:利用非配对视频语言描述构建软邻域结构,引导人类潜动作向机器人潜动作语义靠拢
- 双轴扩展验证:预训练通用视频降低动作预测误差,后训练任务视频提升真机任务成功率
- 无配对数据训练:仅需语言描述作为桥梁,无需昂贵的人机配对交互数据即可建立统一监督信号
- 推理效率优势:相比WAMs避免显式像素预测,在策略学习和执行阶段具备更低推理成本
工程与产业观察
SALA为非配对人类视频赋能机器人学习提供了工程化范式,解决了LAMs语义错位瓶颈。其验证的数据双轴扩展规律表明,规模化利用互联网人类视频进行预训练与任务微调是可行的,为缓解具身智能高质量交互数据稀缺问题指明了低成本技术路径,对多模态具身大模型训练具有直接参考价值。
英文摘要 (Abstract)
Embodied foundation models hinge on diverse robot interaction data, which are costly to collect. Egocentric human videos offer a complementary source of experience, and latent action models (LAMs) provide a popular means of leveraging these videos for policy supervision. Compared with world-action models (WAMs), LAMs can offer greater training efficiency and potentially lower inference costs by avoiding explicit pixel prediction during policy learning and execution. However, most existing LAMs learn latent actions solely through video reconstruction, which can cause their representations to cluster by visual appearance rather than by the underlying actions. As a result, they struggle to align human actions with their robot counterparts. To bridge the gap, we introduce SALA, which emphasizes establishing the semantic correspondence for latent actions. Without requiring paired data, we use the language descriptions of unpaired human and robot clips to estimate semantic similarities and supervise cross-embodiment correspondence. Specifically, we translate these similarities into soft neighborhood structures that guide human latent actions toward semantically related robot latents, while robot action reconstruction grounds the latent space in control. The resulting shared latent action space provides a unified supervision signal for training an embodied foundation model on both human and robot videos. Notably, latent action supervision enables scaling along two axes: (1) pre-training on increasing amounts of general human video consistently reduces robot action prediction error on downstream tasks, and (2) post-training on increasing amounts of task-specific human video substantially improves real-world robot task success rates.
Hugging Face Daily Papers 热度:13 upvotes 作者:Jiachun Jin, Zetong Zhou, Haonan Wang, Zhijie Deng
外部来源
本页事实对应的公开页面,以原文为准。(点击将离开本站)