外观
论文arXiv 2610.2090414 天前
EgoJEPA：超越重定向的从第一视角视频学习灵巧操作
原文链接：EgoJEPA: Beyond Retargeting for Dexterous Manipulation from Egocentric Videos（arXiv 2610.20904 · 发布日期：2026-10-14）
EgoJEPA: Beyond Retargeting for Dexterous Manipulation from Egocentric Videos
EgoJEPA为人类视频到机器人控制迁移去除规则重定向，以三阶段表征与世界建模在真实操作任务上验证。
核心要点
- 第一视角人类视频可扩展，但几何重定向需反复校准且会传递手跟踪误差。
- 框架以JEPA式视觉预测预训练效应感知潜动作，再进行潜动作世界建模。
- 预训练与机器人控制参数化解耦，并用轻量动作适配器适配不同机器人。
- 在五个真实操作任务、三种机器人形态上，域外成功率74%，高于65%基线且耗时更低。
验证范围与局限：验证范围为五个真实世界操作任务，覆盖三种机器人形态。
英文摘要 (Abstract)
Egocentric human videos offer a scalable alternative to costly robot demonstra- tions, yet transferring their manipulation priors to robot control remains chal- lenging. The commonly adopted retargeting-based approaches convert human wrist and finger poses into embodiment-specific action commands through hand- crafted geometric mappings. However, such mappings require meticulous recal- ibration across different embodiments and risk propagating hand-tracking errors into downstream robot control pipelines. To this end, we propose EgoJEPA, a three-stage framework that learns embodiment-agnostic representations from ego- centric human videos without rule-based human-to-robot retargeting: 1) Human- action pre-training learns effect-aware latent actions through JEPA-style visual prediction conditioned on hand-motion differences; 2) Latent-action-based world modeling jointly predicts future visual dynamics and latent actions, which decou- ples human prior pre-training from robot-specific control parameterizations; 3) Embodiment-specific tuning adapts the pretrained model to diverse robots through lightweight action adapters. Experiments on five real-world manipulation tasks across three robot embodiments demonstrate the effectiveness and efficiency of EgoJEPA. It achieves a 74% average success rate under the out-of-domain evalu- ation, compared with 65% for the retargeting baseline, while reducing cumulative cross-embodiment wall-clock cost by 47.2%, from 743.2 to 392.7 hours.
作者：Meng Cao, Can Zhao, Jinjie Wei, Haonan Zhao, Yulong Zhang, Duo Han, Xiaofan Li, Ziwu Song, Rongtao Xu, Ruyi Gan
来源
本页事实对应的公开页面，以原文为准。