外观
论文arXiv 2610.1757215 天前
EgoCPT：通过接触感知优化从单个第一视角视频学习灵巧操作
原文链接：EgoCPT: Learning Dexterous Manipulation from a Single Ego-Centric Video with Contact-Aware Optimization（arXiv 2610.17572 · 发布日期：2026-10-13）
EgoCPT: Learning Dexterous Manipulation from a Single Ego-Centric Video with Contact-Aware Optimization
为从人类第一视角视频学习灵巧操作，EgoCPT用接触感知优化重定向手轨迹，并在五个任务上验证。
核心要点
- 专家机器人示教难采集，而人类第一视角视频丰富，但 embodiment gap 使物理合理轨迹重建困难。
- 方法仅需第一视角RGB视频和参考手姿态轨迹，并迭代优化机器人手与物体的接触点。
- 该框架对噪声手姿态更鲁棒，在 noisy EgoDex 与 TACO 重建上均优于多个基线。
- 五个操控任务中，noisy EgoDex 成功率为67.3%，TACO 高质量重建为92.0%。
验证范围与局限：验证范围为五个操控任务，以及 noisy EgoDex 和基于动捕硬件采集的 TACO 手姿态重建。
英文摘要 (Abstract)
While expert demonstrations on real robots are difficult to collect, human ego-centric videos are easy to collect and abundant on the internet. However, learning dexterous manipulation policies from human videos remains challenging---one of the most important challenges being the difficulty of reconstructing physically grounded robot trajectories due to the large embodiment gap. In this paper, we propose EgoCPT, a novel paradigm for learning high-quality retargeted dexterous hand trajectories directly from an ego-centric video, which can then be deployed directly on real robots. The only inputs to our method are an ego-centric RGB video and a reference hand pose trajectory, which can be easily obtained through off-the-shelf software or specialized cameras during recording. Our method is particularly robust to noisy hand poses attributed to our novel contact-aware optimization framework, which iteratively optimizes the contact points between the robot hands and objects. Across five manipulation tasks, our method achieves overall success rates of 67.3% on noisy EgoDex hand-pose reconstructions and 92.0% on high-quality TACO reconstructions collected using motion-capture hardware, outperforming multiple baselines on both benchmarks.
作者：Zhixin Sun, Kairui Shi, Tianao Ren, Ming Yang, Mark R. Cutkosky, James Jingxi Xu
来源
本页事实对应的公开页面，以原文为准。