外观
论文arXiv 2610.1085521 天前
OmniHOI：从单目人类视频生成灵巧手-物体交互
原文链接：OmniHOI: Dexterous Hand-Object Interaction from Monocular Human Video（arXiv 2610.10855 · 发布日期：2026-10-07）
OmniHOI: Dexterous Hand-Object Interaction from Monocular Human Video
OmniHOI将RGB手物交互视频转为灵巧手轨迹，并在动捕、单目视频和真实双臂机器人任务上验证。
核心要点
- 单目人类操作视频示范丰富，但单视角重建交互并迁移到机器人手仍困难。
- 方法在重建、重定向和物理环优化中分别利用图像证据、接触几何和动力学。
- 在150条动捕轨迹迁移到五种6至22 DoF灵巧手上，成功率39-89%。
- 在60段单目视频上成功率53%，高于既有最佳视频到机器人流程的28%。
验证范围与局限：验证包括动捕轨迹、单目视频片段，以及在真实双臂机器人上的多样任务执行。
英文摘要 (Abstract)
Monocular videos of human manipulation provide abundant dexterous demonstrations, yet reconstructing hand-object interaction from a single view and transferring it to robot hands remain difficult, limiting their direct use for robot execution. Prior methods either require task-specific RL training, limiting scalability, or assume clean motion-capture trajectories and thus cannot operate directly on video. We present OmniHOI, a pipeline that turns an RGB video of hand-object interaction into an interaction-faithful trajectory on dexterous hands. The key idea is to enforce physical consistency using the evidence available at each stage: image evidence during reconstruction, contact geometry during retargeting, and dynamics during physics-in-the-loop refinement. Each stage optimizes the corresponding representation directly, correcting errors before they propagate downstream or must be absorbed by a learned policy. Across 150 motion-capture trajectories transferred to each of five dexterous hands with 6 to 22 DoF, we achieve 39-89% success, compared with at most 31% for prior transfer methods. On 60 monocular video clips, we achieve 53% success, compared with 28% for the best prior video-to-robot pipeline. Its trajectories also execute on a real bimanual robot across diverse tasks.
作者：Ting Mao, Yanming Shao, Ziheng Wang, Haoyu Liu, Yiqun Wang, Xuanye Wu, Yao Mu
来源
本页事实对应的公开页面，以原文为准。