# OmniHOI：从单目人类视频生成灵巧手-物体交互

> 原文链接：https://robotics.hellobike.com/papers/2026-10-07-2610-10855
> 频道：papers | 更新日期：2026-10-07 | 标签：arxiv, cs.RO
> 核心摘要：OmniHOI将RGB手物交互视频转为灵巧手轨迹，并在动捕、单目视频和真实双臂机器人任务上验证。

# OmniHOI：从单目人类视频生成灵巧手-物体交互

::: v-pre
> 原文链接：[OmniHOI: Dexterous Hand-Object Interaction from Monocular Human Video](https://arxiv.org/abs/2610.10855)（arXiv 2610.10855 · 发布日期：2026-10-07）

**OmniHOI: Dexterous Hand-Object Interaction from Monocular Human Video**

OmniHOI将RGB手物交互视频转为灵巧手轨迹，并在动捕、单目视频和真实双臂机器人任务上验证。

## 核心要点

- 单目人类操作视频示范丰富，但单视角重建交互并迁移到机器人手仍困难。
- 方法在重建、重定向和物理环优化中分别利用图像证据、接触几何和动力学。
- 在150条动捕轨迹迁移到五种6至22 DoF灵巧手上，成功率39-89%。
- 在60段单目视频上成功率53%，高于既有最佳视频到机器人流程的28%。

**验证范围与局限**：验证包括动捕轨迹、单目视频片段，以及在真实双臂机器人上的多样任务执行。

## 英文摘要 (Abstract)

Monocular videos of human manipulation provide abundant dexterous demonstrations, yet reconstructing hand-object interaction from a single view and transferring it to robot hands remain difficult, limiting their direct use for robot execution. Prior methods either require task-specific RL training, limiting scalability, or assume clean motion-capture trajectories and thus cannot operate directly on video. We present OmniHOI, a pipeline that turns an RGB video of hand-object interaction into an interaction-faithful trajectory on dexterous hands. The key idea is to enforce physical consistency using the evidence available at each stage: image evidence during reconstruction, contact geometry during retargeting, and dynamics during physics-in-the-loop refinement. Each stage optimizes the corresponding representation directly, correcting errors before they propagate downstream or must be absorbed by a learned policy. Across 150 motion-capture trajectories transferred to each of five dexterous hands with 6 to 22 DoF, we achieve 39-89% success, compared with at most 31% for prior transfer methods. On 60 monocular video clips, we achieve 53% success, compared with 28% for the best prior video-to-robot pipeline. Its trajectories also execute on a real bimanual robot across diverse tasks.

作者：Ting Mao, Yanming Shao, Ziheng Wang, Haoyu Liu, Yiqun Wang, Xuanye Wu, Yao Mu
:::

## 原始参考来源 (Sources)

- [arXiv 2610.10855：OmniHOI: Dexterous Hand-Object Interaction from Monocular Human Video](https://arxiv.org/abs/2610.10855)