# EgoMatrix：来自第一视角人类视频的物理忠实灵巧机器人示范

> 原文链接：https://robotics.hellobike.com/papers/2026-10-13-2610-18822
> 频道：papers | 更新日期：2026-10-13 | 标签：arxiv, cs.RO
> 核心摘要：EgoMatrix将单目第一视角人类视频转为双手灵巧轨迹和机器人观测视频，并在TACO、OakInk2和自采数据上验证。

# EgoMatrix：来自第一视角人类视频的物理忠实灵巧机器人示范

::: v-pre
> 原文链接：[EgoMatrix: Physically Faithful Dexterous Robot Demonstrations from Egocentric Human Videos](https://arxiv.org/abs/2610.18822)（arXiv 2610.18822 · 发布日期：2026-10-13）

**EgoMatrix: Physically Faithful Dexterous Robot Demonstrations from Egocentric Human Videos**

EgoMatrix将单目第一视角人类视频转为双手灵巧轨迹和机器人观测视频，并在TACO、OakInk2和自采数据上验证。

## 核心要点

- 第一视角人类视频可扩展采集灵巧操作数据，但单目HOI重建和高质量机器人示范仍具挑战。
- 方法强调相对几何与接触重建，经重定向和强化学习在仿真中复现任务并生成同步观测。
- 在TACO、OakInk2和自采数据上，仿真端到端成功率为54.84%，强基线为14.52%。
- 重建在两个公开基准多数指标达SOTA，项目页面已公开。

**验证范围与局限**：端到端任务成功率在仿真中验证，摘要未报告真实机器人任务执行成功率。

## 英文摘要 (Abstract)

Egocentric human videos offer a scalable alternative to robot teleoperation for collecting dexterous manipulation data. However, two challenges limit their use as reliable training data: accurate hand-object interaction (HOI) reconstruction from monocular RGB and high-quality robot demonstrations, especially visual observations. We present EgoMatrix, an end-to-end data engine that converts monocular egocentric human videos into bimanual dexterous trajectories and paired robot observation videos, with the trajectories directly replayable on real robots. It reconstructs HOI with emphasis on relative geometry and contact rather than exact metric scene recovery, then uses retargeting and reinforcement learning (RL) to reproduce the demonstrated tasks in simulation. We jointly render robot hands and objects from these simulated trajectories to produce physically faithful observations synchronized with state and control records. Rendering the same trajectories in diverse, task-compatible backgrounds expands paired training data while preserving robot-object motion and contact. Across TACO, OakInk2, and self-collected data, EgoMatrix achieves 54.84% end-to-end success in simulation versus 14.52% for the strongest baseline. Reconstruction reaches state-of-the-art performance on most metrics of the two public benchmarks. Qualitative results show more faithful hand-object interactions than prior methods, while feature-space analysis shows closer alignment with real-robot observations than GPT-6 Astra. Our project page is available at https://spikew726.github.io/EgoMatrix/.

作者：Ziheng Wang, Yanming Shao, Ting Mao, Kaihan Chen, Yiqun Wang, Xuanye Wu, Yao Mu
:::

## 原始参考来源 (Sources)

- [arXiv 2610.18822：EgoMatrix: Physically Faithful Dexterous Robot Demonstrations from Egocentric Human Videos](https://arxiv.org/abs/2610.18822)