# RoboTryOn：人类视频转机器人演示，双臂操作成功率升至60%

> 原文链接：https://robotics.hellobike.com/papers/2026-11-03-2611-05163-fe8afec9
> 频道：papers | 更新日期：2026-11-03 | 标签：auto, arxiv, hf-daily-papers
> 核心摘要：提出RoboTryOn框架将人类视频转化为目标机器人演示，结合逆动力学模型生成策略标签。实验显示合成数据训练使新任务双臂操作成功率从2%提升至60%，且性能匹敌真机遥操数据，采集成本降低约三分之二。

# RoboTryOn：人类视频转机器人演示，双臂操作成功率升至60%

::: v-pre
> 本文由 AI 根据公开内容整理，可能有误，请以原文为准：[RoboTryOn: Synthesizing Dexterous Robot Demonstrations from Human Videos via Virtual Try-On](https://arxiv.org/abs/2611.05163)（HF Daily Papers · arXiv 2611.05163）

## 核心内容

针对机器人基础模型适配新任务依赖昂贵遥操数据的痛点，研究提出RoboTryOn框架，将人-机迁移重构为“虚拟试穿”问题。该方法保留原始人类演示的运动、物体与背景，仅将人体替换为目标机器人本体。通过自回归关键帧翻译与手部骨骼条件引导视频合成，并结合深度特征的逆动力学模型（IDM）推断动作标签，解决了高自由度具身转换中的视觉与动作鸿沟。

在多个人形机器人本体上的实验表明，仅使用RoboTryOn合成数据训练的策略，在包括复杂双臂操作在内的新任务中平均成功率从2%跃升至60%，无需任何真实机器人演示。此外，该合成数据训练出的策略性能与纯遥操数据训练持平，但数据采集成本仅为后者的三分之一，验证了利用大规模人类视频进行低成本具身智能策略学习的有效性。

## 关键看点

- **虚拟试穿范式**：将人机迁移视为保留运动与场景的虚拟试穿，通过自回归关键帧与手骨条件引导生成目标机器人视频
- **逆动力学标注**：集成深度特征的逆动力学模型（IDM）从合成视频中自动推断动作标签，支撑无真机数据的策略学习
- **双臂操作突破**：合成数据训练使高难双臂操作等新任务平均成功率从2%提升至60%，完全摆脱对真实演示的依赖
- **采集成本优化**：合成数据训练策略性能匹敌纯遥操数据，数据采集成本降至传统方法的三分之一，显著提升数据效率

## 工程与产业观察

该工作验证了将海量人类视频转化为高质量机器人训练数据的可行性，通过生成式视频与逆动力学结合打破了具身数据瓶颈。对于缺乏真机采集条件的团队，此方案提供了低成本构建双臂及复杂操作策略的有效路径，加速了通用机器人大模型的泛化落地。

## 英文摘要 (Abstract)

Robot foundation models have shown remarkable performance in robot control, yet adapting them to a new task still relies heavily on teleoperated demonstrations. Egocentric human videos offer a scalable substitute, but overlaying rendered robots on human videos or retargeting human poses into robot actions leaves significant visual or action gaps for high-DoF robot embodiments. To address this limitation, we propose RoboTryOn, which treats human-to-robot translation as virtual try-on: try-on keeps the person and swaps the garment, RoboTryOn keeps the demonstrated motion and swaps the human for the robot. Specifically, RoboTryOn converts a human video into a video of the target robot specified by a reference image, while preserving the original task, objects, and background. To maintain visual consistency, we translate selected keyframes autoregressively and use them to guide video synthesis with motion conditioned on the human hand skeleton. Additionally, we use an inverse dynamics model (IDM) that incorporates depth features to infer action labels from synthesized videos for policy learning. Our experiments across multiple humanoid embodiments show that policies trained with our synthetic data improve the average success rate on novel tasks, including challenging bimanual manipulation, from 2% to 60%, without any real robot demonstration of those tasks. Moreover, training with our synthetic data matches the performance of policies trained solely on teleoperated demonstrations at approximately a third of the collection cost, demonstrating the cost efficiency of our approach.

Hugging Face Daily Papers 热度：3 upvotes
作者：Seungyong Lee, Suhyeok Jang, Minyoung Kim, Jinwoo Shin
:::

## 原始参考来源 (Sources)

- [HF Daily Papers · arXiv 2611.05163：RoboTryOn: Synthesizing Dexterous Robot Demonstrations from Human Videos via Virtual Try-On](https://arxiv.org/abs/2611.05163)