论文/arXiv 2609.07747·1 个月前

Dex-X：通过仿真交互从人类视频学习视觉-触觉灵巧操作

DEX-X将单目人类演示重建到仿真中补全触觉监督，训练视觉-触觉策略，并在仿真与真机灵巧手臂任务上验证。

AI 整理

• 仿真与训练
• 数据与采集

原文链接：Dex-X: Learning Visual-Tactile Dexterous Manipulation From Human Videos with Simulated Interaction（arXiv 2609.07747 · 发布日期：2026-09-07）

Dex-X: Learning Visual-Tactile Dexterous Manipulation From Human Videos with Simulated Interaction

DEX-X将单目人类演示重建到仿真中补全触觉监督，训练视觉-触觉策略，并在仿真与真机灵巧手臂任务上验证。

核心要点

• 人类视频包含丰富灵巧操作，但缺少接触交互所需的触觉信息。
• DEX-X在仿真中重建手-物交互，用物理接触动态提供触觉监督。
• 方法将策略蒸馏为使用点云观测和触觉传感的可部署策略。
• 教师策略仿真平均成功率65.9%；真机立方体抓取93%、桌面清洁53%。

验证范围与局限：验证范围为仿真六类任务，以及灵巧手臂平台上的多种抓取和接触丰富工具使用任务。

英文摘要 (Abstract)

Human videos are an abundant source of dexterous manipulation behaviors, but they lack tactile information that is crucial for contact-rich interaction. This raises a fundamental question: can robots learn deployable visual-tactile dexterous manipulation policies from human video demonstrations without robot-side data collection? We present DEX-X, a framework for learning visual-tactile dexterous manipulation from human videos through simulation. Our key insight is that simulation can serve as a tactile completion engine. Given monocular human demonstrations, DEX-X reconstructs hand-object interactions in simulation, where physically grounded contact dynamics provide tactile supervision unavailable in the original videos. Leveraging this recovered tactile information, we train visual-tactile dexterous manipulation policies and distill them into deployable policies operating on point-cloud observations and tactile sensing. We demonstrate zero-shot sim-to-real transfer on a dexterous hand-arm platform across diverse grasping and contact-rich tool-use tasks. The teacher policy achieves 65.9% average success across six task categories in simulation, while the distilled visual-tactile policy achieves 93% success on real-world cube picking and 53% on the challenging table-cleaning task. Zero-shot generalization to unseen object geometries is also observed on object-picking tasks. Our results suggest that simulated interaction is a key bridge between human videos and deployable dexterous manipulation policies, providing the missing physical supervision needed for scalable robot skill learning from Internet-scale human video data.

作者：Ruoqu Chen, Feixiang Ruan, Liu Cao, Zihao Wang, Botian Xu, Shiqin Tong, Jiajun Liu, Mingzhi Pei, Chenyu Zhang, Wanli Xing

来源

本页事实对应的公开页面，以原文为准。

1. arXiv 2609.07747：Dex-X: Learning Visual-Tactile Dexterous Manipulation From Human Videos with Simulated Interaction访问 2026-10-21