论文/arXiv 2609.34182·1 个月前

基于人类示教的灵巧操作统一视觉-触觉-动作建模

针对灵巧操作触觉示教难扩展，提出统一视觉-触觉-动作模型，利用人类触觉示教，并在五项真机任务验证。

AI 整理

• 本体 SDK 与控制
• 数据与采集
• 端侧部署

原文链接：Unified Visual-Tactile-Action Modeling from Human Demonstrations for Dexterous Manipulation（arXiv 2609.34182 · 发布日期：2026-09-28）

Unified Visual-Tactile-Action Modeling from Human Demonstrations for Dexterous Manipulation

针对灵巧操作触觉示教难扩展，提出统一视觉-触觉-动作模型，利用人类触觉示教，并在五项真机任务验证。

核心要点

• 灵巧手遥操作触觉反馈有限，使机器人触觉示教难以规模化，人类示教提供更丰富来源。
• 构建同步记录图像、触觉信号和手部运动的触觉动捕系统，并建立UVTA数据集。
• 模型将人和机器人映射到对齐的触觉与动作表征，联合预测未来动作和触觉轨迹。
• 五项真机任务平均成功率70%，高于视觉-触觉基线29%和结构消融42%；项目页面已公开。

验证范围与局限：验证范围为真实机器人上的五项接触丰富任务。

英文摘要 (Abstract)

Dexterous manipulation requires tactile feedback. However, robot tactile demonstrations are difficult to scale,because dexterous-hand teleoperation provides limited tactile feedback to the operator. In contrast, human demonstrations offer a substantially more scalable source of diverse tactile interactions. Motivated by a simple premise: hands can change, but the underlying physics of interaction does not. We leverage human tactile data to improve dexterous manipulation policies. Specifically, we first build a tactile motion-capture system that synchronously records images, tactile signals, and hand motions. Using this system, we construct the UVTA dataset spanning five contact-rich tasks, with 1,000 human demonstrations covering diverse interaction patterns and 150 robot demonstrations per task. To transfer the underlying physics of human interaction to robot control, we propose a Unified Visual-Tactile-Action Model that maps both embodiments into aligned tactile and action representations and jointly predicts future action and tactile trajectories. The joint objective enables human demonstrations to supervise contact-aware representation learning, while only robot actions are executed during deployment. In real-robot evaluations across five tasks, our method achieves an average success rate of 70%, outperforming the strongest visual-tactile baseline, which achieves 29%, and an architecture ablation, which achieves 42%. Performance improves consistently with additional human demonstrations and exhibits no saturation at 1,000 demonstrations per task, validating the effectiveness of scalable human tactile data for dexterous manipulation. Project page is available at https://uni-vta.github.io/.

作者：Wenqiao Li, Qianyou Zhao, Jiawen Hao, Xuezhou Zhu, Tengyu Liu, Kaifeng Zhang, Chuan Wen, Siyuan Huang

来源

本页事实对应的公开页面，以原文为准。

1. arXiv 2609.34182：Unified Visual-Tactile-Action Modeling from Human Demonstrations for Dexterous Manipulation访问 2026-10-21