外观
论文arXiv 2609.165041 个月前
UniDex-ViTac：从人类视频数据学习统一视觉-触觉灵巧操作策略
原文链接：UniDex-ViTac: Learning Unified Visuo-Tactile Dexterous Manipulation Policy from Human Video Data（arXiv 2609.16504 · 发布日期：2026-09-15）
UniDex-ViTac: Learning Unified Visuo-Tactile Dexterous Manipulation Policy from Human Video Data
UniDex-ViTac用人类视频引导仿真生成带触觉观测的机器人示范，并在仿真和真机物体操作中验证统一策略。
核心要点
- 针对人类视频缺少机器人可执行动作与触觉测量的问题。
- 用人类视频引导仿真，生成带指尖接触观测的机器人示范。
- 以点云、本体感知和四个二值接触信号训练ACT通用策略。
- 仿真与真机均优于仅点云基线，并提供项目页面。
验证范围与局限：在仿真中用十个物体评估，并在真机上跨六个已见和五个未见物体进行110次试验。
英文摘要 (Abstract)
Human videos provide demonstrations of dexterous manipulation but lack robot-executable actions and tactile measurements. We present UniDex-ViTac, a framework that uses human-video-guided simulation to generate robot demonstrations paired with fingertip contact observations for training a deployable visuo-tactile policy. Object-specific residual reinforcement learning specialists adapt annotated human-object interaction references to a robotic arm-hand system. Their successful rollouts pair final robot action targets with robot-side fingertip contact observations. From 50 human demonstrations across ten objects, we collect 10,000 simulated trajectories to train a single Action Chunking with Transformers (ACT) based generalist. The policy combines point clouds, proprioception, and four binary contact signals encoded through fingertip labels and a separate token, without requiring human references or privileged object identity and pose at deployment. The contact-augmented configuration achieves 68.3% macro-average success in simulation, compared with 55.5% for the point-cloud-only baseline. Without real-robot demonstrations or policy fine-tuning, it succeeds in 73/110 physical trials (66.4%) across six seen and five unseen objects, compared with 60/110 (54.5%) for the baseline, an increase of 11.8 percentage points. These results support the feasibility of learning a unified visuo-tactile dexterous manipulation policy from video-guided simulated interactions. Project page: https://unidex-vitac.github.io/
作者：Hyesung Lee, Si‐Hwan Heo, Sungwook Yang
来源
本页事实对应的公开页面，以原文为准。