# UniDex-ViTac：从人类视频数据学习统一视觉-触觉灵巧操作策略

> 原文链接：https://robotics.hellobike.com/papers/2026-09-15-2609-16504
> 频道：papers | 更新日期：2026-09-15 | 标签：arxiv, cs.RO
> 核心摘要：UniDex-ViTac用人类视频引导仿真生成带触觉观测的机器人示范，并在仿真和真机物体操作中验证统一策略。

# UniDex-ViTac：从人类视频数据学习统一视觉-触觉灵巧操作策略

::: v-pre
> 原文链接：[UniDex-ViTac: Learning Unified Visuo-Tactile Dexterous Manipulation Policy from Human Video Data](https://arxiv.org/abs/2609.16504)（arXiv 2609.16504 · 发布日期：2026-09-15）

**UniDex-ViTac: Learning Unified Visuo-Tactile Dexterous Manipulation Policy from Human Video Data**

UniDex-ViTac用人类视频引导仿真生成带触觉观测的机器人示范，并在仿真和真机物体操作中验证统一策略。

## 核心要点

- 针对人类视频缺少机器人可执行动作与触觉测量的问题。
- 用人类视频引导仿真，生成带指尖接触观测的机器人示范。
- 以点云、本体感知和四个二值接触信号训练ACT通用策略。
- 仿真与真机均优于仅点云基线，并提供项目页面。

**验证范围与局限**：在仿真中用十个物体评估，并在真机上跨六个已见和五个未见物体进行110次试验。

## 英文摘要 (Abstract)

Human videos provide demonstrations of dexterous manipulation but lack robot-executable actions and tactile measurements. We present UniDex-ViTac, a framework that uses human-video-guided simulation to generate robot demonstrations paired with fingertip contact observations for training a deployable visuo-tactile policy. Object-specific residual reinforcement learning specialists adapt annotated human-object interaction references to a robotic arm-hand system. Their successful rollouts pair final robot action targets with robot-side fingertip contact observations. From 50 human demonstrations across ten objects, we collect 10,000 simulated trajectories to train a single Action Chunking with Transformers (ACT) based generalist. The policy combines point clouds, proprioception, and four binary contact signals encoded through fingertip labels and a separate token, without requiring human references or privileged object identity and pose at deployment. The contact-augmented configuration achieves 68.3% macro-average success in simulation, compared with 55.5% for the point-cloud-only baseline. Without real-robot demonstrations or policy fine-tuning, it succeeds in 73/110 physical trials (66.4%) across six seen and five unseen objects, compared with 60/110 (54.5%) for the baseline, an increase of 11.8 percentage points. These results support the feasibility of learning a unified visuo-tactile dexterous manipulation policy from video-guided simulated interactions. Project page: https://unidex-vitac.github.io/

作者：Hyesung Lee, Si‐Hwan Heo, Sungwook Yang
:::

## 原始参考来源 (Sources)

- [arXiv 2609.16504：UniDex-ViTac: Learning Unified Visuo-Tactile Dexterous Manipulation Policy from Human Video Data](https://arxiv.org/abs/2609.16504)