外观
论文arXiv 2609.309591 个月前
VisTacAlign：使用视觉-触觉对齐在触觉人类和机器人演示上协同训练灵巧策略
原文链接：VisTacAlign: Co-Training Dexterous Policies on Tactile Human and Robot Demonstrations using Visual-Tactile Alignment（arXiv 2609.30959 · 发布日期：2026-09-25）
VisTacAlign: Co-Training Dexterous Policies on Tactile Human and Robot Demonstrations using Visual-Tactile Alignment
VisTacAlign通过视觉与触觉对齐，将人类和机器人演示共同用于3D视觉触觉灵巧策略训练，并在三项真实任务上验证。
核心要点
- 人类演示成本低，但用于机器人策略共训时，需要弥合策略所用各模态的人机差距。
- 框架将手套跟踪的人手运动重定向到17-DoF触觉机器人手，并对双目视觉进行机器人手替换与点云重算。
- 电容触觉手套在信号空间对齐到机器人指尖传感器，扩散Transformer消费点云、本体和逐指触觉token。
- 在三项需精确力的真实任务中，加入对齐人类演示优于仅用机器人数据；消融显示触觉输入和视觉对齐均必要。
验证范围与局限：验证范围为三个真实世界任务：乐高装配、采摘不同大小草莓，以及启动并提起电钻。
英文摘要 (Abstract)
Human demonstrations are a cheap source of data for dexterous manipulation, but co-training a robot policy on them requires closing the human--robot gap in every modality the policy consumes. We present VisTacAlign, a framework for co-training 3D-visual-tactile dexterous policies on human and robot demonstrations. Glove-tracked human hand motion is retargeted to a 17-DoF tactile robot hand with a one-time fingertip correction. The human hand is then erased from both stereo views and replaced by a posed robot-hand mesh painted with pixels from robot recordings, and a real-time stereo foundation model is re-run on the composite, so the human point clouds carry the same stereo errors and visibility as the robot ones. Finally, a capacitive tactile glove is aligned to the robot's fingertip sensors in its signal space, giving one interpretable per-finger force representation. A diffusion transformer consumes point-cloud, proprioceptive, and per-finger tactile tokens. On three real-world tasks requiring precise force -- Lego assembly, plucking strawberries of varying size, and activating and lifting a power drill -- adding aligned human demonstrations to existing robot data improves over robot-only policies, and ablations show that both tactile input and visual alignment are necessary. Project page: https://vis-tac-align.github.io
作者：Julien Poffet, Matthew Strong, Ankush Dhawan, Baiyu Shi, Shalika Neelaveni, Yujia Yuan, Zhenan Bao, Monroe Kennedy
来源
本页事实对应的公开页面，以原文为准。