外观
论文arXiv 2609.249761 个月前
DexTacWAM：用于灵巧操作的视觉-触觉 World-Action Model
原文链接：DexTacWAM: A Visuo-Tactile World-Action Model for Dexterous Manipulation（arXiv 2609.24976 · 发布日期：2026-09-21）
DexTacWAM: A Visuo-Tactile World-Action Model for Dexterous Manipulation
DexTacWAM将指尖触觉潜变量注入视频扩散世界模型，在六个接触丰富的灵巧操作任务上验证并优于基线。
核心要点
- 视觉中心的WAM难以直接建模视觉部分可观测的接触动力学。
- 方法独立编码各指尖，经手指与位姿感知压缩器聚合并注入视频扩散世界模型。
- 六个任务均获最高分，平均70.6，高于最强基线38.0。
- 持续视觉到触觉学习用约100次示范适配，压缩器加快训练和推理。
验证范围与局限：验证范围为22-DoF双臂平台上的六个接触丰富灵巧操作任务。
英文摘要 (Abstract)
Dexterous manipulation depends on contact dynamics that are often only partially observable from vision. Recent World-Action Models (WAMs) couple predictive video world modeling with action generation, but remain largely vision-centric and therefore cannot directly model these contact dynamics. We present DexTacWAM, a visuo-tactile WAM that encodes each fingertip independently, aggregates the resulting features through a finger- and pose-aware tactile compressor, and injects the tactile latent into a video diffusion world model for joint visuo-tactile world modeling. Across six contact-rich dexterous manipulation tasks on a 22-DoF bimanual platform, DexTacWAM achieves the highest score on every task, averaging 70.6 versus 38.0 for the strongest baseline. Ablations attribute the gain to modeling contact evolution as part of the predicted world state rather than tactile conditioning alone: removing tactile world modeling reduces the four-task mean from 74.7 to 26.6 while keeping the same tactile features and action expert. After four hours of tactile-encoder adaptation with a frozen pretrained vision VAE, our continual vision-to-touch learning extends the pretrained video model to touch using roughly 100 demonstrations per task without tactile midtraining, while retaining visual prediction quality within 0.5 dB of vision-only counterparts. The compressor retains 89.4% of pre-fusion contact recall while enabling 2.26x faster training and 1.29x faster inference. Together, these results show that pretrained video priors can be extended to distributed multi-finger contact dynamics in a data- and compute-efficient manner.
作者：Haoran Yuan, Zekai Wang, Boning Shao, Haoran Lu, Trevor Jackson Darrell, Ismini Lourentzou, Wei Zhan
来源
本页事实对应的公开页面，以原文为准。