具身开发者社区

外观

论文arXiv 2609.342861 个月前

灵巧触觉世界模型 ​

针对操控世界模型难以从视觉捕捉接触事件，DTWM用双手手套触觉调制视频扩散Transformer，在自我中心操作未来帧预测中验证。

AI 整理
  • 数据与采集
  • 具身模型
  • 端侧部署

原文链接：Dexterous Tactile World Model（arXiv 2609.34286 · 发布日期：2026-09-28）

Dexterous Tactile World Model

针对操控世界模型难以从视觉捕捉接触事件，DTWM用双手手套触觉调制视频扩散Transformer，在自我中心操作未来帧预测中验证。

核心要点 ​

  • 操控进展常由接触建立与释放决定，但这些事件视觉难观察、触觉更易感知。
  • DTWM将双手手套触觉信号通过零初始化残差注入对应手部视频token。
  • 模型采用因果掩码避免预测帧访问未来信息，基于已观测视频与触觉预测未来帧。
  • 相较匹配的纯视觉模型，手部运动低估由23%降至9%，手区感知误差降7.4%。

验证范围与局限：验证范围为自我中心操控视频的未来帧预测，使用双手手套触觉信号；摘要未说明具体任务或真机设置。

英文摘要 (Abstract) ​

World models for manipulation are typically trained from video, yet the events that determine how manipulation unfolds, such as making and releasing contact, are difficult to observe visually and are often easier to sense through touch. We present the Dexterous Tactile World Model (DTWM), a video world model for future-frame prediction of egocentric manipulation from both observed video and tactile signals from a glove worn on each hand. We condition a pretrained video diffusion transformer on each hand's tactile signal through a zero-initialized residual at the corresponding hand location in the video tokens, while a causal mask prevents predicted frames from accessing future information. Compared with a vision-only model matched in architecture, parameters, and training, DTWM reduces the underestimation of hand motion from 23% to 9%, while reducing the perceptual error in the hand region by 7.4% across three training runs per model. The benefit also increases over the prediction horizon, with the improvement in the later predicted chunks being about 4.1x larger than in the first. DTWM also outperforms other visual-tactile world models under the same setting, and training with touch improves future-frame prediction even when no touch is available at inference. Ablations show that the model benefits from both the magnitude and spatial location of force: replacing the tactile signal with binary contact states, either per hand or per location, increases prediction error. The observed course of the force indicates whether the interaction will persist or change.

作者：Ziyao Zeng, Xiatao Sun, Hao Wang, Yueyang Pan, Zhengxiang Yu, Fengyu Yang, Tianyu Liu, Zhiwen Fan, Daniel Rakita

来源

本页事实对应的公开页面，以原文为准。

  1. arXiv 2609.34286：Dexterous Tactile World Model访问 2026-10-21