外观
论文arXiv 2608.212902 个月前
VT-MUSE：用于操作的多模态统一序列视觉触觉表征学习
原文链接：VT-MUSE: Multimodal Unified Sequential Visuotactile Representation Learning for Manipulation（arXiv 2608.21290 · 发布日期：2026-08-21）
VT-MUSE: Multimodal Unified Sequential Visuotactile Representation Learning for Manipulation
VT-MUSE通过两阶段视觉触觉序列表征学习与门控交叉注意力策略，面向操作任务在仿真基准和真实实验中验证。
核心要点
- 现有方法常独立编码视觉与触觉，且忽略接触随时间演化。
- 第一阶段用跨模态时间对齐和掩码视图一致性联合适配各模态编码器。
- 第二阶段以条件变分潜变量模型处理掩码视觉序列和完整触觉历史。
- 在仿真基准所有任务上较最强基线高11个百分点，并在真实实验中提升。
验证范围与局限：仅说明在仿真基准和真实世界实验中验证，未给出具体任务、平台或规模。
英文摘要 (Abstract)
We propose VT-MUSE, a Multimodal Unified SEquential representation learning framework for visuotactilemanipulation. Existing approaches often encode visual and tactile observations independently before fusion, limiting their ability to capture fine-grained cross-modal dependencies. Moreover, most methods focus on observations at the current time step and overlook the temporal evolution of contact. VT-MUSE addresses both limitations through a two-stage representation learning framework. In Stage I, modality specific encoders are jointly adapted via cross-modal temporal alignment and masked-view consistency. In Stage II, a conditional variational latent model processes masked visual sequences together with full tactile histories. Auxiliary decoders reconstruct the masked recent visual observations and predict tactile depth changes, encouraging the latent representation to retain both global visual context and local contact dynamics. The learned representation is subsequently integrated into a lightweight Transformer policy through gated cross-attention. On the simulation benchmark, VT-MUSE outperforms the strongest baseline evaluated on all tasks by 11 percentage points and also achieves substantial improvements in real-world experiments.
作者：Congsheng Xu, Qiaochu Yang, Fangyuan Shi, Yifan Han, Baijun Chen, Yiming Wang, Haonan Zhao, Zhe Liu, Yao Mu, Daolin Ma
来源
本页事实对应的公开页面，以原文为准。