外观
论文arXiv 2609.070021 个月前
WM-Craftnet：用于可泛化且鲁棒的灵巧手内操作的 World Synesthesia Model
原文链接：WM-Craftnet: World Synesthesia Model for Generalizable and Robust Dexterous In-Hand Manipulation（arXiv 2609.07002 · 发布日期：2026-09-07）
WM-Craftnet: World Synesthesia Model for Generalizable and Robust Dexterous In-Hand Manipulation
针对噪声观测下手内操作鲁棒性下降，WM-Craftnet学习多模态动作条件世界模型，并在多物体旋转、扰动恢复和Sim-to-Real上验证。
核心要点
- 手内操作需从部分且带噪观测推断物体位姿、几何、接触与潜在滑移。
- 方法从本体、深度、触觉和动作学习潜动态，作为非对称actor-critic的循环任务上下文。
- 通过多模态重建、奖励预测和干净深度监督，提供用于真机部署的去噪几何状态。
- 消融显示预测世界建模、干净深度监督和触觉接触线索会塑造学习状态。
验证范围与局限：验证包括九个z轴物体预训练、49物体下游学习、多物体旋转、未见物体、扰动恢复和Sim-to-Real迁移。
英文摘要 (Abstract)
Generalizable and robust dexterous in-hand manipulation requires a policy to infer object pose, geometry, contact, and potential slip from partial and noisy observations. Although recent tactile and visuotactile RL methods achieve strong in-hand rotation in controlled settings, their robustness often degrades under pose shifts, force disturbances, and object variation. We propose WM-Craftnet, a world-model-conditioned framework that learns compact action-conditioned latent dynamics from proprioception, depth, tactile sensing, and actions, supervised by multimodal reconstruction and reward prediction. Rather than using the world model for latent imagination or policy optimization, WM-Craftnet uses the learned World Synesthesia Model (WSM) as recurrent task context for an asymmetric actor--critic policy. Importantly, WSM is trained to reconstruct clean depth targets from noisy depth inputs, providing a denoised geometric state for real-robot deployment. Ablations over recurrent baselines, auxiliary heads, tactile masking, and WSM modality heads show that predictive world modeling, clean-depth supervision, and tactile contact cues all shape the learned state. A WSM pretrained on nine (z)-axis objects serves as a reusable prior for (49)-object downstream policy learning. This context improves multi-object rotation, with quantitative and qualitative evidence for unseen-object, perturbation-recovery, and sim-to-real transfer.
作者：Jie Yin, Zeyuan Zhao, Tan, Xiaojing, Yang Liu, Chiyu Wang, Xinyang Gu
来源
本页事实对应的公开页面，以原文为准。