# WM-Craftnet：用于可泛化且鲁棒的灵巧手内操作的 World Synesthesia Model

> 原文链接：https://robotics.hellobike.com/papers/2026-09-07-2609-07002
> 频道：papers | 更新日期：2026-09-07 | 标签：arxiv, cs.RO
> 核心摘要：针对噪声观测下手内操作鲁棒性下降，WM-Craftnet学习多模态动作条件世界模型，并在多物体旋转、扰动恢复和Sim-to-Real上验证。

# WM-Craftnet：用于可泛化且鲁棒的灵巧手内操作的 World Synesthesia Model

::: v-pre
> 原文链接：[WM-Craftnet: World Synesthesia Model for Generalizable and Robust Dexterous In-Hand Manipulation](https://arxiv.org/abs/2609.07002)（arXiv 2609.07002 · 发布日期：2026-09-07）

**WM-Craftnet: World Synesthesia Model for Generalizable and Robust Dexterous In-Hand Manipulation**

针对噪声观测下手内操作鲁棒性下降，WM-Craftnet学习多模态动作条件世界模型，并在多物体旋转、扰动恢复和Sim-to-Real上验证。

## 核心要点

- 手内操作需从部分且带噪观测推断物体位姿、几何、接触与潜在滑移。
- 方法从本体、深度、触觉和动作学习潜动态，作为非对称actor-critic的循环任务上下文。
- 通过多模态重建、奖励预测和干净深度监督，提供用于真机部署的去噪几何状态。
- 消融显示预测世界建模、干净深度监督和触觉接触线索会塑造学习状态。

**验证范围与局限**：验证包括九个z轴物体预训练、49物体下游学习、多物体旋转、未见物体、扰动恢复和Sim-to-Real迁移。

## 英文摘要 (Abstract)

Generalizable and robust dexterous in-hand manipulation requires a policy to infer object pose, geometry, contact, and potential slip from partial and noisy observations. Although recent tactile and visuotactile RL methods achieve strong in-hand rotation in controlled settings, their robustness often degrades under pose shifts, force disturbances, and object variation. We propose WM-Craftnet, a world-model-conditioned framework that learns compact action-conditioned latent dynamics from proprioception, depth, tactile sensing, and actions, supervised by multimodal reconstruction and reward prediction. Rather than using the world model for latent imagination or policy optimization, WM-Craftnet uses the learned World Synesthesia Model (WSM) as recurrent task context for an asymmetric actor--critic policy. Importantly, WSM is trained to reconstruct clean depth targets from noisy depth inputs, providing a denoised geometric state for real-robot deployment. Ablations over recurrent baselines, auxiliary heads, tactile masking, and WSM modality heads show that predictive world modeling, clean-depth supervision, and tactile contact cues all shape the learned state. A WSM pretrained on nine \(z\)-axis objects serves as a reusable prior for \(49\)-object downstream policy learning. This context improves multi-object rotation, with quantitative and qualitative evidence for unseen-object, perturbation-recovery, and sim-to-real transfer.

作者：Jie Yin, Zeyuan Zhao, Tan, Xiaojing, Yang Liu, Chiyu Wang, Xinyang Gu
:::

## 原始参考来源 (Sources)

- [arXiv 2609.07002：WM-Craftnet: World Synesthesia Model for Generalizable and Robust Dexterous In-Hand Manipulation](https://arxiv.org/abs/2609.07002)