外观
论文arXiv 2609.3901728 天前
OccluDex：用于自遮挡下自我中心灵巧操作的层次化3D视觉-触觉表征学习
原文链接：OccluDex: Hierarchical 3D Visuo-Tactile Representation Learning for Egocentric Dexterous Manipulation under Self-Occlusion（arXiv 2609.39017 · 发布日期：2026-09-30）
OccluDex: Hierarchical 3D Visuo-Tactile Representation Learning for Egocentric Dexterous Manipulation under Self-Occlusion
OccluDex融合多尺度3D几何与触觉接触表征，在仿真和Shadow Hand实机上验证自遮挡下灵巧操作。
核心要点
- 自我中心感知中，操作手会遮挡物体表面和接触区域，影响状态估计、闭环控制与未见几何泛化。
- 方法采用多尺度掩码自编码编码部分3D几何，并用跨模态注意力融合触觉接触token与高层几何特征。
- 编码器由同步人类视觉-触觉示教预训练，并作为冻结感知骨干迁移到下游强化学习。
- 仿真中在水龙头旋转和桌面物体重定向任务上，相比最强基线未见物体准确率高12.6%、已见物体高8.3%。
验证范围与局限：验证范围包括仿真中的水龙头整圈顺时针旋转、桌面物体180度重定向，以及Shadow Hand实机未见物体零样本Sim-to-Real。
英文摘要 (Abstract)
Reliable dexterous manipulation requires continuous estimation of object geometry and hand-object contact throughout interaction. With egocentric sensing, however, the manipulating hand frequently occludes task-relevant object surfaces and contact regions, reducing the visual evidence available for state estimation and thereby making robust closed-loop control and generalization to unseen object geometries particularly challenging. To address this, we present OccluDex, a hierarchical 3D visuo-tactile representation learning framework that integrates global geometric structure with local contact information for robust manipulation under dynamic self-occlusion during hand-object interaction. OccluDex adopts multi-scale masked autoencoding to progressively encode partial 3D geometry and fuses tactile contact tokens with high-level geometric features through cross-modal attention. The encoder is pretrained from synchronized human visuo-tactile demonstrations and transferred as a frozen perceptual backbone for downstream reinforcement learning. We evaluate OccluDex on a faucet rotation task, requiring one full clockwise handle revolution, and a tabletop object reorientation task, requiring a 180-degree tabletop object reorientation without toppling. In simulation experiments, OccluDex demonstrated 12.6% higher accuracy for unseen objects and 8.3% higher accuracy for previously seen objects than the strongest state-of-the-art baseline models. Physical experiments were further performed with a Shadow Hand to demonstrate successful zero-shot sim-to-real generalization on unseen physical objects. This results could enable humanoid egocentric object manipulation for seen and unseen objects even when the manipulating robotic hand occludes vision.
作者：Ziheng Xu, Yueyuan Chen, Xinyuan He, Guoxing Liu, Y. H. Tan, Huiming Pan, Bin He, Shuo Jiang, Peter Bradley Shull
来源
本页事实对应的公开页面，以原文为准。