论文HF Daily Papers · arXiv 2611.1116612 天前
DexForesight隐式空间预见框架提升灵巧操作成功率至60.5%
本文由 AI 根据公开内容整理,可能有误,请以原文为准:DexForesight: Grounding Dexterous Policies with Implicit Spatial Foresight(HF Daily Papers · arXiv 2611.11166)
核心内容
针对灵巧操作缺乏空间预见性问题,研究提出DexForesight框架,将动力学感知世界模型隐式嵌入策略潜空间。该方法在训练阶段利用离线场景深度、手部运动学及演示轨迹学习动作条件化世界模型,捕获包括遮挡手指在内的3D手部点云运动与视觉特征变化。在11项DexJoCo仿真任务中,整体成功率由基线49.4%提升至60.5%;在3项真机灵巧操作任务中达58.3%,优于最强基线50.0%。
技术核心在于冻结预训练世界模型后,通过对齐策略查询与预测性世界查询,将空间动力学前瞻能力迁移至策略潜变量以条件化动作生成。部署阶段仅保留策略网络,直接基于RGB图像、自然语言指令及机器人本体状态推理,完全摆脱对实时深度图输入及显式未来帧预测的依赖。这种隐式表征机制有效降低了灵巧手高维状态估计的计算负载,同时解决了视觉遮挡下的手指状态推断难题。
关键看点
- 隐式预见机制:将动作条件化世界模型冻结并对齐策略查询,把3D手部动力学与场景变化表征注入策略潜空间,实现无显式预测模块的空间前瞻
- 仿真性能突破:在11项DexJoCo基准测试中,整体任务成功率从基础策略的49.4%显著提升至60.5%,验证了隐式空间表征对复杂操作的有效性
- 真机泛化验证:在3项真实机器人灵巧操作任务中取得58.3%成功率,较最强对比基线高出8.3个百分点,证明Sim-to-Real迁移可行性
- 轻量化部署:推理端仅依赖RGB观测、语言指令与机器人状态,无需深度传感器输入或在线未来预测计算,大幅降低灵巧手系统硬件与算力门槛
工程与产业观察
该工作为具身灵巧操作提供了低算力部署范式,通过离线世界模型蒸馏替代在线显式预测,解决了遮挡手指状态估计痛点。其隐式表征思路可推广至其他高自由度本体控制,对缺乏高精度深度传感器的移动端灵巧作业具有重要工程参考价值。
英文摘要 (Abstract)
Dexterous manipulation requires rich spatial foresight over both the surrounding scene and the robot hand. Existing world-action models often acquire such predictive knowledge through explicit future prediction. We introduce DexForesight, a framework that learns spatial foresight implicitly by injecting dynamics-aware world representations into the latent space of a dexterous policy. During training, DexForesight learns an action-conditioned spatial world model from offline scene depth, kinematic hand geometry, and demonstration trajectories. The world model predicts 3D hand-point motion and visual feature changes, capturing both articulated hand dynamics and scene changes under demonstrated actions, including the motion of fingers occluded from the cameras. We then freeze the world model and align policy queries with its predictive world queries, transferring the learned spatial dynamics into policy latents and using them to condition action generation. At deployment, only the policy is retained, operating on RGB observations, language, and robot state without depth input or explicit future prediction. On 11 DexJoCo simulation tasks, DexForesight improves the overall success rate of the base policy from 49.4% to 60.5%. On three real-robot dexterous manipulation tasks, DexForesight achieves 58.3% success, compared with 50.0% for the strongest baseline. Project page https://dexforesight.github.io/
Hugging Face Daily Papers 热度:23 upvotes 作者:Zhongxi Chen, Yi Yang, Xinyi Chen, Mingzhe Ye, Yaping Li, Zhihong Liu, Congsheng Xu, Haonan Wang, Zhijie Deng
外部来源
本页事实对应的公开页面,以原文为准。(点击将离开本站)