外观
论文arXiv 2609.100501 个月前
面向通用灵巧控制器的生成视频计划仿真落地
原文链接：Grounding Generated Video Plans in Simulation Towards Versatile Dexterous Controllers（arXiv 2609.10050 · 发布日期：2026-09-09）
Grounding Generated Video Plans in Simulation Towards Versatile Dexterous Controllers
将生成HOI视频与仿真落地结合，训练多物体多轨迹跟踪器，并在仿真和真实闭环操控中验证。
核心要点
- 生成HOI视频可提出操控动作，但仿真跟踪扩展性受可靠参考动作不足限制。
- 训练时用生成视频提供多样参考，部署时由视频模型生成计划并由跟踪器执行。
- 通过少量人工干预的HOI重建，在仿真中落地1500多个视频，成功率高基线25个百分点以上。
- 真实闭环实验实现多样抓取、非抓取操控和抓取后物体姿态跟踪，视频与代码已公开。
验证范围与局限：验证包括仿真中1500多个生成视频，以及真实闭环的多样抓取、非抓取操控和抓取后物体姿态跟踪。
英文摘要 (Abstract)
Generated hand-object interaction (HOI) videos provide a controllable way to propose manipulation motions. Simulation-based HOI tracking can translate such kinematic references into feasible low-level control, but its scalability is limited by the lack of reliable reference motions. We therefore combine generated videos with simulation-based HOI grounding: during training, generated videos provide diverse motion references for learning a multi-object, multi-trajectory HOI tracker, and at deployment, the video model produces motion plans that are executed by the learned tracker. In particular, we propose a method that enables scalable reference generation by HOI reconstruction with minimal manual intervention and successfully grounds more than 1,500 generated videos in simulation, achieving success rates over 25 percentage points higher than those of baselines during simulation-based training. In real-world closed-loop experiments, it achieves diverse grasps, including functional grasps, non-prehensile manipulation, and post-grasp object-pose tracking. Videos and code are available at https://boyuan-an.github.io/GALATEA/.
作者：Tianyue Wu, Boyuan An, Shuqi Zhao, Heyu Guo, Wanli Xing, Yi Ma, Kaifeng Zhang, Ruihai Wu, Masayoshi Tomizuka
来源
本页事实对应的公开页面，以原文为准。