Skip to content

论文arXiv 2611.219084 天前

PriDex框架:置信度引导RL将人类视频转为双臂灵巧操作数据 ​

本文由 AI 根据公开内容整理,可能有误,请以原文为准:From Human Videos to Bimanual Dexterous Manipulation Data: Confidence-Guided RL Refinement of Video Priors(arXiv 2611.21908)

核心内容 ​

针对双臂灵巧操作示教成本高昂及人体运动直接迁移困难的问题,研究提出PriDex框架,旨在将人类视频先验转化为机器人可执行的灵巧操作演示数据。该方法在仿真环境中利用强化学习对视频提取的运动进行精细化重构,保留了任务关键的交互结构并适配机器人本体特性。实验覆盖倒水、扫入簸箕、拧瓶盖及擦盘子四项双臂任务,验证了其在生成高质量仿真数据及训练下游视觉运动策略方面的有效性。

技术上,PriDex利用人-物交互提供功能可供性线索以筛选抓取姿态,并结合物体轨迹与相对双手腕轨迹作为双重运动引导。针对遮挡导致的估计不确定性,系统引入物体运动置信度机制,高置信度时侧重物体级引导,低置信度时自动增加手腕级引导权重。强化学习在满足机器人运动学与物理约束的前提下优化上述先验,生成的策略轨迹最终用于训练基于视觉的扩散策略(Diffusion Policy),实现了从非结构化视频到具身智能数据的闭环转化。

关键看点 ​

  • 置信度自适应机制:依据物体运动估计置信度动态调整物体轨迹与手腕轨迹的引导权重,解决视频遮挡与视觉模糊导致的先验不可靠问题
  • 双重轨迹引导架构:结合物体级任务参考与相对双手腕协调引导,在保留任务交互结构的同时适配机器人特定的运动学约束
  • 功能可供性抓取筛选:利用视频中的人-物交互信息作为Affordance线索,从几何候选集中筛选出符合任务语义的机器人专用抓取姿态
  • 下游策略验证:在倒水、拧盖等4项双臂任务中生成仿真演示数据,成功用于训练Vision-based Diffusion Policy并完成端到端性能评估

工程与产业观察 ​

该工作为具身智能提供了从海量互联网人类视频低成本获取双臂灵巧操作数据的技术路径。其核心的置信度加权多模态先验融合机制,有效解决了Sim-to-Real数据生成中的分布偏移与噪声干扰问题,对构建大规模通用操作基础模型的数据飞轮具有重要工程参考价值。

英文摘要 (Abstract) ​

Collecting demonstrations for bimanual dexterous manipulation is costly, motivating the use of human videos as a source of robot training data. However, uncertainty in video-derived motion estimates and differences between human and robot embodiments make direct motion transfer challenging. In this work, we introduce PriDex, a framework that transforms human-video Priors into executable robot Dexterous manipulation demonstrations through reinforcement learning in simulation. PriDex preserves task-relevant interaction structure while allowing embodiment-specific behavior to be regenerated for the robot. Human-object interactions provide affordance cues for selecting task-relevant, robot-specific grasps from geometry-based candidates. For motion refinement, object trajectories provide a task-level reference, while relative bimanual wrist trajectories offer complementary guidance for coordinated motion. Since object-motion estimates can become unreliable under occlusion or visual ambiguity, PriDex uses object-motion confidence to adaptively balance these two priors, emphasizing object-level guidance when confidence is high and increasing the contribution of wrist-level guidance as confidence decreases. Reinforcement learning refines these video-derived priors under the robot's kinematic and physical constraints, and the resulting policies are rolled out to generate robot demonstrations for training a vision-based diffusion policy. We evaluate PriDex on four bimanual dexterous manipulation tasks, including pouring, sweeping into a dustpan, bottle-cap unscrewing, and plate cleaning, assessing both simulation-based robot data generation and downstream visuomotor policy learning.

作者:Yanghong Liu, Yang Chen, Kailang Xu, Bang Du, Feiyang Yang, Jiawei Xia, Zhengtao Yao, Yubiao Zhang, Masayoshi Tomizuka, Ruihai Wu, Wei Zhan

基于本页技术方向关联的整机参数、核心零部件、实测报告与同赛道选型横评。

外部来源

本页事实对应的公开页面,以原文为准。(点击将离开本站)

  1. arXiv 2611.21908:From Human Videos to Bimanual Dexterous Manipulation Data: Confidence-Guided RL Refinement of Video Priors ↗访问 2026-11-18