论文arXiv 2611.0857613 天前
Q-OIL算法优化人机干预学习,双臂操作任务成功率提升50%
本文由 AI 根据公开内容整理,可能有误,请以原文为准:Q-OIL: Optimistic Exploration in Learning from Human Interventions(arXiv 2611.08576)
核心内容
arXiv论文指出传统机器人学习中的人类模型假设与真实干预行为存在偏差,实证分析显示人类多在智能体停滞时介入且动作常非最优。基于此提出的Q-OIL方法引入进度感知模型,在仿真基准测试中与真实用户算法排名相关性更高,为具身智能在线适应提供了更可靠的评估基础。
Q-OIL在强化学习中对干预转换添加乐观奖励以引导探索,并采用解耦Critic架构防止奖励向干预前状态回溯,确保训练稳定性。实验表明该方法学习效率显著优于竞争基线,在仿真和真实机器人双臂操作任务中成功率提升10%至50%,并能有效微调预训练视觉语言动作(VLA)模型,验证了其在物理系统中的泛化能力。
关键看点
- 干预建模:揭示人类干预具有滞后性与次优性特征,构建进度感知模型使仿真评测与真实用户排序高度一致
- 算法机制:对干预转换施加乐观奖励引导探索,利用解耦Critic阻断奖励回溯,解决非最优反馈导致的训练不稳定问题
- 性能指标:在仿真及真机双臂操作任务中,Q-OIL较主流基线方法成功率提升10%至50%,样本效率显著增强
- 模型适配:验证该方法可直接用于改进预训练VLA模型,为人机协同场景下的大模型具身微调提供标准化技术路径
工程与产业观察
该研究修正了具身智能领域对人类反馈的理想化假设,证实解耦Critic是处理次优干预数据的关键工程组件。其10-50%的真机性能增益表明,将人类行为学特征融入RL算法设计,比单纯追求理论最优更能解决VLA落地中的长尾适应难题,为低成本人机协同数据采集与模型迭代确立了新范式。
英文摘要 (Abstract)
As robots are deployed in the real world, they will inevitably make mistakes. Human interventions provide valuable feedback for improving their behavior, enabling them to adapt online to new scenarios. However, studying human-in-the-loop robot learning is difficult: real-world experiments are slow and expensive, so most algorithm development relies on simulated human models. We show that common human modeling assumptions do not match how real people intervene in practice. Through analyzing intervention data from human participants, we find that humans tend to intervene when agent progress stalls, and that their corrective actions are often suboptimal. Motivated by these findings, we introduce a progress-based model of human intervention that achieves higher rank correlation with real-user algorithm rankings, enabling realistic algorithm benchmarking and development in simulation. Building on these insights, we propose Q-OIL, a method that adds optimistic bonuses to intervention transitions in reinforcement learning and softly guides exploration toward human corrections. We find that decoupled critics are critical for stability because they prevent the bonuses from backing up to pre-intervention states. As a result, Q-OIL learns more efficiently than competitive baselines, achieving 10-50% higher success in simulation and on real robots, and improving pretrained VLAs on two bimanual manipulation tasks. As we ground both evaluation and algorithms in realistic human behavior, this work provides a practical path toward scalable robot learning from interventions.
作者:Pranav Teegavarapu, Sriyash Poddar, Rico Qi, Cleah Winston, Krisha Khandelwal, Aykut Onyol, Natasha Jaques, Abhishek Gupta
外部来源
本页事实对应的公开页面,以原文为准。(点击将离开本站)