外观
论文arXiv 2606.228604 个月前
HiL-ResRL：VLA模型残差强化学习微调，1.5小时真机训练成功率超95%
本文由 AI 根据公开内容整理，可能有误，请以原文为准：HiL-ResRL: A Model-Agnostic Finetuning Adapter via Human-in-the-loop Residual Reinforcement Learning（arXiv 2606.22860）
核心内容
针对生成式模仿学习中行为克隆（BC）存在的复合误差与分布偏移问题，研究团队提出HiL-ResRL框架，旨在实现视觉-语言-动作（VLA）模型的鲁棒部署。该方案采用即插即用设计，不依赖特定模型架构，通过在真机环境进行在线强化学习微调，仅耗时1.5小时便将平均任务成功率提升至95%以上，验证了其在实际机器人操作中的有效性。
技术上将VLA生成的动作视为统一接口，训练一个残差策略网络来修正次优动作并补偿分布偏移，实现对各类VLA模型的适配。同时引入人在回路（Human-in-the-loop）引导机制，在保障真机探索安全性的前提下最大化样本效率。该方法有效克服了传统RL微调对模型结构的强耦合限制，为BC模型从实验室走向工业场景提供了标准化的工程化解决方案。
关键看点
- 残差策略架构：将VLA输出作为统一接口训练残差网络，修正次优动作并解决模仿学习固有的分布偏移问题
- 高效真机微调：在真实机器人环境中仅需1.5小时在线RL训练，平均任务成功率即突破95%
- 模型无关适配：采用即插即用设计解耦特定架构，支持多种VLA模型的通用化部署与安全探索
- 人在回路机制：通过人工引导干预保障真机强化学习探索安全性，显著提升训练样本效率
工程与产业观察
该工作为VLA模型落地提供了关键的“最后一公里”微调范式。通过残差策略与人在回路机制，将真机RL训练时间压缩至1.5小时级，大幅降低了具身智能算法从仿真/离线数据到工业现场部署的门槛与试错成本，对解决当前大模型机器人操作泛化性不足具有重要工程参考价值。
英文摘要 (Abstract)
Recent advancements in generative imitation learning have significantly propelled the field of robotic manipulation. However, the majority of existing models rely heavily on Behavior Cloning (BC), a paradigm that suffers from compounding errors and distributional shift. Consequently, the efficacy of these models in practical industrial deployments remains limited. To address these challenges, we introduce a novel, plug-and-play fine-tuning pipeline designed to facilitate the robust deployment of Vision-Language-Action (VLA) models in real-world environments. In contrast to contemporary reinforcement learning (RL) fine-tuning strategies, which are often constrained by specific model architectures, our proposed framework is model-agnostic and adaptable to a diverse range of VLA models. We conceptualize VLA-generated actions as a unified interface, upon which we train a residual policy. This policy is designed to rectify suboptimal actions and address the distributional shift inherent in imitation learning. Additionally, we incorporate human-in-the-loop guidance to ensure safe exploration and maximize training efficiency. We conduct experiments directly in real-world robotic settings. The results demonstrate that within only 1.5 hour of real-world online RL training, the average success rate exceeds 95% on real robots. Our work presents a practical solution for deploying behavior cloning models in industrial scenarios.
作者：Jingyi Liu, Zhaohong Mai, ShunSen He, Hang Ren, Chao Wang, Shunbo Zhou, XiaoDong Wu, Heng Zhang
来源
本页事实对应的公开页面，以原文为准。