# HiL-ResRL：VLA模型残差强化学习微调，1.5小时真机训练成功率超95%

> 原文链接：https://robotics.hellobike.com/papers/2026-06-22-2606-22860-4b7a9ecd
> 频道：papers | 更新日期：2026-06-22 | 标签：auto, arxiv, cs.RO
> 核心摘要：针对VLA模型行为克隆误差累积痛点，提出模型无关的残差强化学习微调框架。通过人机协同安全探索与动作修正机制，仅需1.5小时真机在线训练即可将任务成功率提升至95%以上，解决工业部署难题。

# HiL-ResRL：VLA模型残差强化学习微调，1.5小时真机训练成功率超95%

::: v-pre
> 本文由 AI 根据公开内容整理，可能有误，请以原文为准：[HiL-ResRL: A Model-Agnostic Finetuning Adapter via Human-in-the-loop Residual Reinforcement Learning](https://arxiv.org/abs/2606.22860)（arXiv 2606.22860）

## 核心内容

针对生成式模仿学习中行为克隆（BC）存在的复合误差与分布偏移问题，研究团队提出HiL-ResRL框架，旨在实现视觉-语言-动作（VLA）模型的鲁棒部署。该方案采用即插即用设计，不依赖特定模型架构，通过在真机环境进行在线强化学习微调，仅耗时1.5小时便将平均任务成功率提升至95%以上，验证了其在实际机器人操作中的有效性。

技术上将VLA生成的动作视为统一接口，训练一个残差策略网络来修正次优动作并补偿分布偏移，实现对各类VLA模型的适配。同时引入人在回路（Human-in-the-loop）引导机制，在保障真机探索安全性的前提下最大化样本效率。该方法有效克服了传统RL微调对模型结构的强耦合限制，为BC模型从实验室走向工业场景提供了标准化的工程化解决方案。

## 关键看点

- **残差策略架构**：将VLA输出作为统一接口训练残差网络，修正次优动作并解决模仿学习固有的分布偏移问题
- **高效真机微调**：在真实机器人环境中仅需1.5小时在线RL训练，平均任务成功率即突破95%
- **模型无关适配**：采用即插即用设计解耦特定架构，支持多种VLA模型的通用化部署与安全探索
- **人在回路机制**：通过人工引导干预保障真机强化学习探索安全性，显著提升训练样本效率

## 工程与产业观察

该工作为VLA模型落地提供了关键的“最后一公里”微调范式。通过残差策略与人在回路机制，将真机RL训练时间压缩至1.5小时级，大幅降低了具身智能算法从仿真/离线数据到工业现场部署的门槛与试错成本，对解决当前大模型机器人操作泛化性不足具有重要工程参考价值。

## 英文摘要 (Abstract)

Recent advancements in generative imitation learning have significantly propelled the field of robotic manipulation. However, the majority of existing models rely heavily on Behavior Cloning (BC), a paradigm that suffers from compounding errors and distributional shift. Consequently, the efficacy of these models in practical industrial deployments remains limited. To address these challenges, we introduce a novel, plug-and-play fine-tuning pipeline designed to facilitate the robust deployment of Vision-Language-Action (VLA) models in real-world environments. In contrast to contemporary reinforcement learning (RL) fine-tuning strategies, which are often constrained by specific model architectures, our proposed framework is model-agnostic and adaptable to a diverse range of VLA models. We conceptualize VLA-generated actions as a unified interface, upon which we train a residual policy. This policy is designed to rectify suboptimal actions and address the distributional shift inherent in imitation learning. Additionally, we incorporate human-in-the-loop guidance to ensure safe exploration and maximize training efficiency. We conduct experiments directly in real-world robotic settings. The results demonstrate that within only 1.5 hour of real-world online RL training, the average success rate exceeds 95% on real robots. Our work presents a practical solution for deploying behavior cloning models in industrial scenarios.

作者：Jingyi Liu, Zhaohong Mai, ShunSen He, Hang Ren, Chao Wang, Shunbo Zhou, XiaoDong Wu, Heng Zhang
:::

## 原始参考来源 (Sources)

- [arXiv 2606.22860：HiL-ResRL: A Model-Agnostic Finetuning Adapter via Human-in-the-loop Residual Reinforcement Learning](https://arxiv.org/abs/2606.22860)