外观
论文arXiv 2508.022192025-08-04
CO-RFT：分块离线RL微调VLA模型，30样本提升57%成功率
本文由 AI 根据公开内容整理，可能有误，请以原文为准：CO-RFT: Efficient Fine-Tuning of Vision-Language-Action Models through Chunked Offline Reinforcement Learning（arXiv 2508.02219）
核心内容
针对视觉-语言-动作（VLA）模型在机器人控制中面临的样本效率低、动作分块兼容差及训练不稳定问题，研究团队提出CO-RFT算法。该方法专为VLA设计，通过全参数模仿学习初始化策略，再利用分块离线强化学习进行优化。在真实机器人环境中，仅使用30至60条少量演示数据，CO-RFT相比传统监督学习方法实现了57%的任务成功率提升和22.3%的循环时间缩减。
技术核心在于扩展时序差分（TD）学习以适配VLA的动作分块特性，构建Chunked RL框架。算法流程上，先通过全参数微调完成骨干网络与策略的IL初始化，随后引入动作分块机制执行离线RL策略优化。实验验证显示，该方法具备显著的位置泛化能力，在训练集未覆盖的新位置仍保持44.3%的成功率，有效解决了小样本条件下VLA模型从仿真到真机部署的鲁棒性瓶颈。
关键看点
- 小样本高效微调：仅需30至60条演示数据即可完成VLA模型优化，大幅降低具身智能数据采集成本与门槛
- 性能显著提升：真实环境测试中任务成功率较监督学习提升57%，操作循环时间缩短22.3%
- 分块RL架构创新：扩展时序差分学习以原生支持动作分块，解决VLA模型与强化学习的兼容性难题
- 强位置泛化能力：在未见过的新位置仍达44.3%成功率，验证了离线RL对空间鲁棒性的增强效果
工程与产业观察
CO-RFT证明了在极少样本下通过离线RL微调VLA模型的可行性，其动作分块TD扩展机制为具身大模型落地提供了关键工程范式。对于缺乏海量真机数据的研发团队，该方案显著降低了从预训练模型到实际部署的边际成本，是解决VLA长尾场景适配与操作效率优化的重要参考路径。
英文摘要 (Abstract)
Vision-Language-Action (VLA) models demonstrate significant potential for developing generalized policies in real-world robotic control. This progress inspires researchers to explore fine-tuning these models with Reinforcement Learning (RL). However, fine-tuning VLA models with RL still faces challenges related to sample efficiency, compatibility with action chunking, and training stability. To address these challenges, we explore the fine-tuning of VLA models through offline reinforcement learning incorporating action chunking. In this work, we propose Chunked RL, a novel reinforcement learning framework specifically designed for VLA models. Within this framework, we extend temporal difference (TD) learning to incorporate action chunking, a prominent characteristic of VLA models. Building upon this framework, we propose CO-RFT, an algorithm aimed at fine-tuning VLA models using a limited set of demonstrations (30 to 60 samples). Specifically, we first conduct imitation learning (IL) with full parameter fine-tuning to initialize both the backbone and the policy. Subsequently, we implement offline RL with action chunking to optimize the pretrained policy. Our empirical results in real-world environments demonstrate that CO-RFT outperforms previous supervised methods, achieving a 57% improvement in success rate and a 22.3% reduction in cycle time. Moreover, our method exhibits robust positional generalization capabilities, attaining a success rate of 44.3% in previously unseen positions.
作者：Dongchi Huang, Zhirui Fang, Tianle Zhang, Yihang Li, Lin Zhao, Chunhe Xia
来源
本页事实对应的公开页面，以原文为准。