外观
论文arXiv 2510.004062025-10-01
VLA-RFT：世界模型强化微调400步超越VLA监督基线
本文由 AI 根据公开内容整理，可能有误，请以原文为准：VLA-RFT: Vision-Language-Action Reinforcement Fine-tuning with Verified Rewards in World Simulators（arXiv 2510.00406）
核心内容
针对视觉-语言-动作（VLA）模型依赖模仿学习导致的复合误差与分布偏移问题，研究团队提出VLA-RFT强化微调框架。该方法利用真实交互数据训练可控世界模拟器，通过动作条件化视觉观测预测实现策略推演，并基于目标达成参考生成轨迹级密集奖励信号。实验表明，该框架在少于400个微调步骤下即可超越强监督学习基线，且在扰动条件下保持稳定任务执行，确立了基于世界模型的RFT作为VLA后训练范式的有效性。
技术核心在于构建数据驱动的世界模型作为可控模拟器，替代高成本真机交互或存在Sim-to-Real Gap的传统仿真器。系统根据当前动作预测未来视觉观测，使策略能在模拟环境中进行Rollout并获得与动作对齐的学习信号，大幅降低样本需求。相比传统基于模拟器的强化学习，VLA-RFT不仅训练效率更高，还有效缓解了因环境差异导致的策略失效问题，为提升具身智能体在开放动态环境中的泛化能力提供了低资源消耗的工程解法。
关键看点
- 极低样本消耗：仅需少于400步微调即可超越强监督基线，显著降低VLA后训练算力与数据门槛
- 世界模型驱动：基于真实数据训练动作条件视觉预测器，提供无Sim-to-Real Gap的可控模拟环境
- 密集奖励机制：从目标达成参考中导出轨迹级奖励，解决稀疏反馈难题并提供动作对齐学习信号
- 抗扰动鲁棒性：在环境扰动条件下维持稳定任务执行，验证了强化微调对分布偏移的修正能力
工程与产业观察
VLA-RFT证明了“世界模型+强化微调”是解决具身大模型落地长尾问题的关键路径。对于算法工程师，该方案将RL从昂贵的真机迭代转化为高效的模型内推演，极大降低了VLA从Demo走向鲁棒部署的边际成本；其400步收敛的特性，使其成为现有VLA基座模型低成本增强泛化能力的标准化后处理工具。
英文摘要 (Abstract)
Vision-Language-Action (VLA) models enable embodied decision-making but rely heavily on imitation learning, leading to compounding errors and poor robustness under distribution shift. Reinforcement learning (RL) can mitigate these issues yet typically demands costly real-world interactions or suffers from sim-to-real gaps. We introduce VLA-RFT, a reinforcement fine-tuning framework that leverages a data-driven world model as a controllable simulator. Trained from real interaction data, the simulator predicts future visual observations conditioned on actions, allowing policy rollouts with dense, trajectory-level rewards derived from goal-achieving references. This design delivers an efficient and action-aligned learning signal, drastically lowering sample requirements. With fewer than 400 fine-tuning steps, VLA-RFT surpasses strong supervised baselines and achieves greater efficiency than simulator-based RL. Moreover, it exhibits strong robustness under perturbed conditions, sustaining stable task execution. Our results establish world-model-based RFT as a practical post-training paradigm to enhance the generalization and robustness of VLA models. For more details, please refer to https://vla-rft.github.io/.
作者：Hengtao Li, Pengxiang Ding, Runze Suo, Yihao Wang, Zirui Ge, Dongyuan Zang, Kexian Yu, Mingyang Sun, Hongyin Zhang, Donglin Wang, Weihua Su
来源
本页事实对应的公开页面，以原文为准。