外观
论文arXiv 2511.2063311 个月前
Prophet世界模型与FlowScale强化VLA策略，真机成功率提升30%
本文由 AI 根据公开内容整理，可能有误，请以原文为准：Reinforcing Action Policies by Prophesying（arXiv 2511.20633）
核心内容
针对视觉-语言-动作（VLA）模型纯模仿学习导致的分布偏移与脆弱性问题，研究提出基于世界模型的强化学习后训练方案。该方案包含统一动作到视频的世界模型Prophet及适配流匹配动作头的FlowScale算法，旨在解决真机交互昂贵与传统仿真器迁移难的问题，为VLA提供数据与计算高效的后训练路径。
Prophet利用大规模异构机器人数据预训练动作-结果动态，经少样本适配即可作为新环境的Rollout仿真器；FlowScale耦合Flow-GRPO与内在逐步重加权机制以稳定梯度优化。实验验证表明，该方法兼容多种VLA骨干网络，在公开基准测试中成功率提升5-17%，在真实机器人部署中成功率提升24-30%，显著增强了策略鲁棒性。
关键看点
- Prophet世界模型：基于异构数据预训练动作到视频生成能力，支持少样本适配新机器人与环境，提供低成本Rollout仿真
- FlowScale算法：结合Flow-GRPO与内在逐步重加权机制，解决流匹配动作头在RL后训练中的梯度不稳定问题
- 真机性能验证：跨多种VLA骨干网络测试，公开基准成功率提升5-17%，真实机器人任务成功率提升24-30%
工程与产业观察
该工作验证了“通用世界模型+流匹配RL”是解决VLA真机泛化瓶颈的有效范式。Prophet降低了高保真仿真构建门槛，FlowScale解决了生成式动作头的优化难题。这对具身智能从离线模仿迈向在线自我进化具有关键工程参考价值，指明了利用合成数据提升真机鲁棒性的技术方向。
英文摘要 (Abstract)
Vision-Language-Action (VLA) policies excel in aligning language, perception, and robot control. However, most VLAs are trained purely by imitation, which overfits to demonstrations, and is brittle under distribution shift. Reinforcement learning (RL) directly optimizes task reward and thus addresses this misalignment, but real-robot interaction is expensive and conventional simulators are hard to engineer and transfer. We address both data efficiency and optimization stability in VLA post-training via a learned world model and an RL procedure tailored to flow-based action heads. Specifically, we first introduce Prophet, a unified action-to-video robot world model pretrained on large-scale, heterogeneous robot data to learn reusable action-outcome dynamics and then few-shot adapted to new robots, objects, and environments, yielding a rollout-ready simulator. Upon Prophet, we reinforce action policies with our proposed FlowScale, which couples Flow-GRPO with intrinsic stepwise reweighting to stabilize gradients. Together, our solution provides a practical, data- and compute-efficient path to VLA post-training. Experiments show 5-17% success gains on public benchmarks and 24-30% on real robots across diverse VLA backbones.
作者：Jiahui Zhang, Ze Huang, Chun Gu, Zipei Ma, Li Zhang
来源
本页事实对应的公开页面，以原文为准。