# IRL-VLA：逆强化学习奖励模型驱动VLA闭环训练获NAVSIM v2 SOTA

> 原文链接：https://robotics.hellobike.com/papers/2025-08-07-2508-06571-d9ff0dec
> 频道：papers | 更新日期：2025-08-07 | 标签：auto, arxiv, cs.RO
> 核心摘要：针对VLA模型开环模仿学习性能受限及闭环仿真低效痛点，提出IRL-VLA框架。通过逆强化学习构建轻量级奖励世界模型并结合PPO优化，在NAVSIM v2端到端驾驶基准获SOTA，并斩获CVPR2025自动驾驶挑战赛亚军。

# IRL-VLA：逆强化学习奖励模型驱动VLA闭环训练获NAVSIM v2 SOTA

::: v-pre
> 本文由 AI 根据公开内容整理，可能有误，请以原文为准：[IRL-VLA: Training an Vision-Language-Action Policy via Reward World Model](https://arxiv.org/abs/2508.06571)（arXiv 2508.06571）

## 核心内容

针对现有视觉-语言-动作（VLA）模型依赖开环模仿学习导致策略次优，以及高保真闭环仿真存在域差距与计算瓶颈的问题，研究团队提出IRL-VLA框架。该方法在NAVSIM v2端到端驾驶基准测试中取得State-of-the-Art性能，并获得CVPR2025 Autonomous Grand Challenge亚军，验证了基于奖励世界模型的闭环强化学习在提升自动驾驶VLA策略有效性方面的突破。

技术架构采用三阶段范式：首先通过模仿学习预训练VLA策略；其次利用逆强化学习（IRL）构建轻量级奖励世界模型，替代高成本传感器仿真以实现高效闭环奖励计算；最后设计基于PPO的奖励引导强化学习算法，联合优化安全性、舒适性与通行效率。该方案有效解决了传统闭环训练中环境交互成本高及奖励函数设计难的问题，为具身智能体在复杂动态场景下的策略迭代提供了可迁移的低成本训练范式。

## 关键看点

- **三阶段训练范式**：依次执行VLA模仿学习预训练、IRL奖励世界模型构建及PPO强化学习微调，实现从行为克隆到价值对齐的闭环策略进化
- **轻量级奖励模型**：基于逆强化学习构建奖励世界模型替代高保真物理仿真，显著降低闭环训练算力开销并缓解Sim-to-Real域差距问题
- **多目标联合优化**：通过定制化奖励引导RL算法平衡安全事件、驾驶舒适度与交通效率，解决单一指标优化导致的策略偏科现象
- **权威基准验证**：在NAVSIM v2端到端驾驶基准达成SOTA，并获CVPR2025自动驾驶挑战赛亚军，证实算法在复杂长尾场景下的泛化能力

## 工程与产业观察

该工作证明了在具身智能训练中，利用逆强化学习从数据中提取隐式奖励函数可作为高保真仿真的低成本替代方案。对于机器人算法工程师而言，这种“预训练+奖励模型+RL”的三阶段范式为解决真机交互昂贵、仿真失真等共性难题提供了可复用的技术路径，尤其适用于缺乏精确物理建模的非结构化作业场景。

## 英文摘要 (Abstract)

Vision-Language-Action (VLA) models have demonstrated potential in autonomous driving. However, two critical challenges hinder their development: (1) Existing VLA architectures are typically based on imitation learning in open-loop setup which tends to capture the recorded behaviors in the dataset, leading to suboptimal and constrained performance, (2) Close-loop training relies heavily on high-fidelity sensor simulation, where domain gaps and computational inefficiencies pose significant barriers. In this paper, we introduce IRL-VLA, a novel close-loop Reinforcement Learning via \textbf{I}nverse \textbf{R}einforcement \textbf{L}earning reward world model with a self-built VLA approach. Our framework proceeds in a three-stage paradigm: In the first stage, we propose a VLA architecture and pretrain the VLA policy via imitation learning. In the second stage, we construct a lightweight reward world model via inverse reinforcement learning to enable efficient close-loop reward computation. To further enhance planning performance, finally, we design specialized reward world model guidence reinforcement learning via PPO(Proximal Policy Optimization) to effectively balance the safety incidents, comfortable driving, and traffic efficiency. Our approach achieves state-of-the-art performance in NAVSIM v2 end-to-end driving benchmark, 1st runner up in CVPR2025 Autonomous Grand Challenge. We hope that our framework will accelerate VLA research in close-loop autonomous driving.

作者：Anqing Jiang, Yu Gao, Yiru Wang, Zhigang Sun, Shuo Wang, Yuwen Heng, Hao Sun, Shichen Tang, Lijuan Zhu, Jinhao Chai, Jijun Wang, Zichong Gu, Hao Jiang, Li Sun
:::

## 原始参考来源 (Sources)

- [arXiv 2508.06571：IRL-VLA: Training an Vision-Language-Action Policy via Reward World Model](https://arxiv.org/abs/2508.06571)