外观
论文arXiv 2610.1978314 天前
通过 Advantage Guidance 进行强化学习
原文链接：Reinforcement Learning via Advantage Guidance（arXiv 2610.19783 · 发布日期：2026-10-14）
Reinforcement Learning via Advantage Guidance
本文提出 Advantage Guidance，以监督学习式优势条件训练和对比扩散引导优化策略，并在26个长时程稀疏奖励任务上验证。
核心要点
- 强化学习有望超越数据示范，但实践中常受不稳定和超参数敏感影响。
- 方法保留优势条件的监督学习结构，并在采样时使用对比扩散引导。
- 该方法提升模型响应条件优势信号的能力，并用于逼近KL正则化最优策略。
- 实验显示其在26个长时程稀疏奖励任务上显著优于既有最先进RL方法。
验证范围与局限：验证范围为26个长时程、稀疏奖励任务及不同最优性程度的数据集。
英文摘要 (Abstract)
Reinforcement learning (RL) holds the promise of learning a policy that can outperform what is shown in the data. However, in practice, such systems suffer from instability and hyperparameter sensitivity. In contrast, supervised learning provides a stable and scalable training objective, but is fundamentally limited by the quality of the demonstrated behavior. In this paper, we ask whether we can perform policy optimization with something as simple and stable as supervised learning but as performant as RL. Unfortunately, naively training the policy with supervised learning and conditioning on future rewards or advantages typically underperform dedicated RL methods. We introduce a new policy extraction method, Advantage Guidance, that retains the simple supervised learning structure with advantage conditioning while outperforming prior RL methods through effective conditional training and contrastive diffusion guidance during sampling. We show that Advantage Guidance can significantly improve a model's ability to respond to conditional advantage signals, and the contrastive guidance during sampling can use the learned advantage-conditioned policy to target the KL-regularized optimal policy. Empirically, we find that Advantage Guidance significantly outperforms prior state-of-the-art RL methods on 26 long-horizon, sparse reward tasks across datasets with varying degrees of optimality.
作者：Zhiyuan Zhou, Sergey Levine
来源
本页事实对应的公开页面，以原文为准。