外观
论文arXiv 2610.0139927 天前
在 Proximal Policy Optimization 中复用过去样本：何时以及如何有帮助？
原文链接：Reusing Past Samples in Proximal Policy Optimization: When and How Does It Help?（arXiv 2610.01399 · 发布日期：2026-10-01）
Reusing Past Samples in Proximal Policy Optimization: When and How Does It Help?
本文研究PPO样本复用，提出多重重要性加权下的wPPO-U/BH，并在连续控制任务上验证其对效率与性能的影响。
核心要点
- PPO等同策略方法样本效率低，离策略回放虽高效但可能带来训练不稳定或大量调参。
- 作者在多重重要性加权框架下构建两种PPO变体，仅复用近期迭代窗口内的样本。
- wPPO-U使用普通重要性权重，wPPO-BH使用平衡启发式校正权重，并推导策略改进下界。
- 实验用于分析连续控制任务中，数据复用何时以及如何影响PPO样本效率或最终性能。
验证范围与局限：验证范围限于摘要所述连续控制任务，未说明真机或具体任务。
英文摘要 (Abstract)
Among on-policy deep reinforcement learning methods, Proximal Policy Optimization (PPO) has become the de facto standard, due to its consistently strong empirical performance across diverse application domains. However, on-policy methods are inherently sample inefficient: fresh data collected under the current policy is used for just a few updates before being discarded. Off-policy methods avoid this inefficiency via experience replay, achieving notable sample efficiency gains, but at the cost of training instabilities or extensive tuning. This motivated the rise of hybrid strategies that augment PPO with off-policy data reuse. Existing sample-reuse variants of PPO demonstrated improved sample efficiency over vanilla PPO, yet a systematic study of when reuse helps, in which scenarios, and to what extent remains missing. In this work, we study the effectiveness of sample reuse in PPO by instantiating two variants within a multiple importance weighting framework. Both retain the core PPO mechanics, reusing only samples from a window of recent iterations, thereby isolating the effect of data reuse from other factors. The variants, termed wPPO-U and wPPO-BH, employ vanilla importance weights or balance-heuristic-corrected ones, respectively. For both, we derive policy improvement lower bounds providing theoretical grounding for their respective losses. We use them to empirically study when and how data reuse improves sample efficiency or final performance of PPO across continuous control tasks.
作者：Alessandro Montenegro, Riccardo Venturelli, Marco Mussi, Matteo Papini, Alberto Maria Metelli
来源
本页事实对应的公开页面，以原文为准。