外观
论文arXiv 2610.0601923 天前
用于高效 On-Policy 强化学习的自适应专家指导
原文链接：Adaptive Expert Guidance for Efficient On-Policy Reinforcement Learning（arXiv 2610.06019 · 发布日期：2026-10-05）
Adaptive Expert Guidance for Efficient On-Policy Reinforcement Learning
提出在训练回合内让学习器与专家交替控制并联合优化专家占比，在34个任务上提升on-policy强化学习样本效率。
核心要点
- 从零学习样本效率低，难以利用启发式、模型控制器或相关任务策略等次优专家。
- 方法将专家控制占比作为单一可学习参数，与策略使用同一on-policy目标联合优化。
- 随着学习器能力提升，专家占比逐步下降并最终为零，使学习器可独立行动。
- 在两个基准的34个任务上，相比有无指导基线提升样本效率，且超参数变化较少。
验证范围与局限：验证范围为两个基准的34个任务，覆盖离散和连续动作空间，并使用学习型与模型型专家。
英文摘要 (Abstract)
With massively parallel simulation, on-policy Reinforcement Learning methods such as PPO have become standard in many domains. However, learning from scratch is sample-inefficient and fails to exploit the potential existence of a suboptimal expert, such as a heuristic, a model-based controller, or a policy trained on a related task. Such an expert is often available and can guide early training, but its sub-optimality limits final performance. The challenge then becomes balancing expert guidance against learning from rewards. Existing methods set the expert's influence through a blending weight, a schedule, or an evaluation-driven curriculum. Alternatively, they adapt it with additional learned components such as critics over expert actions or auxiliary agents. However, none optimizes it using the same on-policy objective as the policy itself. We propose a method in which the learner and the expert alternate control within each training episode, and the expert's share of control is a single learnable parameter optimized jointly with the policy. The learner benefits from the expert early in training, but its share of control declines as the learner becomes more competent, until eventually vanishing completely. This leaves the learner acting alone and better than the suboptimal expert. We evaluate our method on 34 tasks across two benchmarks, spanning discrete and continuous action spaces, using both learned and model-based experts. Our method improves sample efficiency over guided and unguided baselines while requiring minimal hyperparameter variation. The expert's share decays to zero as the learner improves, vanishing when the expert is no longer useful.
作者：Daniele Affinita, Ming Xu, Rudolf Reiter, Davide Scaramuzza, Pascal Fua
来源
本页事实对应的公开页面，以原文为准。