论文/arXiv 2610.02198·27 天前

FERPO：前向熵正则化策略优化

提出FERPO，用无需动作求导的前向KL拟合目标动作分布，在MuJoCo Playground和ManiSkill验证性能与样本效率。

AI 整理

• 本体 SDK 与控制
• 仿真与训练
• 具身模型

原文链接：FERPO: Forward Entropy-Regularized Policy Optimization（arXiv 2610.02198 · 发布日期：2026-10-01）

FERPO: Forward Entropy-Regularized Policy Optimization

提出FERPO，用无需动作求导的前向KL拟合目标动作分布，在MuJoCo Playground和ManiSkill验证性能与样本效率。

核心要点

• 现有连续控制方法依赖评论器动作梯度，价值预测准确不一定带来可靠动作导数。
• FERPO从熵与KL正则化的策略改进目标推导目标动作分布，并用SNIS估计前向KL训练actor。
• KL正则限制目标分布偏离rollout策略，使重要性权重更稳定；前向KL鼓励覆盖多个高价值模式。
• 在MuJoCo Playground和ManiSkill实验及消融中表现有竞争力，并显示样本效率提升；actor更新快于REPPO。

验证范围与局限：验证范围为MuJoCo Playground和ManiSkill实验、消融及计算基准。

英文摘要 (Abstract)

Several state-of-the-art methods for online reinforcement learning in continuous control improve policies using action gradients of a learned critic. However, critics are typically trained to predict returns, and accurate value predictions do not necessarily yield accurate action derivatives, potentially leading to unreliable policy updates. We propose Forward Entropy-Regularized Policy Optimization (FERPO), an on-policy maximum entropy reinforcement learning algorithm that performs policy improvement using critic values without differentiating the critic with respect to actions. FERPO derives an optimal target action distribution from a policy-improvement objective regularized by entropy and Kullback-Leibler (KL) divergence. We then fit the actor to this target by minimizing a forward-KL objective, estimated using self-normalized importance sampling (SNIS) with actions drawn from the rollout policy. By limiting the target distribution's deviation from the rollout policy, the KL regularization helps keep these importance weights well behaved. In contrast to reverse-KL objectives, which can favor a subset of the target distribution's modes, the forward-KL objective encourages coverage of multiple high-value modes and thereby promotes exploration. Experiments and ablations on MuJoCo Playground and ManiSkill show competitive performance and sample-efficiency gains. Computational benchmarks also demonstrate faster actor updates than Relative Entropy Pathwise Policy Optimization (REPPO).

作者：Sebastian Sanokowski, Alireza Sarmadi, Majid Khadiv

来源

本页事实对应的公开页面，以原文为准。

1. arXiv 2610.02198：FERPO: Forward Entropy-Regularized Policy Optimization访问 2026-10-21