论文/arXiv 2609.21108·1 个月前

REFINEPPO：通过迭代动作优化学习连续控制策略

该文针对连续控制策略一次性决策难以再修正的问题，提出IAR并结合PPO，在14个基准控制任务上验证。

AI 整理

• 本体 SDK 与控制
• 感知与导航
• 具身模型

原文链接：REFINEPPO: Learning Continuous Control Policies by Iterative Action Refinement（arXiv 2609.21108 · 发布日期：2026-09-17）

REFINEPPO: Learning Continuous Control Policies by Iterative Action Refinement

该文针对连续控制策略一次性决策难以再修正的问题，提出IAR并结合PPO，在14个基准控制任务上验证。

核心要点

• 现有DRL连续控制策略多由状态一次前馈生成动作，难以重新考虑或逐步改进初始预测。
• IAR从初始动作提议出发，用共享优化网络依据状态和当前动作反复学习残差修正。
• 作者将迭代动作构造机制集成到PPO中，形成REFINEPPO用于连续控制策略学习。
• 在14个基准控制任务中，REFINEPPO达到或超过标准PPO，并在若干任务上收敛更快。

验证范围与局限：验证范围为14个基准控制任务及优化深度、更新日程等消融分析。

英文摘要 (Abstract)

Deep reinforcement learning (DRL) has achieved strong performance across a wide range of continuous-control problems. These continuous-control policies, however, are often defined as direct mappings from an observed state to an action or action distribution, requiring a single feed-forward network to construct an optimal control decision in one pass. While effective, this formulation leaves little opportunity for the policy to reconsider or progressively improve an action once an initial prediction has been formed. In this work, we explore an alternative approach: rather than learning only to directly predict an action, can a policy learn to iteratively improve one, and can this iterative process provide advantages during policy learning? We introduce Iterative Action Refinement (IAR), an iterative action-construction method that constructs control actions through a sequence of learned residual corrections. Starting from an initial proposal, a shared refinement network repeatedly conditions on the observed state and the current action proposal, allowing each refinement step to revise the action constructed by preceding steps. The final refined proposal is then used to determine the action executed by the agent. We integrate this iterative action-construction mechanism with Proximal Policy Optimization (PPO), yielding REFINEPPO. We evaluate REFINEPPO across 14 benchmark control tasks, complemented by controlled ablations of refinement depth and update schedules and analyses aimed at understanding why iterative refinement is effective. Across these environments, REFINEPPO matches or exceeds the performance of standard PPO while demonstrating faster convergence on several tasks.

作者：Sachini Weerasekara, Sagar Kamarthi, Jacqueline Isaacs

来源

本页事实对应的公开页面，以原文为准。

1. arXiv 2609.21108：REFINEPPO: Learning Continuous Control Policies by Iterative Action Refinement访问 2026-10-21