外观
论文arXiv 2609.362501 个月前
带反馈校正的动作分块近端策略优化
原文链接：Action Chunking Proximal Policy Optimization with Feedback Correction（arXiv 2609.36250 · 发布日期：2026-09-28）
Action Chunking Proximal Policy Optimization with Feedback Correction
面向高维机器人控制中的动作分块学习与反馈不足，提出ACPPO及在线校正器，并在25个仿真任务验证。
核心要点
- 现有动作分块方法常依赖块级价值函数，且块内开环执行削弱接触任务反应性。
- ACPPO采用分块actor与标准状态价值critic，避免学习分块Q函数。
- ACPPO-Corr加入逐步反馈校正器，在每个动作块内在线调整规划动作。
- 在25个仿真机器人任务中ACPPO-Corr总体最优；消融显示中等块长和正则化重要，代码已开源。
验证范围与局限：验证范围为IsaacGym和Bi-DexHands中的25个仿真任务，涵盖运动、机械臂操作和灵巧手-物体交互。
英文摘要 (Abstract)
Action chunking provides temporal abstraction in reinforcement learning by selecting short action sequences instead of individual actions, but many existing approaches face two limitations in high-dimensional robotic control. First, many rely on value functions over action chunks, which can be difficult to learn as action dimensionality and chunk length grow. Second, executing chunks open-loop removes within-chunk feedback, limiting reactivity in contact-rich tasks. We present Action Chunking PPO (ACPPO), a PPO extension that uses a chunked actor while retaining a standard state-value critic, thereby avoiding chunked Q-functions. We further propose ACPPO-Corr, which augments the chunk planner with a stepwise feedback corrector that adjusts planned actions online within each chunk. Across 25 simulated robotics tasks from IsaacGym and Bi-DexHands, spanning locomotion, arm manipulation, and dexterous hand-object interaction, ACPPO-Corr achieves the strongest aggregate performance among evaluated methods and performs best on both decision-frequency-sensitive and decision-frequency-neutral task subsets. Ablations show that moderate chunk lengths work best and that corrector regularization is important for balancing chunk-level planning with local feedback. These results suggest that action chunking can be effective in online PPO when chunk-level planning is paired with closed-loop correction. The code is available at: https://github.com/hshhahn/ACPPO.
作者：Sanghyun Hahn, Jonghyun Choi
来源
本页事实对应的公开页面，以原文为准。