外观
论文arXiv 2609.198501 个月前
GR2PO：用于连续机器人控制的组相对回报策略优化
原文链接：GR2PO: Group Relative Return Policy Optimization for Continuous Robot Control（arXiv 2609.19850 · 发布日期：2026-09-17）
GR2PO: Group Relative Return Policy Optimization for Continuous Robot Control
提出无Critic的GR2PO，通过轨迹折扣回报与组归一化优化策略，并在机器人控制仿真和边缘设备上验证。
核心要点
- Actor-critic需学习价值网络，带来训练开销且可能受价值估计误差影响。
- GR2PO从并行采集轨迹估计折扣回报，按 rollout 时间索引组归一化并用相对优势更新策略。
- 实验显示其优于使用即时奖励的无Critic基线，并与先进actor-critic方法表现相当。
- 论文接收后将公开完整代码，包括框架实现、实验配置及训练评估脚本。
验证范围与局限：验证范围包括机器人控制仿真环境，以及在NVIDIA Jetson TX2边缘平台上的推理部署测试。
英文摘要 (Abstract)
Actor-critic architecture has been widely used in continuous robot control. However, they rely on learning a value network, introducing additional computational overhead during training. Moreover, policy learning may also be affected by the approximation error of value estimation. Critic-free group relative policy optimization methods provide a simpler training approach by removing the need for a critic. However, they fail to learn long-term action outcomes when directly applying immediate rewards to policy optimization in dense-reward environments. To address these problems, we propose Group Relative Return Policy Optimization (GR2PO), a critic-free reinforcement learning framework for continuous robot control. GR2PO estimates the discounted returns from the parallelly collected trajectories, performs group normalization at each rollout time index, and uses relative advantages and clipped targets to update the policy. To evaluate the effectiveness of the proposed framework, we instantiate it on robot control simulation environments and deploy the model to a real-world edge device. The results show that GR2PO significantly outperforms critic-free baselines that use immediate rewards and performs competitively against state-of-the-art actor-critic methods. Furthermore, GR2PO demonstrates competitive training efficiency. Inference tests on NVIDIA Jetson TX2 demonstrate the feasibility of deploying the learned policies on edge platforms. Further ablation experiments analyze the effects of parallel group size, return estimation methods, and target clipping ratio on learning performance. To support follow-up research, we will make the complete code publicly available after the paper is accepted, including the framework implementation, experimental configuration, and training and evaluation scripts.
作者：Pengqin Wang, Qiming Zhang, Shaojie Shen, Jun Ma
来源
本页事实对应的公开页面，以原文为准。