外观
论文arXiv 2610.0250527 天前
多保真策略梯度稳定数据稀缺的强化学习
针对目标域数据稀缺导致的on-policy RL不稳定，提出MFPG-PPO，并在仿真机器人运动和实体Franka臂上验证。
Franka FR3原文链接：Multi-Fidelity Policy Gradients Stabilize Data-Scarce Reinforcement Learning（arXiv 2610.02505 · 发布日期：2026-10-01）
Multi-Fidelity Policy Gradients Stabilize Data-Scarce Reinforcement Learning
针对目标域数据稀缺导致的on-policy RL不稳定，提出MFPG-PPO，并在仿真机器人运动和实体Franka臂上验证。
核心要点
- 昂贵稀缺的高保真目标域数据会使on-policy策略梯度估计噪声大、训练不稳定。
- MFPG仅用低保真数据构造控制变量，在不引入偏差下降低方差、提升高保真数据效率。
- MFPG-PPO重设计采样、优势估计和控制变量，并监测估计不确定性以避免方差膨胀。
- 仿真运动任务中几乎所有设置优于仅用高保真数据的PPO，真机Franka臂上实现稳定学习。
验证范围与局限：验证范围包括GPU并行仿真中的机器人运动任务和一台实体Franka臂，真机每次更新仅用4个真实机器人episode。
英文摘要 (Abstract)
Policy gradient methods for on-policy reinforcement learning (RL) can become unstable when expensive, scarce target-domain data yield noisy gradient estimates. We address this challenge by complementing limited high-fidelity (HF) target-domain data with abundant, cheap, but biased low-fidelity (LF) data, e.g., from a simplified simulator. Most existing methods directly optimize biased objectives based on LF data. In contrast, the recently introduced multi-fidelity policy gradient (MFPG) framework uses LF data solely to construct a control variate that reduces variance and improves HF data efficiency without biasing the policy gradient estimator. However, published work on MFPG is limited to REINFORCE on small-scale simulation tasks. We develop MFPG for modern actor-critic learning in GPU-parallel simulation and on a physical robot. Our analysis and experiments show that naive extensions to proximal policy optimization (PPO) can lose cross-fidelity correlation or inflate variance. Our MFPG-PPO addresses these failures by redesigning the sampling, advantage estimation, and control variate construction to preserve cross-fidelity correlation, and by monitoring estimator uncertainty to prevent variance inflation. We also introduce a budget-aware MFPG-PPO to divide a fixed sampling budget among high- and low-fidelity data sources. Across simulated robot locomotion tasks of varying LF-to-HF transfer difficulty and HF data budgets, MFPG-PPO improves upon PPO trained on HF data alone in nearly all settings, and consistently matches the performance of PPO trained with 16x more HF data on the hardest task at the smallest HF budgets. In contrast, most baselines that use LF data perform well only where direct LF-to-HF transfer succeeds. MFPG-PPO enables stable learning on a physical Franka arm using only 4 real-robot episodes per update and no human demonstrations.
作者：Xinjie Liu, Ruihan Zhao, Anirban Chaudhuri, Cyrus Neary, Ufuk Topcu, David Fridovich-Keil
来源
本页事实对应的公开页面，以原文为准。