论文/arXiv 2610.14936·18 天前

CFG-Policy：基于 Classifier-free Guidance 的统一 Diffusion Policy 学习框架

针对扩散策略强化学习中性能提升与多样性保持的矛盾，CFG-Policy用双生成策略引导采样，并在MuJoCo和RoboTwin任务上验证。

AI 整理

• 本体 SDK 与控制
• 仿真与训练
• 具身模型

原文链接：CFG-Policy: Unified Diffusion Policy Learning Framework via Classifier-free Guidance（arXiv 2610.14936 · 发布日期：2026-10-10）

CFG-Policy: Unified Diffusion Policy Learning Framework via Classifier-free Guidance

针对扩散策略强化学习中性能提升与多样性保持的矛盾，CFG-Policy用双生成策略引导采样，并在MuJoCo和RoboTwin任务上验证。

核心要点

• 扩散策略可表达多模态动作分布，但强化学习优化可能降低行为多样性并限制探索能力。
• CFG-Policy将探索与利用分为两个生成策略，并在采样时通过classifier-free guidance组合其场。
• 在线强化学习中冻结探索分支，离线到在线强化学习中冻结预训练利用分支以保持能力。
• 该框架保持底层强化学习目标，并在QSM、QVPO、Flow-SDE+PPO设置下提升性能，提供项目页面。

验证范围与局限：验证范围为5个MuJoCo任务和4个RoboTwin操作任务。

英文摘要 (Abstract)

Diffusion policies are increasingly used in robotic control and manipulation, vision-language-action models, and world-model-based agents. Their ability to represent multimodal action distributions offers rich opportunities for exploration in reinforcement learning (RL). However, optimizing diffusion policies with RL can reduce the behavioral diversity and further limit their exploration capability. In RL post-training, directly fine-tuning a pretrained policy may also disrupt its acquired capabilities, while action-space residuals may lack the expressive power for multimodal exploration. Together, these issues reveal a common challenge across online and offline-to-online diffusion RL: improving task performance while retaining the behavioral diversity and capabilities needed for continued adaptation. We introduce CFG-Policy, a unified, plug-and-play framework that separates exploration and exploitation into two generative policies and combines their score or velocity fields during sampling through classifier-free guidance. For online RL, a frozen exploration branch sustains behavioral diversity while an exploitation branch learns from task rewards; for offline-to-online RL, a pretrained exploitation branch remains frozen while a trainable branch steers the composed behavior. CFG-Policy preserves the underlying RL objectives and improves performance with QSM and QVPO on 5 MuJoCo tasks and with Flow-SDE+PPO on 4 RoboTwin manipulation tasks. Project page is at https://zhy-liu.github.io/cfg-policy-page.

作者：Shutong Ding, Zhaoyang Liu, Yimiao Zhou, Jingya Wang, Jiayuan Gu, Ye Shi

来源

本页事实对应的公开页面，以原文为准。

1. arXiv 2610.14936：CFG-Policy: Unified Diffusion Policy Learning Framework via Classifier-free Guidance访问 2026-10-21