# 用于从未归一化密度采样的 Score-Calibrated Flow 及其在生成式在线强化学习中的应用

> 原文链接：https://robotics.hellobike.com/papers/2026-10-03-2610-04696
> 频道：papers | 更新日期：2026-10-03 | 标签：arxiv, cs.RO
> 核心摘要：针对在线RL中生成式策略难以从未归一化Boltzmann密度训练的问题，SCF用自一致流训练，并在RL基准验证。

# 用于从未归一化密度采样的 Score-Calibrated Flow 及其在生成式在线强化学习中的应用

::: v-pre
> 原文链接：[Score-Calibrated Flow for Sampling from Unnormalized Densities with Applications to Generative Online Reinforcement Learning](https://arxiv.org/abs/2610.04696)（arXiv 2610.04696 · 发布日期：2026-10-03）

**Score-Calibrated Flow for Sampling from Unnormalized Densities with Applications to Generative Online Reinforcement Learning**

针对在线RL中生成式策略难以从未归一化Boltzmann密度训练的问题，SCF用自一致流训练，并在RL基准验证。

## 核心要点

- 问题/动机：critic给出未归一化Boltzmann密度而非样本，重要性采样易高方差并增加成本。
- 方法：SCF通过自一致性学习目标流，无需重要性采样或对采样轨迹反向传播。
- 理论与训练：将速度条件写成固定点方程，用停止梯度目标保持CFM式训练结构。
- 实验：在RL基准上，SCF匹配或优于最先进生成式策略基线，并显著缩短训练时间。

**验证范围与局限**：实验验证范围为摘要所述RL基准，未说明具体任务或真实机器人验证。

## 英文摘要 (Abstract)

Diffusion and flow models provide expressive policy classes for online reinforcement learning (RL), enabling multimodal behaviors and improved performance. However, training these policies remains challenging: the critic specifies the desired policy as an unnormalized Boltzmann density but does not provide direct samples from it. Many existing methods rely on importance sampling to construct training signals, which can suffer from high variance, increasing computational cost and destabilizing training. We propose Score-Calibrated Flow (SCF), a simple and efficient algorithm for training generative models to sample from unnormalized densities without importance sampling or backpropagation through the sampling trajectory. We learn the desired flow by enforcing self-consistency, bypassing target posterior mean estimation. By jointly exploiting the prescribed target score and the structure of flow matching, we establish these self-consistency requirements as score-calibrated optimality conditions, first for the terminal density and then for the trainable velocity field. We prove that their unique solutions are, respectively, the target density and the ideal flow model that conditional flow matching (CFM) would recover if target samples were available. We formulate the velocity condition as a fixed-point equation and exploit its conditional-expectation structure to construct a stop-gradient objective for enforcing it. The resulting training procedure retains the scalable sample-interpolate-regress structure of CFM despite the absence of target samples, using endpoints generated by the current flow. For online RL, the critic gradient supplies the target score at the generated actions, yielding a direct approach to actor training. Experiments on RL benchmarks demonstrate that SCF matches or improves upon state-of-the-art generative-policy baselines, while substantially reducing training time.

作者：Zeyang Li, Yunan Wang, Risheek Garrepalli, Mohammad Ghavamzadeh, Navid Azizan
:::

## 原始参考来源 (Sources)

- [arXiv 2610.04696：Score-Calibrated Flow for Sampling from Unnormalized Densities with Applications to Generative Online Reinforcement Learning](https://arxiv.org/abs/2610.04696)