# QF3：带过滤 Q 梯度的快速 Flow 强化学习

> 原文链接：https://robotics.hellobike.com/papers/2026-10-06-2610-08789
> 频道：papers | 更新日期：2026-10-06 | 标签：arxiv, cs.RO
> 核心摘要：QF3用流匹配结合过滤的critic动作梯度训练流策略，并在人形运动、运动跟踪及操作微调任务上验证。

# QF3：带过滤 Q 梯度的快速 Flow 强化学习

::: v-pre
> 原文链接：[QF3: Fast Flow RL with Filtered Q-Gradients](https://arxiv.org/abs/2610.08789)（arXiv 2610.08789 · 发布日期：2026-10-06）

**QF3: Fast Flow RL with Filtered Q-Gradients**

QF3用流匹配结合过滤的critic动作梯度训练流策略，并在人形运动、运动跟踪及操作微调任务上验证。

## 核心要点

- Flow策略常用于从示范学习机器人行为，但仍需强化学习通过交互改进或从零学习。
- QF3是在线离策略RL算法，将流匹配与critic动作梯度结合，并经一步输出预测反传。
- 为提升更新可靠性，QF3只在接近回放动作的动作维度上施加critic梯度。
- 在人形运动和运动跟踪中相较FPO++实现10倍墙钟加速，并用于ABC-Sim和Robomimic操作微调。

**验证范围与局限**：验证范围包括人形运动与运动跟踪、零样本迁移到硬件，以及ABC-Sim和Robomimic操作微调。

## 英文摘要 (Abstract)

Flow policies have become a standard policy class for learning robot behaviors from demonstrations, but reinforcement learning is still critical for improving pre-trained flow policies or learning them from scratch through interaction. We introduce QF3 (Fast Flow RL with Filtered Q-Gradients), an online off-policy RL algorithm that trains a flow policy with flow matching plus the critic's action gradient, backpropagated through a one-step prediction of the flow's output. To keep updates where the critic and this prediction are reliable, QF3 applies the critic gradient only to action dimensions that stay near the replay action. To our knowledge, QF3 is the first off-policy flow RL method to train humanoid locomotion policies from scratch and transfer them zero-shot to hardware. Paired with a high-throughput off-policy training recipe, it trains humanoid locomotion and motion-tracking policies with a 10x wall-clock speedup over FPO++, a recent on-policy flow RL method. We further apply QF3 to fine-tune pretrained flow-based manipulation policies on both ABC-Sim and Robomimic tasks. These results suggest that QF3 can both learn robot policies from scratch and refine those acquired from demonstrations. Website: https://qf3-rl.github.io/

作者：Chung Min Kim, Brent Yi, David McAllister, Hongsuk Choi, Himanshu Singh, Jinkun Cao, Ken Y. Goldberg, Pieter Abbeel, Carmelo Sferrazza, Angjoo Kanazawa
:::

## 原始参考来源 (Sources)

- [arXiv 2610.08789：QF3: Fast Flow RL with Filtered Q-Gradients](https://arxiv.org/abs/2610.08789)