# 使用标量伴随匹配的 Q-Learning

> 原文链接：https://robotics.hellobike.com/papers/2026-10-07-2610-10437
> 频道：papers | 更新日期：2026-10-07 | 标签：arxiv, cs.RO
> 核心摘要：针对流策略离策略RL微调中逐步伴随计算昂贵的问题，SQAM用标量伴随和值惩罚，在OGBench和真实双臂机器人上验证。

# 使用标量伴随匹配的 Q-Learning

::: v-pre
> 原文链接：[Q-Learning with Scalar Adjoint Matching](https://arxiv.org/abs/2610.10437)（arXiv 2610.10437 · 发布日期：2026-10-07）

**Q-Learning with Scalar Adjoint Matching**

针对流策略离策略RL微调中逐步伴随计算昂贵的问题，SQAM用标量伴随和值惩罚，在OGBench和真实双臂机器人上验证。

## 核心要点

- 流策略经多步生成动作，用学习到的价值函数微调时，逐步传播价值信息计算代价高。
- 作者观察到预训练流策略的批均速度Jacobian集中在对角线，据此推导闭式标量伴随。
- SQAM将最终动作的价值梯度按流时间缩放，并在策略生成动作处加入价值惩罚。
- 在四个最难OGBench域上成功率较最强基线高18至35个百分点；真机三任务优于监督微调。

**验证范围与局限**：验证包括四个最难OGBench域，以及一台真实双臂机器人上的三个任务。

## 英文摘要 (Abstract)

Flow policies capture rich and diverse action distributions, and fine-tuning them with off-policy RL to improve beyond the demonstrations has drawn growing interest. However, fine-tuning a flow policy against a learned value function is not trivial, because the policy generates its action over many flow steps. Adjoint matching offers a principled way to update the flow model itself by propagating value information from the final action back to each flow step, but it requires a vector--Jacobian product through the policy at every step, a cost that grows with the number of flow steps and the policy size. We observe that the batch-averaged velocity Jacobian of pretrained flow policies concentrates on its diagonal. Motivated by this finding, we derive a closed-form scalar adjoint that scales the value gradient at the final action by the flow time, eliminating the per-step vector--Jacobian products. We further find that controlling the critic's value at policy-generated actions is particularly important under the scalar adjoint. Based on these findings, we propose Q-learning with Scalar Adjoint Matching (SQAM), which combines the scalar adjoint with a value penalty at those actions. SQAM's gains concentrate on the four hardest OGBench domains, where its success rate exceeds that of the strongest baseline in each domain by 18 to 35 percentage points. To test whether SQAM extends to large pretrained policies, we also fine-tune a vision-language-action policy on a real bimanual robot. SQAM improves over supervised fine-tuning on all three tasks.

作者：Yonghoon Dong, Minsung Yoon, Jaehyuk Kim, Jungwoo Park, Changyeon Kim, Jinwoo Shin
:::

## 原始参考来源 (Sources)

- [arXiv 2610.10437：Q-Learning with Scalar Adjoint Matching](https://arxiv.org/abs/2610.10437)