# 用于高效扩散式离线强化学习的组合式平均速度建模

> 原文链接：https://robotics.hellobike.com/papers/2026-10-10-2610-14941
> 频道：papers | 更新日期：2026-10-10 | 标签：arxiv, cs.RO
> 核心摘要：提出InsQL以组合式平均速度建模实现单步生成策略，并在离线RL基准和真实机器人操作任务上验证效率与性能。

# 用于高效扩散式离线强化学习的组合式平均速度建模

::: v-pre
> 原文链接：[Compositional Average Velocity Modeling for Efficient Diffusion Offline Reinforcement Learning](https://arxiv.org/abs/2610.14941)（arXiv 2610.14941 · 发布日期：2026-10-10）

**Compositional Average Velocity Modeling for Efficient Diffusion Offline Reinforcement Learning**

提出InsQL以组合式平均速度建模实现单步生成策略，并在离线RL基准和真实机器人操作任务上验证效率与性能。

## 核心要点

- 扩散策略依赖多步去噪和训练时BPTT，带来计算开销并限制可扩展部署。
- InsQL以平均速度替代边际速度参数化，构建跨尺度一致的组合式目标。
- 框架结合行为正则化Actor-Critic，仅需单个策略网络且无需蒸馏、多阶段或雅可比计算。
- 在标准离线RL基准和真实机器人操作任务中，性能匹配或超过扩散方法并缩短训练和推理时间。

**验证范围与局限**：验证范围为标准离线RL基准和真实世界机器人操作任务。

## 英文摘要 (Abstract)

Diffusion-based policies have recently emerged as a powerful paradigm for offline reinforcement learning, but their reliance on multi-step denoising introduces substantial computational overhead and requires backpropagation through time during training, limiting scalability and practical deployment. In this work, we propose \textbf{Instant Flow Q-Learning (InsQL)}, a single-step generative policy learning framework that retains the expressive capacity of diffusion models while eliminating iterative sampling and BPTT. InsQL builds on the flow-matching perspective of diffusion policies, but replaces marginal velocity parameterization with \emph{average-velocity} modeling and introduces a principled compositional mechanism, yielding a theoretically grounded objective that learns average velocity consistently across scales. This formulation provides a direct noise-to-action mapping and enables single-step action generation. InsQL further unifies compositional average-velocity modeling with behavior-regularized actor--critic optimization in a single framework, requiring only a single policy network and avoiding distillation, multi-stage procedures, and Jacobian computations during training. At inference time, actions are generated in a single forward pass and can be refined via value-guided candidate selection to improve performance. Extensive experiments on standard offline RL benchmarks and real-world robotic manipulation tasks show that InsQL matches or surpasses state-of-the-art diffusion-based methods while substantially reducing both training and inference time. Overall, these results establish compositional average-velocity modeling as an efficient and scalable alternative to multi-step generative policies for offline RL.

作者：Thanh Nguyen, Abu Hanif Muhammad Syarubany, Hwanhee Kim, Chang D. Yoo
:::

## 原始参考来源 (Sources)

- [arXiv 2610.14941：Compositional Average Velocity Modeling for Efficient Diffusion Offline Reinforcement Learning](https://arxiv.org/abs/2610.14941)