论文/arXiv 2610.14941·18 天前

用于高效扩散式离线强化学习的组合式平均速度建模

提出InsQL以组合式平均速度建模实现单步生成策略，并在离线RL基准和真实机器人操作任务上验证效率与性能。

AI 整理

• 感知与导航
• 具身模型
• 端侧部署

原文链接：Compositional Average Velocity Modeling for Efficient Diffusion Offline Reinforcement Learning（arXiv 2610.14941 · 发布日期：2026-10-10）

Compositional Average Velocity Modeling for Efficient Diffusion Offline Reinforcement Learning

提出InsQL以组合式平均速度建模实现单步生成策略，并在离线RL基准和真实机器人操作任务上验证效率与性能。

核心要点

• 扩散策略依赖多步去噪和训练时BPTT，带来计算开销并限制可扩展部署。
• InsQL以平均速度替代边际速度参数化，构建跨尺度一致的组合式目标。
• 框架结合行为正则化Actor-Critic，仅需单个策略网络且无需蒸馏、多阶段或雅可比计算。
• 在标准离线RL基准和真实机器人操作任务中，性能匹配或超过扩散方法并缩短训练和推理时间。

验证范围与局限：验证范围为标准离线RL基准和真实世界机器人操作任务。

英文摘要 (Abstract)

Diffusion-based policies have recently emerged as a powerful paradigm for offline reinforcement learning, but their reliance on multi-step denoising introduces substantial computational overhead and requires backpropagation through time during training, limiting scalability and practical deployment. In this work, we propose \textbf{Instant Flow Q-Learning (InsQL)}, a single-step generative policy learning framework that retains the expressive capacity of diffusion models while eliminating iterative sampling and BPTT. InsQL builds on the flow-matching perspective of diffusion policies, but replaces marginal velocity parameterization with \emph{average-velocity} modeling and introduces a principled compositional mechanism, yielding a theoretically grounded objective that learns average velocity consistently across scales. This formulation provides a direct noise-to-action mapping and enables single-step action generation. InsQL further unifies compositional average-velocity modeling with behavior-regularized actor--critic optimization in a single framework, requiring only a single policy network and avoiding distillation, multi-stage procedures, and Jacobian computations during training. At inference time, actions are generated in a single forward pass and can be refined via value-guided candidate selection to improve performance. Extensive experiments on standard offline RL benchmarks and real-world robotic manipulation tasks show that InsQL matches or surpasses state-of-the-art diffusion-based methods while substantially reducing both training and inference time. Overall, these results establish compositional average-velocity modeling as an efficient and scalable alternative to multi-step generative policies for offline RL.

作者：Thanh Nguyen, Abu Hanif Muhammad Syarubany, Hwanhee Kim, Chang D. Yoo

来源

本页事实对应的公开页面，以原文为准。

1. arXiv 2610.14941：Compositional Average Velocity Modeling for Efficient Diffusion Offline Reinforcement Learning访问 2026-10-21