外观
论文arXiv 2609.345751 个月前
用于长时程离线目标条件强化学习的 Diffusion Subgoal Planning
原文链接：Diffusion Subgoal Planning for Long-Horizon Offline Goal-Conditioned Reinforcement Learning（arXiv 2609.34575 · 发布日期：2026-09-28）
Diffusion Subgoal Planning for Long-Horizon Offline Goal-Conditioned Reinforcement Learning
针对长时程离线GCRL价值引导不稳定问题，DSP用扩散式子目标生成进行分层规划，并在离线GCRL基准验证。
核心要点
- 长时程任务中，稀疏奖励和折扣使目标条件价值函数提供的指导不稳定。
- DSP将高层规划表述为目标条件子目标上的引导生成推断，学习条件与无条件流。
- 通过classifier-free guidance在推理时引入目标偏置，避免高层显式价值引导。
- 在离线GCRL导航和操作基准上优于已有方法，迷宫环境表现尤为突出。
验证范围与局限：验证范围为离线GCRL基准中的导航和操作任务，包括需要多步子目标规划的迷宫环境。
英文摘要 (Abstract)
Offline goal-conditioned reinforcement learning (GCRL) learns goal-directed policies from reward-free data, but in long-horizon tasks, goal-conditioned value functions often provide unstable guidance due to sparse rewards and discounting. Hierarchical methods partially mitigate this issue via subgoal decomposition; however, high-level decision-making still relies on noise-sensitive value estimates, leading to unstable behavior in complex environments. We address this limitation by proposing \textbf{D}iffusion \textbf{S}ubgoal \textbf{P}lanning (\textbf{DSP}), a diffusion-based framework for high-level subgoal generation. DSP casts high-level planning as guided generative inference over goal-conditioned subgoals and learns both conditional and unconditional flows, enabling classifier-free guidance to introduce a goal-directed bias at inference time. By removing explicit value-based guidance from high-level planning, DSP generates reachable and goal-directed subgoals through a generative model while retaining hierarchical execution. Experiments on offline GCRL benchmarks demonstrate that DSP outperforms prior methods on a range of navigation and manipulation tasks, with particularly strong performance in maze environments that require multi-step subgoal planning.
作者：Hengrui Zhang, Yuhu Cheng, C. L. Philip Chen, Xuesong Wang
来源
本页事实对应的公开页面，以原文为准。