# S3：通过约束层次强化学习中粗粒度动力学的不确定性实现稳定子目标选择

> 原文链接：https://robotics.hellobike.com/papers/2026-07-21-2607-19232
> 频道：papers | 更新日期：2026-07-21 | 标签：arxiv, cs.RO
> 核心摘要：本文针对HRL高层反馈稀疏且依赖低层执行的问题，引入粗粒度动力学不确定性内在奖励，并在非平稳长时程环境中验证。

# S3：通过约束层次强化学习中粗粒度动力学的不确定性实现稳定子目标选择

::: v-pre
> 原文链接：[S3: Stable Subgoal Selection by Constraining Uncertainty of Coarse Dynamics in Hierarchical Reinforcement Learning](https://arxiv.org/abs/2607.19232)（arXiv 2607.19232 · 发布日期：2026-07-21）

**S3: Stable Subgoal Selection by Constraining Uncertainty of Coarse Dynamics in Hierarchical Reinforcement Learning**

本文针对HRL高层反馈稀疏且依赖低层执行的问题，引入粗粒度动力学不确定性内在奖励，并在非平稳长时程环境中验证。

## 核心要点

- HRL高层代理面临稀疏、延迟反馈，且策略表现依赖低层执行能力。
- 方法在高层时间尺度使用多步聚合的粗粒度动力学，避免依赖原始转移的广覆盖。
- 通过MDN近似并评估不同离散度指标，最小化粗粒度动力学预测不确定性。
- 实验观察到稠密的动力学感知内在奖励带来风险规避子目标选择，并优于先进HRL方法。

**验证范围与局限**：验证范围为非平稳长时程环境中的经验实验。

## 英文摘要 (Abstract)

Hierarchical Reinforcement Learning (HRL) intends to separate strategic planning from primitive execution. It has been widely successful in solving long-horizon and complex tasks, where flat-RL algorithms have difficulty in learning. However, while the low-level agent in HRL benefits from dense feedback and abundant trial opportunities, the high-level agent receives sparse, delayed feedback from the environment and its performance depends on the low-level execution capability. In this paper, we study whether subgoal selection by the high-level agent can be performed more strategically, by providing it with dynamics-aware intrinsic motivation. Since motivation based on primitive transition dynamics would require broad coverage of the state-action space, we propose to use coarse dynamics, i.e., environment transitions aggregated over multiple steps at the temporal scale at which the high-level agent operates. This approach stabilizes the high-level policy by learning to minimize the predictive uncertainty associated with the coarse dynamics, and provides a guided structure for navigation. We model the predictive uncertainty by evaluating different dispersion metrics as approximated by a Mixture Density Network (MDN). Empirically, we observe that a dense, dynamics-aware intrinsic reward leads to risk-averse subgoal selection, enabling it to outperform state-of-the-art HRL methods in non-stationary long-horizon environments.

作者：Kshitij Kumar Srivastava, Kshitij Jerath
:::

## 原始参考来源 (Sources)

- [arXiv 2607.19232：S3: Stable Subgoal Selection by Constraining Uncertainty of Coarse Dynamics in Hierarchical Reinforcement Learning](https://arxiv.org/abs/2607.19232)