论文/arXiv 2610.12135·20 天前

用于分层强化学习的 Q-Shaped Options

针对长时域目标条件任务中的动作抽象学习，QSO以层级Q函数塑造共享编码器，并在离线运动与操作环境验证。

AI 整理

• 本体 SDK 与控制
• 具身模型

原文链接：Q-Shaped Options for Hierarchical Reinforcement Learning（arXiv 2610.12135 · 发布日期：2026-10-08）

Q-Shaped Options for Hierarchical Reinforcement Learning

针对长时域目标条件任务中的动作抽象学习，QSO以层级Q函数塑造共享编码器，并在离线运动与操作环境验证。

核心要点

• 长时域目标条件任务需要跨时间和状态推理，HRL成效取决于合适的动作抽象。
• 现有方法要么丢失最优控制所需的选项差异，要么保留不必要差异、削弱状态抽象。
• QSO在各层使用独立状态价值函数、Q函数和策略，并由相邻层Q函数共同塑造共享编码器。
• 在离线目标条件运动与操作环境中，QSO学习语义化选项空间并优于基线，部分任务取得非零表现。

验证范围与局限：验证范围为离线目标条件运动与操作环境。

英文摘要 (Abstract)

Learning to tackle long-horizon, goal-conditioned tasks requires an agent to reason over extended timescales and act across a broad range of states. In principle, Hierarchical Reinforcement Learning (HRL) addresses both challenges through the interaction between action (temporal) and state (spatial) abstraction. First, using an action abstraction to represent temporally extended behaviour as options reduces the effective decision horizon. Second, enabling different state abstractions at each level of the decision process permits greater data aggregation for learning. However, realising these two benefits of a hierarchical policy depends on learning an appropriate action abstraction. Current HRL algorithms fail in one of two ways. Some discard distinctions between options needed for optimal control, undermining hierarchy altogether. Others retain unnecessary distinctions, preserving horizon reduction, but forfeiting coarser state abstraction. In this work, we characterise three desiderata for an action abstraction. We introduce Q-Shaped Options (QSO) to address all three. QSO builds on an architecture with distinct state-value functions, Q functions and policies at each level of the hierarchy. It learns the action abstraction between consecutive levels as a shared encoder shaped by their respective Q functions. The low-level Q function uses the option as a goal, encouraging the abstraction to retain distinctions necessary for optimal control. The high-level Q function uses it as an action, encouraging unnecessary distinctions to be discarded. Across offline goal-conditioned locomotion and manipulation environments, QSO learns semantically meaningful option spaces and outperforms baselines, achieving non-zero performance in tasks where all other evaluated algorithms fail.

作者：Clarisse Wibault, Antoine Gorceix, Antonio León Villares, Alexey Zakharov, Evangelos Chatzaroulas, Michael Matthews, Eduardo Pignatelli, Jakob Foerster

来源

本页事实对应的公开页面，以原文为准。

1. arXiv 2610.12135：Q-Shaped Options for Hierarchical Reinforcement Learning访问 2026-10-21