# Q(γ_H)：多折扣 Q-Learning 中的熵引导视野混合

> 原文链接：https://robotics.hellobike.com/papers/2026-10-15-2610-21764
> 频道：papers | 更新日期：2026-10-15 | 标签：arxiv, cs.RO
> 核心摘要：针对固定折扣限制单一规划视野的问题，Q(γ_H)以熵置信号混合多折扣头，并在经典控制和深度强化学习基准上验证。

# Q(γ_H)：多折扣 Q-Learning 中的熵引导视野混合

::: v-pre
> 原文链接：[Q($γ_H$): Entropy-Guided Horizon Mixing in Multi-Discount Q-Learning](https://arxiv.org/abs/2610.21764)（arXiv 2610.21764 · 发布日期：2026-10-15）

**Q($γ_H$): Entropy-Guided Horizon Mixing in Multi-Discount Q-Learning**

针对固定折扣限制单一规划视野的问题，Q(γ_H)以熵置信号混合多折扣头，并在经典控制和深度强化学习基准上验证。

## 核心要点

- 固定折扣因子使 Q-learning 全程采用单一规划视野，难以同时适应早期探索和长期信用分配需求。
- 该方法维护多个折扣特定头，在控制时用熵导出的置信号生成状态相关权重进行自适应混合。
- 随着策略置信度提升，控制从短视野头转向长视野头，且不修改各个 Bellman 更新。
- 在经典控制和深度强化学习基准中，相比强固定折扣基线，在延迟信用任务上改进最明显。

**验证范围与局限**：验证范围为经典控制和深度强化学习基准，摘要未涉及真实机器人实验。

## 英文摘要 (Abstract)

A fixed discount factor in Q-learning imposes a single planning horizon throughout training, despite the differing demands of early exploration and long-term credit assignment. We propose Q($γ_H$), a multi-horizon Q-learning framework that maintains multiple discount-specific heads and adaptively combines them at control time using an entropy-derived confidence signal. The resulting state-dependent mixing weights shift control from shorter- to longer-horizon heads as policy confidence increases, enabling adaptive horizon mixing without modifying individual Bellman updates. Across classical control and deep reinforcement-learning benchmarks, Q($γ_H$) shows its strongest improvements on delayed-credit tasks relative to strong fixed-discount baselines. By decoupling horizon adaptation from the underlying learning updates, Q($γ_H$) provides a lightweight framework for adaptive planning-horizon control in reinforcement learning, avoiding the need for manual discount-factor selection while remaining compatible with standard value-learning methods.

作者：Arwa Albelaihi, Abdulrahman Altahhan
:::

## 原始参考来源 (Sources)

- [arXiv 2610.21764：Q($γ_H$): Entropy-Guided Horizon Mixing in Multi-Discount Q-Learning](https://arxiv.org/abs/2610.21764)