外观
论文arXiv 2610.2176413 天前
Q(γ_H)：多折扣 Q-Learning 中的熵引导视野混合
原文链接：Q($γ_H$): Entropy-Guided Horizon Mixing in Multi-Discount Q-Learning（arXiv 2610.21764 · 发布日期：2026-10-15）
Q($γ_H$): Entropy-Guided Horizon Mixing in Multi-Discount Q-Learning
针对固定折扣限制单一规划视野的问题，Q(γ_H)以熵置信号混合多折扣头，并在经典控制和深度强化学习基准上验证。
核心要点
- 固定折扣因子使 Q-learning 全程采用单一规划视野，难以同时适应早期探索和长期信用分配需求。
- 该方法维护多个折扣特定头，在控制时用熵导出的置信号生成状态相关权重进行自适应混合。
- 随着策略置信度提升，控制从短视野头转向长视野头，且不修改各个 Bellman 更新。
- 在经典控制和深度强化学习基准中，相比强固定折扣基线，在延迟信用任务上改进最明显。
验证范围与局限：验证范围为经典控制和深度强化学习基准，摘要未涉及真实机器人实验。
英文摘要 (Abstract)
A fixed discount factor in Q-learning imposes a single planning horizon throughout training, despite the differing demands of early exploration and long-term credit assignment. We propose Q($γ_H$), a multi-horizon Q-learning framework that maintains multiple discount-specific heads and adaptively combines them at control time using an entropy-derived confidence signal. The resulting state-dependent mixing weights shift control from shorter- to longer-horizon heads as policy confidence increases, enabling adaptive horizon mixing without modifying individual Bellman updates. Across classical control and deep reinforcement-learning benchmarks, Q($γ_H$) shows its strongest improvements on delayed-credit tasks relative to strong fixed-discount baselines. By decoupling horizon adaptation from the underlying learning updates, Q($γ_H$) provides a lightweight framework for adaptive planning-horizon control in reinforcement learning, avoiding the need for manual discount-factor selection while remaining compatible with standard value-learning methods.
作者：Arwa Albelaihi, Abdulrahman Altahhan
来源
本页事实对应的公开页面，以原文为准。