外观
论文arXiv 2609.127491 个月前
SCQ：用 Sigmoid 有界熵稳定 Conservative Q-Learning
原文链接：SCQ: Stabilizing Conservative Q-Learning with Sigmoid-Bounded Entropy（arXiv 2609.12749 · 发布日期：2026-09-11）
SCQ: Stabilizing Conservative Q-Learning with Sigmoid-Bounded Entropy
SCQ用正的sigmoid有界熵稳定离线到在线强化学习，并在D4RL、仿真与真实视觉任务验证。
核心要点
- 动机是离线到在线强化学习虽降低交互成本，但价值估计持续不稳定。
- 方法以始终为正的 sigmoid 有界熵替代标准 log 熵项，并保留保守 Q 正则。
- 在状态与视觉基准上，SCQ 匹配或超过基线，且训练动态更稳定。
- 干预与正分数对照表明，正性而非特定得分形状驱动了大部分改进。
验证范围与局限：验证范围包括 D4RL（Minari）单演示与标准数据集、仿真和真实视觉任务，以及四个真实机器人平台。
英文摘要 (Abstract)
Offline-to-online reinforcement learning reduces interaction cost for real-world robot learning but suffers from persistent value estimation instability. Existing methods address this through pessimistic regularization, lower-bound calibration, and architectural normalization, but an overlooked source of instability lies in the entropy formulation: the standard log-entropy term can become negative, destabilizing policy updates. We introduce SCQ (Sigmoid-Bounded Conservative Q-Learning), which replaces this term with a sigmoid-bounded formulation that stays strictly positive. SCQ retains conservative Q regularization and return-based lower-bound calibration, stabilizing policy optimization without sacrificing exploration. We evaluate SCQ on D4RL (Minari) benchmarks under both single-demonstration and standard dataset settings, as well as on simulation and real-world visual tasks. SCQ matches or exceeds baseline performance while exhibiting more stable training dynamics across state-based and visual benchmarks, and transfers to four real-robot platforms including manipulation, wheeled, quadruped, and humanoid systems. A direct clipping intervention that removes negative log-probability contributions, together with gradient-matched positive-score controls, indicates that positivity rather than a particular score shape alone drives much of the improvement. Project website: https://scq-rl.github.io.
作者：Xiefeng Wu, Shu Zhang, Zhaojie Chu, Mingyu Hu
来源
本页事实对应的公开页面，以原文为准。