外观
论文arXiv 2610.2190813 天前
倾斜采样问题：乐观、后验采样与对数遗憾
原文链接：The Tilted Sampling Problem: Optimism, Posterior Sampling, and Logarithmic Regret（arXiv 2610.21908 · 发布日期：2026-10-15）
The Tilted Sampling Problem: Optimism, Posterior Sampling, and Logarithmic Regret
本文研究未知奖励诱导Gibbs分布的在线倾斜采样，提出采样遗憾框架与UCS、TS^2并给出对数遗憾理论保证。
核心要点
- 动机来自熵正则强化学习、RLHF和奖励引导生成建模等需采样多样高质量输出的系统。
- 将目标定义为从未知奖励函数诱导的Gibbs分布中采样，并用温度归一化累积KL散度刻画采样遗憾。
- 提出UCB和Thompson sampling的采样对应方法UCS和TS^2，给出统一遗憾分解。
- 结果表明固定温度的标准设置中，采样遗憾由平方置信宽度主导，可呈对数级增长。
验证范围与局限：摘要未提实证验证，仅描述标准设置、固定温度下的理论分析。
英文摘要 (Abstract)
Many modern learning systems are used not to select a single best action, but to sample diverse high-quality outputs. Examples include entropy-regularized reinforcement learning, RLHF, and reward-guided generative modeling. Motivated by these settings, we study tilted sampling: an online learning problem in which the goal is to sample from the Gibbs distribution induced by an unknown reward function. We define sampling regret as the temperature-normalized cumulative KL divergence between the learner's distribution and the Gibbs target, corresponding to the suboptimality gap in entropy-regularized reward maximization. Our framework generalizes recent sampling-based and KL-regularized formulations in bandits and reinforcement learning. Our main contribution is a general regret framework that captures the different uncertainty structure of tilted sampling and classical regret minimization. While standard regret scales as $\widetilde{O}(\sqrt{T})$ due to first-order uncertainty, sampling regret is governed by squared confidence widths, yielding logarithmic-in-$T$ regret in standard settings at fixed temperature. We develop sampling analogues of UCB and Thompson sampling (UCS and TS$^2$) and provide unified regret decompositions that explain and extend recent logarithmic guarantees, including the first Bayesian sampling-regret decomposition based on new bounds for posterior-random Gibbs distributions. Our results show that learning to sample from a soft-optimal distribution is statistically distinct from learning to identify a hard optimum.
作者：Mohammad Pedramfar, Siamak Ravanbakhsh
来源
本页事实对应的公开页面，以原文为准。