# 倾斜采样问题：乐观、后验采样与对数遗憾

> 原文链接：https://robotics.hellobike.com/papers/2026-10-15-2610-21908
> 频道：papers | 更新日期：2026-10-15 | 标签：arxiv, cs.RO
> 核心摘要：本文研究未知奖励诱导Gibbs分布的在线倾斜采样，提出采样遗憾框架与UCS、TS^2并给出对数遗憾理论保证。

# 倾斜采样问题：乐观、后验采样与对数遗憾

::: v-pre
> 原文链接：[The Tilted Sampling Problem: Optimism, Posterior Sampling, and Logarithmic Regret](https://arxiv.org/abs/2610.21908)（arXiv 2610.21908 · 发布日期：2026-10-15）

**The Tilted Sampling Problem: Optimism, Posterior Sampling, and Logarithmic Regret**

本文研究未知奖励诱导Gibbs分布的在线倾斜采样，提出采样遗憾框架与UCS、TS^2并给出对数遗憾理论保证。

## 核心要点

- 动机来自熵正则强化学习、RLHF和奖励引导生成建模等需采样多样高质量输出的系统。
- 将目标定义为从未知奖励函数诱导的Gibbs分布中采样，并用温度归一化累积KL散度刻画采样遗憾。
- 提出UCB和Thompson sampling的采样对应方法UCS和TS^2，给出统一遗憾分解。
- 结果表明固定温度的标准设置中，采样遗憾由平方置信宽度主导，可呈对数级增长。

**验证范围与局限**：摘要未提实证验证，仅描述标准设置、固定温度下的理论分析。

## 英文摘要 (Abstract)

Many modern learning systems are used not to select a single best action, but to sample diverse high-quality outputs. Examples include entropy-regularized reinforcement learning, RLHF, and reward-guided generative modeling. Motivated by these settings, we study tilted sampling: an online learning problem in which the goal is to sample from the Gibbs distribution induced by an unknown reward function. We define sampling regret as the temperature-normalized cumulative KL divergence between the learner's distribution and the Gibbs target, corresponding to the suboptimality gap in entropy-regularized reward maximization. Our framework generalizes recent sampling-based and KL-regularized formulations in bandits and reinforcement learning. Our main contribution is a general regret framework that captures the different uncertainty structure of tilted sampling and classical regret minimization. While standard regret scales as $\widetilde{O}(\sqrt{T})$ due to first-order uncertainty, sampling regret is governed by squared confidence widths, yielding logarithmic-in-$T$ regret in standard settings at fixed temperature. We develop sampling analogues of UCB and Thompson sampling (UCS and TS$^2$) and provide unified regret decompositions that explain and extend recent logarithmic guarantees, including the first Bayesian sampling-regret decomposition based on new bounds for posterior-random Gibbs distributions. Our results show that learning to sample from a soft-optimal distribution is statistically distinct from learning to identify a hard optimum.

作者：Mohammad Pedramfar, Siamak Ravanbakhsh
:::

## 原始参考来源 (Sources)

- [arXiv 2610.21908：The Tilted Sampling Problem: Optimism, Posterior Sampling, and Logarithmic Regret](https://arxiv.org/abs/2610.21908)