# EliteReplay：通过质量-多样性搜索在RL后训练中保留多样化解法

> 原文链接：https://robotics.hellobike.com/papers/2026-10-15-2610-22290
> 频道：papers | 更新日期：2026-10-15 | 标签：arxiv, cs.RO
> 核心摘要：EliteReplay用质量-多样性搜索构建精英档案并结合GRPO，在多个text-to-SQL任务上提升准确率并保留解法多样性。

# EliteReplay：通过质量-多样性搜索在RL后训练中保留多样化解法

::: v-pre
> 原文链接：[EliteReplay: Preserving Diverse Solutions in RL Post-Training via Quality-Diversity Search](https://arxiv.org/abs/2610.22290)（arXiv 2610.22290 · 发布日期：2026-10-15）

**EliteReplay: Preserving Diverse Solutions in RL Post-Training via Quality-Diversity Search**

EliteReplay用质量-多样性搜索构建精英档案并结合GRPO，在多个text-to-SQL任务上提升准确率并保留解法多样性。

## 核心要点

- 强化学习后训练虽能提升LLM可验证任务准确率，却可能缩窄模型可产生的解法集合。
- EliteReplay先用冻结基座模型收集不同思路解法，训练时将GRPO与档案回放损失结合。
- 在多个text-to-SQL任务上，其pass@128高于所有基线，正确策略最多比最强基线多77%。
- 在AmbiQT模糊问题上，EliteReplay覆盖两种解释的频率接近最强基线的两倍。

**验证范围与局限**：验证范围为多个text-to-SQL任务和AmbiQT模糊问题。

## 英文摘要 (Abstract)

Reinforcement learning has become a common post-training technique to improve the accuracy of LLMs on verifiable tasks. However, such training can narrow the set of solutions the model can produce. Several approaches have attempted to address this behavior-sharpening effect, but they often optimize metrics that do not reflect the diversity of correct solutions. Ideally, we want a model that can produce multiple good solutions, not only a single reliable one. We propose EliteReplay, a post-training recipe that pairs GRPO with an "elite" archive built by quality-diversity search. Before training, we use the frozen base model to collect solutions that span distinct approaches; during training, GRPO is combined with a replay loss on this archive. The result is a post-trained model that improves accuracy while preserving more of the base model's solution diversity. On multiple text-to-SQL tasks, EliteReplay achieves a higher pass@128 than all baselines, producing up to 77% more distinct correct strategies than the strongest baselines. It preserves most of the base model's correct strategies, outperforming existing methods in strategy retention. This diversity translates into practical benefits on ambiguous questions, where EliteReplay covers both interpretations nearly twice as often as the strongest baseline on AmbiQT.

作者：Hanane Nour Moussa, Doeun Lee, Sachin Kumar
:::

## 原始参考来源 (Sources)

- [arXiv 2610.22290：EliteReplay: Preserving Diverse Solutions in RL Post-Training via Quality-Diversity Search](https://arxiv.org/abs/2610.22290)