# 面向稀疏长时域环境的方差规避 n-Step 离线强化学习

> 原文链接：https://robotics.hellobike.com/papers/2026-10-06-2610-07899
> 频道：papers | 更新日期：2026-10-06 | 标签：arxiv, cs.RO
> 核心摘要：针对异质离线数据中生成式策略易选高方差动作的问题，VAN-Flow用方差规避分布式评价与流匹配actor在D4RL和OGBench任务上验证。

# 面向稀疏长时域环境的方差规避 n-Step 离线强化学习

::: v-pre
> 原文链接：[Variance-Averse $n$-Step Offline Reinforcement Learning for Sparse Long-Horizon Environments](https://arxiv.org/abs/2610.07899)（arXiv 2610.07899 · 发布日期：2026-10-06）

**Variance-Averse $n$-Step Offline Reinforcement Learning for Sparse Long-Horizon Environments**

针对异质离线数据中生成式策略易选高方差动作的问题，VAN-Flow用方差规避分布式评价与流匹配actor在D4RL和OGBench任务上验证。

## 核心要点

- 异质数据中，生成式策略可能复现回报高方差且不稳定的动作模式。
- 方法结合分类分布式critic、方差规避期望算子和拒绝采样引导的流匹配actor。
- 该算子重分配分类回报分布的概率质量，避免硬截断或辅助惩罚项。
- 在D4RL和OGBench的40多个任务上优于强基线，长时域和高方差场景增益最大。

**验证范围与局限**：验证范围为D4RL和OGBench的40多个任务。

## 英文摘要 (Abstract)

Generative actors are transforming offline reinforcement learning (RL) by enabling expressive policy classes that model complex action distributions. However, this expressiveness also exposes a key challenge in heterogeneous datasets: generative policies can reproduce unreliable action modes whose return distributions exhibit high variance, occasionally yielding high returns by chance but lacking consistency. Consequently, maximizing the expected $Q$-value alone is insufficient for identifying reliable actions. We propose VAN-Flow (Variance-Averse $n$-step Flow), a framework that promotes reliable actions in generative offline RL. VAN-Flow combines (i) a categorical distributional critic, (ii) a variance-averse expectation operator that smoothly reweights atom probabilities to favor actions with both high returns and low dispersion, and (iii) a flow-matching generative actor guided via rejection sampling. Unlike CVaR or mean-variance objectives, the operator redistributes probability mass over the categorical return distribution without hard truncation or auxiliary penalty terms. Across more than 40 tasks from D4RL and OGBench, VAN-Flow consistently outperforms strong baselines, with the largest gains in long-horizon and high-variance regimes where reliable action selection becomes critical.

作者：Guhyeon Kang, Minhae Kwon
:::

## 原始参考来源 (Sources)

- [arXiv 2610.07899：Variance-Averse $n$-Step Offline Reinforcement Learning for Sparse Long-Horizon Environments](https://arxiv.org/abs/2610.07899)