# SERO：从可执行预言机延续中学习的长时程智能体强化学习

> 原文链接：https://robotics.hellobike.com/papers/2026-10-14-2610-19397
> 频道：papers | 更新日期：2026-10-14 | 标签：arxiv, cs.RO
> 核心摘要：SERO从当前策略到达状态恢复环境，让训练用预言机继续交互并按策略可达性加权学习，在ALFWorld和WebShop上验证。

# SERO：从可执行预言机延续中学习的长时程智能体强化学习

::: v-pre
> 原文链接：[SERO: Learning from Executable Oracle Continuations for Long-Horizon Agentic Reinforcement Learning](https://arxiv.org/abs/2610.19397)（arXiv 2610.19397 · 发布日期：2026-10-14）

**SERO: Learning from Executable Oracle Continuations for Long-Horizon Agentic Reinforcement Learning**

SERO从当前策略到达状态恢复环境，让训练用预言机继续交互并按策略可达性加权学习，在ALFWorld和WebShop上验证。

## 核心要点

- 长时程智能体RL受当前策略访问未来限制，未遇到的状态和动作难获监督。
- SERO恢复当前策略实际到达的状态，由仅训练用预言机在真实环境中继续交互。
- 方法用策略可达性权重，按行为策略与预言机分支的兼容性加权每步监督。
- 在ALFWorld和WebShop评估；ALFWorld上完整延续优于仅监督根动作，并提升分布外性能。

**验证范围与局限**：验证限于ALFWorld和WebShop两个交互式智能体基准，使用Qwen2.5-1.5B-Instruct和Qwen2.5-7B-Instruct。

## 英文摘要 (Abstract)

Long-horizon agentic RL is constrained by the futures visited by the current policy. When rollouts fail to reach useful behaviors, improved credit assignment cannot provide supervision for states and actions that were never encountered. On-policy distillation (OPD) introduces teacher guidance on policy-induced states, and recent multi-turn variants improve how this guidance is scheduled, replayed, refined, or validated. However, they do not directly use the oracle-controlled future itself as the training trajectory. We introduce \textbf{State-Rooted Executable Rollout Optimization (SERO)}, which augments on-policy agentic RL with executable oracle continuations. SERO restores a state actually reached by the current policy and lets a training-only oracle continue interacting with the real environment, making the downstream observations and oracle actions themselves available for learning. It then applies \textbf{policy-reachability weighting}, which weights each oracle step by the frozen behavior policy's compatibility with the preceding oracle actions. This allows deeper supervision to become effective as the policy becomes more compatible with the oracle branch. The oracle loss is optimized separately from the base RL estimator and no oracle is used at inference time. We evaluate SERO on two challenging interactive-agent benchmarks, ALFWorld and WebShop, using Qwen2.5-1.5B-Instruct and Qwen2.5-7B-Instruct. On ALFWorld, learning the full executable continuation is substantially more effective than supervising only the root oracle action, while policy-reachability weighting further improves out-of-distribution performance.

作者：Dingyi Rong, Yue Shi, Chaofan Ma, Zongrui Wang, Jiezhang Cao, Yixin Cao, Liang He, Ning Liu
:::

## 原始参考来源 (Sources)

- [arXiv 2610.19397：SERO: Learning from Executable Oracle Continuations for Long-Horizon Agentic Reinforcement Learning](https://arxiv.org/abs/2610.19397)