外观
论文arXiv 2610.1093221 天前
用于目标条件强化学习的 World-Model Policy Arbiter
原文链接：World-Model Policy Arbiter for Goal-Conditioned Reinforcement Learning（arXiv 2610.10932 · 发布日期：2026-10-07）
World-Model Policy Arbiter for Goal-Conditioned Reinforcement Learning
WMPA针对离线GCRL中单一策略难以通用的问题，借助状态空间世界模型在OGBench上选择冻结策略并提升成功率。
核心要点
- 离线GCRL中，不同环境、目标乃至任务阶段的最佳策略并不一致。
- WMPA在测试时将冻结策略作为组合，在世界模型中想象展开并用共享目标条件价值函数评分。
- 系统选择得分最高策略执行一个短承诺区间，以减少频繁切换带来的控制不稳定。
- 在18个状态OGBench数据集上，宏平均成功率由44%升至58%，12个数据集显著提升。
验证范围与局限：验证范围为OGBench官方协议下的18个状态数据集，涵盖迷宫导航以及方块、场景和拼图操控。
英文摘要 (Abstract)
Offline goal-conditioned reinforcement learning (GCRL) has produced a diverse set of goal-reaching algorithms, yet no single algorithm performs best across environments, goals, and even different phases of the same task. Rather than deploying only the best-performing policy, we ask whether a set of frozen goal-conditioned policies can be used collectively as a portfolio, deciding at every state which policy should act. Choosing a policy at each state is not straightforward. The policies' own value functions cannot be compared directly: they may use different scales, and some policies have no value function. We need to judge each policy by the states it is likely to reach, even though we can execute only one policy at a time. We also need to avoid switching so often that control becomes unstable. To address these challenges, we introduce World-Model Policy Arbiter (WMPA), a test-time framework that, given a set of frozen policies as input, rolls out each frozen policy in a learned state-space world model, evaluates the imagined futures with a shared goal-conditioned value function, and executes the highest-scoring policy for a short commitment interval before the next round of arbitration (policy selection). WMPA assumes access to a bank of frozen goal-conditioned policies and requires neither policy retraining nor privileged task-specific knowledge. Under the official OGBench evaluation protocol on 18 state-based datasets spanning maze navigation as well as cube, scene, and puzzle manipulation, WMPA improves the macro-average success rate from the 44% achieved by the best policy selected per dataset to 58%, with statistically significant gains on 12 datasets. These gains include +33 percentage points on cube-double-play and +36 percentage points on scene-play.
作者：Junwei Quan, Evgenii Opryshko, Nicholas Rhinehart, Igor Gilitschenski
来源
本页事实对应的公开页面，以原文为准。