外观
论文arXiv 2610.1542817 天前
用于鲁棒离线元强化学习的悲观潜在任务感知优化
原文链接：Pessimistic Latent Task-aware Optimization for Robust Offline Meta-Reinforcement Learning（arXiv 2610.15428 · 发布日期：2026-10-11）
Pessimistic Latent Task-aware Optimization for Robust Offline Meta-Reinforcement Learning
针对COMRL在参数外推任务中潜变量越出训练支持的问题，PLATO通过外推扰动潜变量并用悲观权重想象转移，在MuJoCo基准验证。
核心要点
- 以往COMRL侧重任务表示，隐含假设干净潜变量能迁移到训练支持外任务。
- PLATO将潜变量从训练质心向外扰动，合成反事实任务以训练策略。
- 解码器集成在扰动潜变量上短滚动，并按分歧施加悲观权重更新。
- 八个MuJoCo连续控制基准上，支持外回报优于强基线，分布内仍具竞争力；代码已开源。
验证范围与局限：验证限于八个MuJoCo连续控制基准上的支持外与分布内任务。
英文摘要 (Abstract)
Context-based offline meta-reinforcement learning (COMRL) has advanced almost entirely through better task representations, which separate task identity from artifacts of the behavior policy so that the latent reflects the task alone. The implicit expectation has been that a clean representation will transfer to out-of-support (parameter-extrapolation) tasks. On such a task, however, the encoder produces a latent outside the training support, where the policy has never been trained. We address this gap with Pessimistic Latent Task-aware Optimization (PLATO), which exposes the policy to off-support latents during training. PLATO assumes a mild manifold hypothesis: task latents lie on a low-dimensional manifold, and out-of-support tasks lie farther from the training centroid. It exploits this geometry by perturbing inferred latents outward from the centroid to synthesize counterfactual tasks. A learned decoder ensemble then rolls out short trajectories at each perturbed latent, and the policy is updated on the imagined transitions under a pessimism weight that down-weights steps on which the ensemble disagrees. We prove that the perturbation leaves the training support by a controllable margin and that the out-of-support generalization gap is bounded by the same perturbation reach, and we give explicit conditions, verified empirically, under which the epistemic disagreement at the perturbed latent stays bounded away from zero. On eight MuJoCo continuous-control benchmarks, PLATO consistently improves out-of-support returns over strong COMRL baselines while remaining competitive on in-distribution tasks. Our code is available at https://github.com/deephoony/plato.
作者：Chang-Hoon Jeong, Kisung Shin, Hanwool Sul, Byoung-Tak Zhang
来源
本页事实对应的公开页面，以原文为准。