# 超越策略对齐：用学习到的世界模型闭合机器人控制的规划-学习循环

> 原文链接：https://robotics.hellobike.com/papers/2026-09-30-2609-39751
> 频道：papers | 更新日期：2026-09-30 | 标签：arxiv, cs.RO
> 核心摘要：PL-MPC通过改进评论器监督、终端价值估计与动作蒸馏，处理规划-学习闭环，并在HumanoidBench、DMControl及KUKA真机任务上验证。

# 超越策略对齐：用学习到的世界模型闭合机器人控制的规划-学习循环

::: v-pre
> 原文链接：[Beyond Policy Alignment: Closing the Planning-Learning Loop for Robot Control with Learned World Models](https://arxiv.org/abs/2609.39751)（arXiv 2609.39751 · 发布日期：2026-09-30）

**Beyond Policy Alignment: Closing the Planning-Learning Loop for Robot Control with Learned World Models**

PL-MPC通过改进评论器监督、终端价值估计与动作蒸馏，处理规划-学习闭环，并在HumanoidBench、DMControl及KUKA真机任务上验证。

## 核心要点

- 规划器决定学习经验，评论器和演员又评分并提出未来计划，形成闭环反馈。
- PL-MPC加入混合多步TD目标、分歧感知终端估计和回报加权演员蒸馏。
- 在HumanoidBench的balance-hard和hurdle上TAR提升，整体表现依任务而定，并在DMControl保持竞争力。
- 代码和数据将发布；消融显示两个任务中不同组件交互不同。

**验证范围与局限**：验证范围包括HumanoidBench、DMControl，以及7-DoF KUKA IIWA14上扳手-螺母对齐的零样本Sim-to-Real。

## 英文摘要 (Abstract)

Planning with learned world models combines online trajectory optimization with learned value and policy functions for high-dimensional control. Because the planner determines the experience used for learning, while the learned critic and actor in turn score and propose future plans, planning and learning form a closed feedback loop. TD-MPC is a prominent instance of this design. Recent policy-constrained variants strengthen one part of the loop by aligning the learned policy with planner behavior. We introduce PL-MPC (Planning-Learning MPC), which additionally modifies critic supervision and planner terminal-value estimation. Hybrid multi-step TD targets expose critic updates to more realized rewards before bootstrapping; disagreement-aware terminal estimates reduce the influence of uncertain critic values during MPPI planning; and return-weighted actor distillation emphasizes planner-executed actions from high-return episodes. The world-model architecture and MPPI optimizer are otherwise unchanged. On HumanoidBench, the largest gains occur on \texttt{balance-hard}, where Total Average Return (TAR) increases from $98\pm18$ to $387\pm255$, and \texttt{hurdle}, from $199\pm13$ to $466\pm200$; performance across the broader benchmark remains task dependent, and PL-MPC remains competitive on DMControl. Controlled ablations show different component interactions across the two tasks. We further demonstrate zero-shot sim-to-real transfer on wrench-nut alignment with a 7-DoF KUKA IIWA14, obtaining higher observed success than TD-M(PC)^2 on the training object size and two unseen sizes. Code and data will be available at: https://pl-mpc-humanoid.github.io.

作者：Kowndinya Boyalakuntla, Yuhan Liu, Abdeslam Boularias
:::

## 原始参考来源 (Sources)

- [arXiv 2609.39751：Beyond Policy Alignment: Closing the Planning-Learning Loop for Robot Control with Learned World Models](https://arxiv.org/abs/2609.39751)