外观
论文arXiv 2609.3975128 天前
超越策略对齐：用学习到的世界模型闭合机器人控制的规划-学习循环
原文链接：Beyond Policy Alignment: Closing the Planning-Learning Loop for Robot Control with Learned World Models（arXiv 2609.39751 · 发布日期：2026-09-30）
Beyond Policy Alignment: Closing the Planning-Learning Loop for Robot Control with Learned World Models
PL-MPC通过改进评论器监督、终端价值估计与动作蒸馏，处理规划-学习闭环，并在HumanoidBench、DMControl及KUKA真机任务上验证。
核心要点
- 规划器决定学习经验，评论器和演员又评分并提出未来计划，形成闭环反馈。
- PL-MPC加入混合多步TD目标、分歧感知终端估计和回报加权演员蒸馏。
- 在HumanoidBench的balance-hard和hurdle上TAR提升，整体表现依任务而定，并在DMControl保持竞争力。
- 代码和数据将发布；消融显示两个任务中不同组件交互不同。
验证范围与局限：验证范围包括HumanoidBench、DMControl，以及7-DoF KUKA IIWA14上扳手-螺母对齐的零样本Sim-to-Real。
英文摘要 (Abstract)
Planning with learned world models combines online trajectory optimization with learned value and policy functions for high-dimensional control. Because the planner determines the experience used for learning, while the learned critic and actor in turn score and propose future plans, planning and learning form a closed feedback loop. TD-MPC is a prominent instance of this design. Recent policy-constrained variants strengthen one part of the loop by aligning the learned policy with planner behavior. We introduce PL-MPC (Planning-Learning MPC), which additionally modifies critic supervision and planner terminal-value estimation. Hybrid multi-step TD targets expose critic updates to more realized rewards before bootstrapping; disagreement-aware terminal estimates reduce the influence of uncertain critic values during MPPI planning; and return-weighted actor distillation emphasizes planner-executed actions from high-return episodes. The world-model architecture and MPPI optimizer are otherwise unchanged. On HumanoidBench, the largest gains occur on \texttt{balance-hard}, where Total Average Return (TAR) increases from $98\pm18$ to $387\pm255$, and \texttt{hurdle}, from $199\pm13$ to $466\pm200$; performance across the broader benchmark remains task dependent, and PL-MPC remains competitive on DMControl. Controlled ablations show different component interactions across the two tasks. We further demonstrate zero-shot sim-to-real transfer on wrench-nut alignment with a 7-DoF KUKA IIWA14, obtaining higher observed success than TD-M(PC)^2 on the training object size and two unseen sizes. Code and data will be available at: https://pl-mpc-humanoid.github.io.
作者：Kowndinya Boyalakuntla, Yuhan Liu, Abdeslam Boularias
来源
本页事实对应的公开页面，以原文为准。