# 分块式基于模型的策略优化

> 原文链接：https://robotics.hellobike.com/papers/2026-10-10-2610-14564
> 频道：papers | 更新日期：2026-10-10 | 标签：arxiv, cs.RO
> 核心摘要：针对基于模型RL的误差累积、推理延迟与BPTT不稳，提出分块可微评估优化，并在HumanoidBench和真实四足行走验证。

# 分块式基于模型的策略优化

::: v-pre
> 原文链接：[Chunk-wise Model-based Policy Optimization](https://arxiv.org/abs/2610.14564)（arXiv 2610.14564 · 发布日期：2026-10-10）

**Chunk-wise Model-based Policy Optimization**

针对基于模型RL的误差累积、推理延迟与BPTT不稳，提出分块可微评估优化，并在HumanoidBench和真实四足行走验证。

## 核心要点

- 现有基于模型RL虽样本效率高，但存在循环误差累积、推理延迟高和BPTT不稳定。
- 方法将chunk world model作为训练时可微评估器，无需循环模型调用来优化chunk actor。
- 世界模型与策略协同分块预测，单次前向生成H步假想rollout并直接优化价值展开。
- 在HumanoidBench的19个高维类人任务上评估，并部署到真实世界四足双足行走。

**验证范围与局限**：验证范围为HumanoidBench的19个高维类人任务和真实世界四足双足行走。

## 英文摘要 (Abstract)

Model-based reinforcement learning has shown superior sample efficiency by leveraging the learned latent dynamics either in the form of derivative-free planning/imagined rollout training or dynamics gradient-based policy updates. While these approaches achieve comparatively fast learning, they all suffer, to varying degrees, from issues of recurrent compounding errors, high inference-time latency, and instability in backward propagation through time (BPTT). In this work, we propose a model-based RL approach that uses the chunk world model as a differentiable training-time evaluator to optimize the chunk actor without recurrent model invocation. Through the collaborative chunk-wise prediction of the world model and policy, we can generate H-step hypothetical rollouts in a single forward pass and directly optimize value expansion, avoiding the costly and unstable BPTT. We evaluate our method on 19 challenging high-dimensional humanoid tasks from HumanoidBench and deploy our method to the real-world for quadruped bipedal walking. Project Website: https://song-wx.github.io/cmpo.github.io/

作者：Weixi Song, Donglin Wang, Yong-Lu Li
:::

## 原始参考来源 (Sources)

- [arXiv 2610.14564：Chunk-wise Model-based Policy Optimization](https://arxiv.org/abs/2610.14564)