论文/arXiv 2610.14564·18 天前

分块式基于模型的策略优化

针对基于模型RL的误差累积、推理延迟与BPTT不稳，提出分块可微评估优化，并在HumanoidBench和真实四足行走验证。

AI 整理

• 本体 SDK 与控制
• 具身模型
• 端侧部署

原文链接：Chunk-wise Model-based Policy Optimization（arXiv 2610.14564 · 发布日期：2026-10-10）

Chunk-wise Model-based Policy Optimization

针对基于模型RL的误差累积、推理延迟与BPTT不稳，提出分块可微评估优化，并在HumanoidBench和真实四足行走验证。

核心要点

• 现有基于模型RL虽样本效率高，但存在循环误差累积、推理延迟高和BPTT不稳定。
• 方法将chunk world model作为训练时可微评估器，无需循环模型调用来优化chunk actor。
• 世界模型与策略协同分块预测，单次前向生成H步假想rollout并直接优化价值展开。
• 在HumanoidBench的19个高维类人任务上评估，并部署到真实世界四足双足行走。

验证范围与局限：验证范围为HumanoidBench的19个高维类人任务和真实世界四足双足行走。

英文摘要 (Abstract)

Model-based reinforcement learning has shown superior sample efficiency by leveraging the learned latent dynamics either in the form of derivative-free planning/imagined rollout training or dynamics gradient-based policy updates. While these approaches achieve comparatively fast learning, they all suffer, to varying degrees, from issues of recurrent compounding errors, high inference-time latency, and instability in backward propagation through time (BPTT). In this work, we propose a model-based RL approach that uses the chunk world model as a differentiable training-time evaluator to optimize the chunk actor without recurrent model invocation. Through the collaborative chunk-wise prediction of the world model and policy, we can generate H-step hypothetical rollouts in a single forward pass and directly optimize value expansion, avoiding the costly and unstable BPTT. We evaluate our method on 19 challenging high-dimensional humanoid tasks from HumanoidBench and deploy our method to the real-world for quadruped bipedal walking. Project Website: https://song-wx.github.io/cmpo.github.io/

作者：Weixi Song, Donglin Wang, Yong-Lu Li

来源

本页事实对应的公开页面，以原文为准。

1. arXiv 2610.14564：Chunk-wise Model-based Policy Optimization访问 2026-10-21