Skip to content

论文arXiv 2611.1418611 天前

K-MBRL算法实现四足机器人双足站立与行走实时控制 ​

本文由 AI 根据公开内容整理,可能有误,请以原文为准:Koopman Model-Based Reinforcement Learning(arXiv 2611.14186)

核心内容 ​

针对未知动力学的非线性机器人系统,研究提出Koopman模型强化学习(K-MBRL)算法。该方法通过随机Koopman算子将非线性动力学近似为线性表示,结合Critic函数估计未来成本,仅需单步模型传播即可保留长期成本信息并抑制预测误差累积。理论层面建立了标准随机近似假设下的策略学习收敛性保证。

算法核心优势在于将依赖模型的策略梯度计算简化为常数矩阵乘法,支持无在线优化的实时前馈控制。仿真中HalfCheetah与四足运动任务表现出高样本效率;物理四足机器人实机验证了高难度动作序列:以后足圆端支撑直立、维持双足行走平衡及安全返回地面,证实了该框架在复杂非线性机动中的工程有效性。

关键看点 ​

  • 线性化加速:利用Koopman算子将非线性动力学映射至线性空间,策略梯度退化为常数矩阵乘积,消除在线优化开销以支持实时控制
  • 误差抑制机制:采用单步模型传播替代长时域Rollout,有效缓解多步预测误差累积,同时通过Critic网络保留长期价值评估能力
  • 复杂机动验证:物理四足机器人成功执行后足圆端支撑直立、双足动态行走及安全落地全流程,验证了算法对强非线性系统的实际控制性能
  • 样本效率优势:在HalfCheetah及四足 locomotion 仿真基准测试中,相比经典Model-Free与Model-Based RL方法展现出更高的样本利用率与收敛速度

工程与产业观察 ​

K-MBRL通过将非线性动力学全局线性化,解决了模型基RL在机器人实时控制中计算负载高与长时域预测发散的矛盾。其无需在线MPC求解的特性降低了端侧算力门槛,为四足等欠驱动系统在非结构化环境下的复杂技能学习提供了兼顾理论收敛性与工程实时性的新范式。

英文摘要 (Abstract) ​

We propose a Koopman model-based reinforcement learning (K-MBRL) algorithm for controlling nonlinear robotic systems with unknown dynamics. The K-MBRL learns a control policy by minimizing an approximate policy objective that combines the analytical immediate cost with an estimate of the future cost evaluated at the one-step predicted state. To this end, we develop a stochastic Koopman operator-based learning method to approximate the unknown nonlinear dynamics in a linear representation, while a critic function estimates the remaining future cost. This formulation offers two key advantages. First, the linear Koopman dynamics simplify the model-dependent policy gradient computation to a constant matrix product, enabling computationally efficient policy improvement. Second, one-step model propagation mitigates the accumulation of prediction errors associated with long-horizon rollouts while retaining long-term cost information through the critic. The learned policy enables real-time control through feedforward evaluation without online optimization. We establish policy learning convergence guarantees under standard stochastic approximation assumptions. Simulations on HalfCheetah and quadruped four-legged locomotion tasks demonstrate strong sample efficiency and improved performance compared with classical model-free and model-based reinforcement learning baselines. A pendulum swing-up task is used to assess policy gradient approximation accuracy and conduct ablation studies on the learning mechanism and key hyperparameters. The proposed K-MBRL is further validated on a physical quadruped robot performing a challenging sequence of maneuvers: rising to an upright posture supported by two rounded rear feet, maintaining balance during bipedal walking, and safely returning to the ground, as shown in Fig.1.

作者:Wenjian Hao, Paulo C. Heredia, Shaoshuai Mou

基于本页技术方向关联的整机参数、核心零部件、实测报告与同赛道选型横评。

外部来源

本页事实对应的公开页面,以原文为准。(点击将离开本站)

  1. arXiv 2611.14186:Koopman Model-Based Reinforcement Learning ↗访问 2026-11-11