论文arXiv 2611.150119 天前
A²WM框架实现3.8ms推理时策略修正,冻结权重下提升操作成功率
本文由 AI 根据公开内容整理,可能有误,请以原文为准:Fast Inference-Time Policy Refinement via Action-Affine World Models(arXiv 2611.15011)
核心内容
针对模仿策略在执行复杂操作任务时因微小误差累积导致失败的问题,研究团队提出动作仿射世界模型(A²WM)以实现快速推理时策略修正。该方法核心在于利用预训练策略提供的标称行为,专注于预测小幅度动作修正对结果的影响。在四项模拟操作任务的基准测试中,A²WM取得了最高成功率,且每个控制步骤的平均修正开销仅为3.8ms(不含基础策略采样时间),验证了其在保持模型与策略权重冻结前提下的高效纠错能力。
技术上,A²WM通过动作扰动策略回放数据离线学习局部动作敏感度,将任务相关几何预测建模为“标称轨迹+线性修正响应”,直接预测完整规划视域以避免递归评估中的中间误差传播。在线推理阶段,系统结合预测的任务误差二次惩罚项与线性修正约束,构建低计算成本的凸二次规划(QP)问题。整个修正流程仅需一次模型前向传播和一次QP求解,规避了传统方法所需的额外在线策略学习或高算力实时规划瓶颈,显著降低了部署门槛。
关键看点
- 极低推理延迟:在线修正仅需单次模型评估加一次QP求解,每步平均额外耗时仅3.8ms,满足高频控制实时性要求
- 动作仿射建模:采用标称轨迹加线性响应形式直接预测全视域几何,避免递归模型评估导致的中间误差累积传播
- 零参数更新:推理阶段基础策略与世界模型权重完全冻结,无需额外在线学习或微调即可适配新任务场景
- 仿真性能验证:在四项模拟操作任务基准测试中成功率超越所有对比基线,证实离线学习局部敏感度的有效性
工程与产业观察
该工作为具身智能操作提供了一种轻量级推理时自适应范式。通过将非线性动力学局部线性化并转化为凸优化问题,成功解耦了策略修正与昂贵的重训练/重规划过程。对于已部署但存在长尾失败的机器人系统,这种冻结权重、低延迟的“补丁式”修正方案具有极高的工程落地价值,可作为提升模仿学习鲁棒性的通用中间件。
英文摘要 (Abstract)
Imitation policies can perform complex manipulation, yet small execution errors can accumulate and cause failure. Refinement can correct these errors, but existing approaches often require additional on-policy learning or costly online planning. Our key insight is that a pretrained policy already supplies a useful nominal behavior: refinement can focus on predicting how small action corrections change its outcome. We therefore learn this local response in a form that is cheap to optimize. Our \textit{action-affine world model} (A\textsuperscript{2}WM) learns local action sensitivities offline from action-perturbed policy rollouts. Given interaction history and a policy proposal, it predicts task-relevant geometry as a nominal trajectory plus a linear response to corrections. It predicts the full planning horizon directly, avoiding the propagation of intermediate errors through recursive model evaluations. Quadratic penalties on predicted task errors and linear correction constraints then yield a cheap convex quadratic program (QP). Online refinement requires one model evaluation and one QP solve, while both policy and model weights remain frozen. Across four simulated manipulation tasks, our framework achieves the highest success rate among evaluated baselines, adding only 3.8 ms of mean refinement overhead per control step, excluding base-policy sampling.
作者:Run Wang, Yilin Mo, Liang Wu
外部来源
本页事实对应的公开页面,以原文为准。(点击将离开本站)