具身开发者社区

外观

论文arXiv 2608.238312 个月前

等待中学习行动：推理延迟下通用机器人策略的 RL 微调 ​

针对 VLA 等通用策略推理延迟导致 RL 失效，提出 ARLI 延迟感知框架，并在仿真与真实操作任务中验证有效微调。

AI 整理
  • 具身模型
  • 端侧部署

原文链接：Learning to Act While Waiting: RL Finetuning of Generalist Robot Policies Under Inference Latency（arXiv 2608.23831 · 发布日期：2026-08-24）

Learning to Act While Waiting: RL Finetuning of Generalist Robot Policies Under Inference Latency

针对 VLA 等通用策略推理延迟导致 RL 失效，提出 ARLI 延迟感知框架，并在仿真与真实操作任务中验证有效微调。

核心要点 ​

  • 现代通用策略模型规模大，严重推理延迟会造成停顿或抖动，并改变有效环境动力学。
  • ARLI 结合异步推理，在执行中交错生成动作，用低延迟 RL 策略提升推理窗口内反应性。
  • 通过加入已承诺动作和推理中观测的状态增强，近似恢复 RL 所需的马尔可夫结构。
  • 在仿真和真实操作任务中，标准 RL 在延迟下失败，而该方法可有效微调并达到或超过无延迟标准 RL。

验证范围与局限：验证范围为仿真和真实世界操作任务。

英文摘要 (Abstract) ​

While reinforcement learning (RL) allows generalist robot policies to continually improve during deployment, the large model size of modern generalist policies, such as VLAs, poses a fundamental obstacle to effective RL improvement. In particular, their severe inference latency---which can lead to pauses or jerky movements---can alter the effective environment dynamics and, if not correctly accounted for, break the Markov assumption that RL relies on, causing standard RL algorithms to fail completely. In this work, we introduce a latency-aware framework, Asynchronous RL with Intermediate Information (ARLI), that enables RL-based improvement of generalist policies under inference delays. Our framework builds on asynchronous inference approaches, which interleave action generation with execution to hide latency, and addresses its incompatibility with RL by providing a low-latency RL policy design that maximizes reactivity within the inference window through two contributions: state augmentations that restore near-Markovian structure by incorporating committed actions and a mid-inference observation. We evaluate our approach across simulated and real-world manipulation tasks, and find that it enables effective finetuning under inference delays where standard RL fails entirely, even matching or exceeding the performance of standard RL in idealized no-latency settings.

作者：Brian Zhu, Momen Khalil, E Harrison, Emanuele Poggi, Philipp Schmitt, Bernd Kast, Philine Meister, Pranav Atreya, Qiyang Li, Finn Ferchau

来源

本页事实对应的公开页面，以原文为准。

  1. arXiv 2608.23831：Learning to Act While Waiting: RL Finetuning of Generalist Robot Policies Under Inference Latency访问 2026-10-21