具身开发者社区

外观

论文arXiv 2608.266592 个月前

用于可扩展强化学习的简单 Actor 与深度 Critic ​

LAC将容量投向训练期深度Critic并用轻量确定性Actor，在OGBench上匹配生成式基线且推理延迟更低。

AI 整理
  • 数据与采集
  • 具身模型
  • 端侧部署

原文链接：Simple Actors and Deep Critics for Scalable Reinforcement Learning（arXiv 2608.26659 · 发布日期：2026-08-27）

Simple Actors and Deep Critics for Scalable Reinforcement Learning

LAC将容量投向训练期深度Critic并用轻量确定性Actor，在OGBench上匹配生成式基线且推理延迟更低。

核心要点 ​

  • 离线RL中的扩散和flow-matching actor能建模多模态行为，但部署时每步动作开销较大。
  • 作者将能力分配转向仅训练使用的Critic，并分析深层Critic的三类不稳定失效。
  • 方法采用残差MLP骨干、n-step bootstrap目标和分类交叉熵损失，配合轻量确定性Actor。
  • 在OGBench上，LAC匹配最强生成式基线，推理延迟最高降低4倍，且Critic方案可迁移到不同Actor参数化。

验证范围与局限：验证范围为OGBench实验，摘要未说明真机部署或具体任务范围。

英文摘要 (Abstract) ​

Recent progress in offline reinforcement learning (RL) has been driven by expressive generative actors such as diffusion and flow-matching policies, which capture multimodal behavior in offline datasets. However, these actors require multiple denoising or integration steps per action and thus incur substantial overhead at every decision in deployment. In this work, we revisit where capacity should be invested in an offline actor--critic method. Since the critic is used only during training and is discarded at deployment while the actor runs at every decision step, allocating capacity to the critic rather than the actor is more favorable for inference-time efficiency. However, scaling MLP critics in offline RL is known to introduce several distinct instabilities that have, in practice, kept critics shallow. We identify three distinct failure modes that arise when critics are deepened in offline RL---optimization, bootstrap-noise amplification, and value-range drift---and address each with a corresponding ingredient: a residual MLP backbone, n-step bootstrap targets, and a categorical cross-entropy loss. Combining these ingredients with a lightweight deterministic actor, we propose LAC (Light Actor, deep Critic). On OGBench, LAC matches the strongest diffusion- and flow-matching baselines while achieving up to 4x lower inference latency, comparable to one-step distilled policies without distillation. Its critic recipe also transfers across actor parametrizations.

作者：Guhyeon Kang, Jaehwi Lee, Minhae Kwon

来源

本页事实对应的公开页面，以原文为准。

  1. arXiv 2608.26659：Simple Actors and Deep Critics for Scalable Reinforcement Learning访问 2026-10-21