具身开发者社区

外观

论文arXiv 2610.2285212 天前

通过流反转动作锚实现样本高效的强化学习策略改进 ​

AnchorQ用流反转动作锚与价值引导采样提升通用机器人策略，并在仿真和真实操作任务中验证。

AI 整理
  • 具身模型
  • 端侧部署

原文链接：Sample-Efficient Reinforcement Learning for Policy Improvement via Flow-Reversed Action Anchors（arXiv 2610.22852 · 发布日期：2026-10-16）

Sample-Efficient Reinforcement Learning for Policy Improvement via Flow-Reversed Action Anchors

AnchorQ用流反转动作锚与价值引导采样提升通用机器人策略，并在仿真和真实操作任务中验证。

核心要点 ​

  • 通用机器人策略有较强行为先验，但部署可靠性与吞吐仍受在线交互成本制约。
  • AnchorQ将参考行为映射为状态条件潜在锚点，并在锚点周围采样粒子细化动作。
  • 相同交互预算下优于后训练基线，真实成功率提升37%，仿真吞吐提高97%。
  • 论文提供额外可视化网站，但摘要未声明代码、模型或数据开源。

验证范围与局限：验证范围为仿真与真实世界操作任务，具体机器人和任务细节摘要未说明。

英文摘要 (Abstract) ​

Generalist robot policies offer strong behavioral priors, but achieving deployment-level reliability and throughput remains difficult. Reinforcement learning (RL) can improve these policies, yet its effectiveness is constrained by the cost of online interaction. Existing methods adapt frozen flow-matching policies by treating noise space as a continuous action space. We show that this behaviorally unstructured space makes exploration inefficient, often preventing these methods from extracting meaningful learning signals on challenging tasks. We introduce AnchorQ, which enables coarse-to-fine policy improvement through a structured, refinable latent action representation. At each decision step, AnchorQ uses flow reversal to map reference behaviors to state-conditioned latent anchors, samples local particles around those anchors, and applies value-guided sampling to explore promising refinements. Our analysis shows that anchors reduce identification complexity at the cost of support bias; particle refinement trades this bias against action-class complexity; and critic accuracy determines the reliability of KL-regularized, value-guided selection. Across simulated and real-world manipulation tasks, AnchorQ outperforms state-of-the-art post-training baselines in sample efficiency, delivering a 37% improvement in real-world success rate and 97% higher simulated task throughput under the same interaction budget. Additional visualizations are available on our website https://anchorq-rl.com/.

作者：Carl Qi, Weijun Dong, Xiaojie Wang, Peiyi Chen, Rahul Sajnani, Mingyuan Zhang, Amy Zhang, Jose Augusto Lima Barreiros, Manikatan Nambi, Lillian J. Ratliff

来源

本页事实对应的公开页面，以原文为准。

  1. arXiv 2610.22852：Sample-Efficient Reinforcement Learning for Policy Improvement via Flow-Reversed Action Anchors访问 2026-10-21