论文HF Daily Papers · arXiv 2611.1348212 天前
ExRL:通过执行时长控制实现长动作块策略样本高效在线适配
本文由 AI 根据公开内容整理,可能有误,请以原文为准:ExRL: Sample-Efficient Online Adaptation of Long-Chunk Policies via Execution Length Control(HF Daily Papers · arXiv 2611.13482)
核心内容
针对机器人长动作块(Long-Chunk)策略在真实场景中在线适配样本成本过高的问题,研究提出Execution at the Right Length (ExRL)方法。该方法指出直接优化高维动作块导致现有RL算法收敛困难,转而将适配重心转移至预测每个动作块的执行时长。这一范式转换保持了适配空间的低维特性,同时其表达能力随动作块长度线性增长,有效解决了交互数据昂贵场景下的策略微调难题。
ExRL架构包含一个轻量级适配器模块,用于预测基础策略生成的每个动作块的执行步数,并采用Off-policy RL算法进行训练。该设计避免了全量策略参数的频繁更新,仅需极少任务特定超参数调优。实验表明,ExRL在多个仿真基准及真实机器人操作任务中,均能显著提升各类长动作块策略的在线适配效率,验证了通过时序控制替代动作值优化在具身智能落地中的工程有效性。
关键看点
- 核心机制:将RL优化目标从高维动作块数值转为低维执行时长预测,降低适配空间维度
- 系统架构:引入轻量级适配器预测动作块执行步数,结合Off-policy RL实现高效训练
- 性能优势:在仿真与真机任务中验证样本高效性,无需大量任务特定超参数调优
- 工程价值:解决真实物理环境中交互数据采集昂贵导致的长序列策略在线微调瓶颈
工程与产业观察
ExRL为具身智能长时序策略的Sim-to-Real及在线迭代提供了低成本工程解法。通过解耦动作生成与时序执行,规避了高维动作空间的RL样本爆炸问题。该思路对降低真机数据采集依赖、提升长程操作策略在非标环境下的泛化能力具有直接参考价值,尤其适合数据稀缺的工业与服务机器人场景。
英文摘要 (Abstract)
Predicting long action chunks, often spanning dozens of timesteps, has become a key ingredient in increasingly capable robot policies. In this paper, we find that adapting these long-chunk policies with existing reinforcement learning (RL) methods can require many online samples due to the high dimensionality of the action chunks being optimized, making such adaptation impractical in real-world scenarios where interaction data are costly to collect. Our key insight is to shift adaptation from optimizing actions to optimizing how long to execute each predicted action chunk. This keeps the adaptation space low-dimensional while its expressivity grows with the chunk size. We thus introduce Execution at the Right Length (ExRL), a simple approach which augments the base policy with a lightweight adapter that predicts the execution length of each chunk and is trained using off-policy RL algorithms. Across simulation benchmarks and real-world robotic tasks, ExRL enables sample-efficient adaptation of a diverse range of long-chunk policies with minimal task-specific hyperparameter tuning.
Hugging Face Daily Papers 热度:4 upvotes 作者:Xu Luo, Jiaying Yang, Hao Wu, Junlin Xie, Zijin Hong, Ji Zhang, Lianli Gao, Jingkuan Song
外部来源
本页事实对应的公开页面,以原文为准。(点击将离开本站)