外观
论文arXiv 2310.160292023-10-24
FOWM框架实现真实机器人世界模型离线预训练与在线少样本微调
本文由 AI 根据公开内容整理，可能有误，请以原文为准：Finetuning Offline World Models in the Real World（arXiv 2310.16029）
核心内容
针对强化学习在真实机器人上数据效率低、纯离线RL存在分布偏移的问题，研究提出FOWM（Finetuning Offline World Models）框架。该方法利用真实机器人采集的离线数据预训练世界模型，再通过模型规划生成的在线数据进行微调，旨在兼顾数据效率与新任务适应性，解决了固定数据集导致的泛化受限难题。
技术核心在于测试时对规划器进行正则化，通过平衡估计回报与认知不确定性来缓解在线交互中的外推误差。实验覆盖多种视觉运动控制任务，结果表明在离线数据有限条件下，该方法仍能实现对已知及未知任务的少样本微调。相关视频、代码及数据集已在项目主页开源，为具身智能算法验证提供了完整基准。
关键看点
- 混合训练范式：采用离线数据预训练世界模型并结合在线规划数据微调，突破纯离线RL分布偏移限制
- 不确定性正则化：测试时规划引入认知不确定性与回报平衡机制，有效抑制在线交互中的模型外推误差
- 少样本泛化验证：在仿真与真机视觉运动控制任务中证实，有限离线数据下即可实现新旧任务少样本适配
- 开源资产完备：同步开放项目代码、演示视频及真实机器人数据集，支持社区复现与算法基准评测
工程与产业观察
该工作为具身智能体从离线数据向真实环境迁移提供了工程化范式。通过不确定性感知规划解决世界模型外推风险，降低了真机微调的数据门槛。其开源的真机数据集与代码填补了视觉运动控制领域离线RL基准空白，对提升机器人技能学习效率具有直接参考价值。
英文摘要 (Abstract)
Reinforcement Learning (RL) is notoriously data-inefficient, which makes training on a real robot difficult. While model-based RL algorithms (world models) improve data-efficiency to some extent, they still require hours or days of interaction to learn skills. Recently, offline RL has been proposed as a framework for training RL policies on pre-existing datasets without any online interaction. However, constraining an algorithm to a fixed dataset induces a state-action distribution shift between training and inference, and limits its applicability to new tasks. In this work, we seek to get the best of both worlds: we consider the problem of pretraining a world model with offline data collected on a real robot, and then finetuning the model on online data collected by planning with the learned model. To mitigate extrapolation errors during online interaction, we propose to regularize the planner at test-time by balancing estimated returns and (epistemic) model uncertainty. We evaluate our method on a variety of visuo-motor control tasks in simulation and on a real robot, and find that our method enables few-shot finetuning to seen and unseen tasks even when offline data is limited. Videos, code, and data are available at https://yunhaifeng.com/FOWM .
作者：Yunhai Feng, Nicklas Hansen, Ziyan Xiong, Chandramouli Rajagopalan, Xiaolong Wang
来源
本页事实对应的公开页面，以原文为准。