论文/arXiv 2609.21788·1 个月前

从预训练到熟练：以最少人工干预实现长时程操作的真实世界子任务RL

针对长时程操作中预训练策略卡在关键子任务的问题，PARTS用真实世界子任务RL微调，并在双臂YAM和单臂Franka任务上验证。

AI 整理

• 数据与采集
• 具身模型

Franka FR3

原文链接：From Pretraining to Proficiency: Real-World Subtask RL for Long-Horizon Manipulation with Minimal Human Intervention（arXiv 2609.21788 · 发布日期：2026-09-18）

From Pretraining to Proficiency: Real-World Subtask RL for Long-Horizon Manipulation with Minimal Human Intervention

针对长时程操作中预训练策略卡在关键子任务的问题，PARTS用真实世界子任务RL微调，并在双臂YAM和单臂Franka任务上验证。

核心要点

• 预训练基础策略能完成多数流程，但常在少数关键子任务失败；全任务SFT示教会重复已会行为。
• PARTS冻结预训练策略提供标称动作，用选择器和成功验证器触发残差修正并给局部奖励。
• 结合在线RL与成功重加权再训练，YAM成功率由32%升至61%，Franka由50%升至95%。
• 同等机器人rollout预算下，较现有真实世界RL微调全任务成功率高逾25%，人工更少。

验证范围与局限：仅在摘要所述双臂YAM和单臂Franka任务的真实世界rollout中验证，需人工识别瓶颈并按需物理复位。

英文摘要 (Abstract)

A pretrained robot foundation policy may execute most of a long-horizon task yet repeatedly fail at a few critical subtasks. Collecting additional full-task demonstrations for supervised fine-tuning (SFT) requires operators to repeat behaviors the policy already performs well. Reinforcement learning (RL) fine-tuning offers a promising path to bridge this gap, but existing approaches struggle to solve long-horizon tasks using only sparse rewards. We present PARTS (Policy Adaptation with RL on Targeted Subtasks), a real-world subtask RL framework that concentrates practice at these bottlenecks while allowing training rollouts to proceed with minimal human intervention. The frozen pretrained policy supplies nominal actions throughout execution, while agent-generated selectors and success verifiers activate residual corrections and provide local outcome rewards. These rewards support learning from successful subtasks even when complete-task successes are scarce. Training combines online RL with success-reweighted retraining, and each retrained residual policy is redeployed to collect further experience. Humans identify bottlenecks during setup and perform physical resets when needed. On bimanual YAM and single-arm Franka tasks, PARTS improves complete-task success from 32% to 61% and from 50% to 95%, respectively, using tens of minutes of real-world RL rollouts per task on average. Compared with existing real-world RL fine-tuning methods, PARTS raises full-task success by more than 25% under the same robot-rollout budget while requiring less human involvement.

作者：Sichang Su, Benjamin Yang, Zhiyun Deng, Boyuan Liang, Yip Fun Yeung, Zelin Wang, Lingfeng Sun

来源

本页事实对应的公开页面，以原文为准。

1. arXiv 2609.21788：From Pretraining to Proficiency: Real-World Subtask RL for Long-Horizon Manipulation with Minimal Human Intervention访问 2026-10-21