论文HF Daily Papers · arXiv 2610.407994 天前

CaPtain开源后训练管线：4B小模型RL微调达73.8%操作成功率

针对Code-as-Policy机器人智能体，提出CaPtain开源后训练管线。基于Qwen3.5-4B经SFT与多轮RL微调，引入物理Critic解决长程信用分配，LIBERO成功率达73.8%，SimperEnv达83.3%，超越部分前沿闭源模型并支持真机部署。

AI 整理

仿真与训练
具身模型
端侧部署

本文由 AI 根据公开内容整理，可能有误，请以原文为准：CaPtain: A Pipeline for Post-Training Code-as-Policy Robot Agents（HF Daily Papers · arXiv 2610.40799）

核心内容

针对现有Code-as-Policy（CaP）智能体依赖闭源大模型或VLA、小模型失效的痛点，研究团队发布CaPtain开源后训练管线。该方案无需VLA或特权状态，仅基于Qwen3.5-4B基座模型，通过合成数据SFT对齐接口与多轮强化学习（RL）掌握技能。在LIBERO基准测试中，模型成功率从0%提升至SFT后的55.4%，再经RL优化至73.8%；在LIBERO-PRO上达78.5%，且在SimperEnv WidowX任务中实现83.3%成功率，全面超越多个前沿闭源模型代理。

技术核心在于CaPtain-Harness仿真环境与物理Critic机制。Harness封装底层感知、几何与受护运动原语，统一仿真与真机接口；物理Critic利用策略不可见的模拟器特权状态，量化每个Agent请求对最终成功概率的贡献，解决多轮交互中的长程信用分配难题。实验显示RL带来+17.6至+21.2点的鲁棒性增益，且模型经纯仿真适配即可迁移至真实机器人运行。项目已完整开源代码、数据集与模型权重，验证了小参数VLM经系统化后训练可具备高阶具身决策能力。

关键看点

小模型性能突破：Qwen3.5-4B经SFT+RL训练，LIBERO成功率达73.8%，LIBERO-PRO达78.5%，超越部分闭源前沿模型
物理Critic机制：利用仿真特权状态评估单步请求对终局成功的贡献，解决多轮RL中长程信用分配难题
跨域泛化能力：无目标域训练下SimperEnv WidowX成功率83.3%，三组扰动测试提升17.6~21.2点并支持真机部署
全栈开源资产：发布CaPtain-Harness环境、GPT-6 Astra合成轨迹数据及4B模型权重，复现小模型CaP训练全流程

工程与产业观察

CaPtain证实4B级开源VLM通过后训练可达前沿具身决策水平，其物理Critic为多轮RL信用分配提供工程范式。全栈开源降低了小模型Code-as-Policy研发门槛，表明具身智能算法迭代正从依赖大算力预训练转向高效后训练与仿真反馈闭环，利好端侧轻量化部署。

英文摘要 (Abstract)

Across coding, web, and tool-use tasks, language-model agents are now routinely trained with the same recipe: supervised fine-tuning (SFT) to learn the interface, then multi-turn reinforcement learning (RL) to master it. Code-as-policy (CaP) casts robot manipulation in the same form, as the policy writes code against robot APIs and reads back the results, so the same recipe should apply. Yet existing CaP agents sidestep training: they either prompt closed frontier models or delegate manipulation to pre-trained vision-language-action models (VLAs) and high-level skills, while small open vision-language models (VLMs) fail outright. We present CaPtain, an open post-training pipeline that turns a small open VLM into a capable CaP agent, with no VLA, frontier model, or privileged state at test time. CaPtain trains and deploys the policy inside CaPtain-Harness, which exposes low-level perception, geometry, and guarded-motion primitives across simulation and real hardware. We first align the base model via SFT on synthetic trajectories generated with GPT-6 Astra, then apply multi-turn RL. Multi-turn RL must credit tens of agent requests, whose physical effects often surface turns later, from a single end-of-episode outcome. We therefore introduce a physical critic that reads privileged simulator state, never seen by the policy, and credits each request by how much it changes the predicted chance of success. Starting from Qwen3.5-4B (0% zero-shot), CaPtain-4B reaches 55.4% held-out LIBERO success after SFT and 73.8% after RL. On LIBERO-PRO, it reaches 78.5%, surpassing several frontier-model agents, and its RL gains persist across three perturbation suites (+17.6 to +21.2 points). Without target-domain training, it attains 83.3% on SimplerEnv WidowX, exceeds frontier models on CaP-Bench, and runs on a real robot after simulation-only adaptation. We release all code, data, and model weights.

Hugging Face Daily Papers 热度：18 upvotes 作者：walt stephen, Xilin He, Yilei Jiang, Dongming Wu, Jiaxu Wang, Xiangyu Yue